Python

Jan

2022

How to fix pandas AttributeError: module ‘pandas’ has no attribute ‘timedelta’

Problem:

While running your Python code or Jupyter notebook, you see an error message like

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
/tmp/ipykernel_99995/3617538487.py in <module>
      1 for date in df.index:
----> 2     print(date - pd.timedelta(5, 'years'))

/usr/local/lib/python3.9/dist-packages/pandas/__init__.py in __getattr__(name)
    259         return _SparseArray
    260 
--> 261     raise AttributeError(f"module 'pandas' has no attribute '{name}'")
    262 
    263 

AttributeError: module 'pandas' has no attribute 'timedelta'

Solution:

pandas Timedelta is spelled with a capital T: Timedelta not timedelta.

Jan

2022

How to parse date column in pandas

If you have a pandas column containing a date string

df

which is listed as object:

df["date"]

parse it like this:

df["date"] = pd.to_datetime(df["date"])

After which it will be listed as datetime64[ns]:

df["date"]

Jan

2022

What does sdist mean in Python?

sdist means source distribution. It is an archive (usually a .tar.gz) that contains all the source files and related data for a project in a single archive. A source distribution is what is downloaded from the PyPI package registry when you run pip install.

You can create the source distribution (sdist) for a project using

python setup.py sdist

Jan

2022

How to fix Python CFFI package not finding include file when running pip install

Problem:

When you install your package locally using python setup.py install, all include files are found and the package installs correctly

However you run pip install, the compiler is missing some include files

Solution:

Most likely, your include file does not end up in the sdist i.e. the source distribution .tar.gz file. In order to include it, create a MANIFEST.in file in the same folder as setup.py in order to tell setuptools what to include in addition to any file directly referenced in setup.py.

In my case, the include files are in the src folder so I add this to MANIFEST.in:

graft src/

Now, to check if that worked, run

python setup.py sdist

which will show you which files are included in the source distribution. Example:

C:\Users\ukoeh\cv_algorithms>python setup.py sdist
running sdist
running egg_info
writing cv_algorithms.egg-info\PKG-INFO
writing dependency_links to cv_algorithms.egg-info\dependency_links.txt
writing requirements to cv_algorithms.egg-info\requires.txt
writing top-level names to cv_algorithms.egg-info\top_level.txt
reading manifest file 'cv_algorithms.egg-info\SOURCES.txt'
reading manifest template 'MANIFEST.in'
adding license file 'LICENSE'
writing manifest file 'cv_algorithms.egg-info\SOURCES.txt'
running check
warning: check: missing meta-data: either (author and author_email) or (maintainer and maintainer_email) should be supplied

creating cv_algorithms-1.0.4
creating cv_algorithms-1.0.4\cv_algorithms
creating cv_algorithms-1.0.4\cv_algorithms.egg-info
creating cv_algorithms-1.0.4\src
creating cv_algorithms-1.0.4\test
copying files to cv_algorithms-1.0.4...
copying LICENSE -> cv_algorithms-1.0.4
copying MANIFEST.in -> cv_algorithms-1.0.4
copying README.md -> cv_algorithms-1.0.4
copying setup.py -> cv_algorithms-1.0.4
copying cv_algorithms\__init__.py -> cv_algorithms-1.0.4\cv_algorithms
copying cv_algorithms\_checks.py -> cv_algorithms-1.0.4\cv_algorithms
copying cv_algorithms\_ffi.py -> cv_algorithms-1.0.4\cv_algorithms
copying cv_algorithms\classification.py -> cv_algorithms-1.0.4\cv_algorithms
copying cv_algorithms\colorspace.py -> cv_algorithms-1.0.4\cv_algorithms
copying cv_algorithms\contours.py -> cv_algorithms-1.0.4\cv_algorithms
copying cv_algorithms\distance.py -> cv_algorithms-1.0.4\cv_algorithms
copying cv_algorithms\grassfire.py -> cv_algorithms-1.0.4\cv_algorithms
copying cv_algorithms\morphology.py -> cv_algorithms-1.0.4\cv_algorithms
copying cv_algorithms\neighbours.py -> cv_algorithms-1.0.4\cv_algorithms
copying cv_algorithms\popcount.py -> cv_algorithms-1.0.4\cv_algorithms
copying cv_algorithms\text.py -> cv_algorithms-1.0.4\cv_algorithms
copying cv_algorithms\thinning.py -> cv_algorithms-1.0.4\cv_algorithms
copying cv_algorithms\utils.py -> cv_algorithms-1.0.4\cv_algorithms
copying cv_algorithms.egg-info\PKG-INFO -> cv_algorithms-1.0.4\cv_algorithms.egg-info
copying cv_algorithms.egg-info\SOURCES.txt -> cv_algorithms-1.0.4\cv_algorithms.egg-info
copying cv_algorithms.egg-info\dependency_links.txt -> cv_algorithms-1.0.4\cv_algorithms.egg-info
copying cv_algorithms.egg-info\not-zip-safe -> cv_algorithms-1.0.4\cv_algorithms.egg-info
copying cv_algorithms.egg-info\requires.txt -> cv_algorithms-1.0.4\cv_algorithms.egg-info
copying cv_algorithms.egg-info\top_level.txt -> cv_algorithms-1.0.4\cv_algorithms.egg-info
copying src\common.hpp -> cv_algorithms-1.0.4\src
copying src\distance.cpp -> cv_algorithms-1.0.4\src
copying src\grassfire.cpp -> cv_algorithms-1.0.4\src
copying src\neighbours.cpp -> cv_algorithms-1.0.4\src
copying src\popcount.cpp -> cv_algorithms-1.0.4\src
copying src\thinning.cpp -> cv_algorithms-1.0.4\src
copying src\windows.cpp -> cv_algorithms-1.0.4\src
copying test\TestColorspace.py -> cv_algorithms-1.0.4\test
copying test\TestContours.py -> cv_algorithms-1.0.4\test
copying test\TestDistance.py -> cv_algorithms-1.0.4\test
copying test\TestGrassfire.py -> cv_algorithms-1.0.4\test
copying test\TestNeighbours.py -> cv_algorithms-1.0.4\test
copying test\TestPopcount.py -> cv_algorithms-1.0.4\test
copying test\TestThinning.py -> cv_algorithms-1.0.4\test
copying test\TestUtils.py -> cv_algorithms-1.0.4\test
copying test\__init__.py -> cv_algorithms-1.0.4\test
Writing cv_algorithms-1.0.4\setup.cfg
Creating tar archive
removing 'cv_algorithms-1.0.4' (and everything under i

After that, you need to re-release your package. See my cv_algorithms project for a complete example.

Jan

2022

How to fix Python ModuleNotFoundError: No module named ‘cv2’ on Windows

Problem:

You see an error message like the following one when running some Python code on Windows:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\ukoeh\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\cv_algorithms\__init__.py", line 5, in <module>
    from .text import *
  File "C:\Users\ukoeh\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\cv_algorithms\text.py", line 7, in <module>
    import cv2
ModuleNotFoundError: No module named 'cv2'

Solution:

Install the opencv-python package using

pip install opencv-python

and retry.

Jan

2022

How to fix Python CFFI error LNK2001: unresolved external symbol PyInit__…

Problem:

When trying to install your Python library using a C module on Windows, you see an error message like

LINK : error LNK2001: unresolved external symbol PyInit__cv_algorithms
build\temp.win-amd64-3.6\Release\src_cv_algorithms.cp36-win_amd64.lib : fatal error LNK1120: 1 unresolved externals
error: command 'C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\BIN\x86_amd64\link.exe' failed with exit status 1120

Solution:

On Windows, CFFI does not properly generate the.

You can easily fix that by adding a template function and including that only on windows: I suggest creating windows.cpp like this:

/**
 * This contains hacks to get the installation on Windows working.
 * This fixes error LNK2001: unresolved external symbol PyInit__cv_algorithms
 */
void PyInit__cv_algorithms(void) { }

Copy & paste the name of the function from the error message! The fix will not work if you use the wrong function name.

Example how to include that file only on Windows:

platform_src = ["src/windows.cpp"] if os.name == 'nt' else []

mod_cv_algorithms = Extension('cv_algorithms._cv_algorithms',
                         sources=['src/main.cpp'] + platform_src)

For a full example, see cv_algorithms where I first implemented this fix.

Jan

2022

Paramiko SSH client minimal example: How to connect to SSH server and execute command

This example shows how to use paramiko to connect to [email protected] using the SSH key stored in ~/.ssh/id_ed25519, execute the command ls and print its output. Since paramiko is a pure Python implementation of SSH, this does not require SSH clients to be installed.

import os.path
import paramiko

ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect("192.168.1.112", username="user",
            key_filename=os.path.join(os.path.expanduser('~'), ".ssh", "id_ed25519"))

# Example command:
stdin, stdout, stderr = ssh.exec_command("ls")
output = stdout.read()
print(output)

# Cleanup
ssh.close()

Jan

2022

Paramiko: How to execute command and get output as string

Using exec_command() in Paramiko returns a tuple (stdin, stdout, stderr). Most of the time, you just want to read stdout and ignore stdin and stderr. You can get the output the command by using stdout.read()(returns a string) or stdout.readlines() (returns a list of lines).

Example:

stdin, stdout, stderr = ssh.exec_command("ls")
output = stdout.read()
print(output)

Also see our full example: Paramiko SSH client minimal example: How to connect to SSH server and execute command

Jan

2022

Paramiko SSH client: How to connect using public key authentication

This example shows how to use paramiko to connect to [email protected] using the SSH key stored in ~/.ssh/id_ed25519 using Python:

import os.path
import paramiko

ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect("192.168.1.112", username="user",
            key_filename=os.path.join(os.path.expanduser('~'), ".ssh", "id_ed25519"))

Also see our full example: Paramiko SSH client minimal example: How to connect to SSH server and execute command

Jan

2022

How to fix Paramiko SSHException: Server … not found in known_hosts

Problem:

When trying to connect to a SSH server using paramiko using code like

ssh = paramiko.SSHClient()
ssh.connect("192.168.1.112")

you see an error message like

SSHException: Server '192.168.1.112' not found in known_hosts

Solution:

The simplest solution is to add

ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())

before the call to connect():

ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect("192.168.1.112")

This will automatically add unknown host keys to the known hosts store. Note that this to some extent defeats the purpose of verifying host keys and you should manually verify host keys if possible. However, in many applications, there is no-one there to verify host keys in an automated process and thus this solution is more practical. Just keep in mind its security implications.

Jan

2022

How to fix ModuleNotFoundError: No module named ‘conans’

Problem:

When running conanfile.py or other Python scripts, you see an error message like

Traceback (most recent call last):
  File "/home/myuser/myproject/conanfile.py", line 1, in <module>
    from conans import ConanFile, CMake
ModuleNotFoundError: No module named 'conans'

Solution:

Install conan using one of the recommended methods such as the recommended install via pip

sudo pip3 install conan

Jan

2022

How to fix bellows Error: Invalid value for “-D” / “–database”: File “/home/user/.config/bellows/app.db” does not exist.

Problem:

When running

bellows devices

you see this error message:

Error: Invalid value for "-D" / "--database": File "/home/uli/.config/bellows/app.db" does not exist.

Solution:

Just create the directory and create an empty file for the database:

mkdir -p ~/.config/bellows
touch ~/.config/bellows/app.db

Then retry running your original command like

bellows devices

Jan

2022

How to search pubmed entrez API with Python and filter results by metadata

If you want to apply more filters to search results of Pubmed than given in their web interface, you might want to use the entrez API.

The following example shows how you can sort alphabetically by the journal the articles originally appeared in.

I would recommend processing the data in the .json format.

import requests
import json

db = 'pubmed'
domain = 'https://www.ncbi.nlm.nih.gov/entrez/eutils'
nresults = 10
query = "depression"
retmode='json'

# standard query
queryLinkSearch = f'{domain}/esearch.fcgi?db={db}&retmax={nresults}&retmode={retmode}&term={query}'
response = requests.get(queryLinkSearch)
pubmedJson = response.json()

results = []

for paperId in pubmedJson["esearchresult"]["idlist"]:
    # metadata query
    queryLinkSummary = f'{domain}/esummary.fcgi?db={db}&id={paperId}&retmode={retmode}'
    results.append({'paperId': paperId, 'metadata': requests.get(queryLinkSummary).json()})
    
    # check the journalnames 
    # print(results[-1]["metadata"]["result"][paperId]["fulljournalname"])

resultsSorted = sorted(results, key=lambda x: x["metadata"]["result"][x["paperId"]]["fulljournalname"])

with open('resultsSorted.json', 'w') as f:
    json.dump(resultsSorted, f)

Dec

2021

How to set username & password in Paho-MQTT

Set username & password in Paho-MQTT using

client.username_pw_set("myusername", "aeNg8aibai0oiloo7xiad1iaju1uch")

You need to call that before calling connect()!

Example of how to connect with username & password:

client = mqtt.Client("mqtt-test") # client ID "mqtt-test"
client.on_connect = on_connect
client.on_message = on_message
client.username_pw_set("myusername", "aeNg8aibai0oiloo7xiad1iaju1uch")
client.connect('127.0.0.1', 1883)
client.loop_forever()  # Start networking daemon

Dec

2021

How to fix Paho-MQTT result code 5

When you see result code 5 in paho-mqtt this means Unauthorized! Typically it means you don’t have the correct username and password set.

Set username & password using

client.username_pw_set("myusername", "aeNg8aibai0oiloo7xiad1iaju1uch")

Example of how to connect with username & password:

client = mqtt.Client("mqtt-test") # client ID "mqtt-test"
client.on_connect = on_connect
client.on_message = on_message
client.username_pw_set("myusername", "aeNg8aibai0oiloo7xiad1iaju1uch")
client.connect('127.0.0.1', 1883)
client.loop_forever()  # Start networking daemon

Dec

2021

Python MQTT subscribe minimal example (Paho-MQTT)

#!/usr/bin/env python3
import paho.mqtt.client as mqtt

def on_connect(client, userdata, flags, rc):
    # This will be called once the client connects
    print(f"Connected with result code {rc}")
    # Subscribe here!
    client.subscribe("my-topic")

def on_message(client, userdata, msg):
    print(f"Message received [{msg.topic}]: {msg.payload}")

client = mqtt.Client("mqtt-test") # client ID "mqtt-test"
client.on_connect = on_connect
client.on_message = on_message
client.username_pw_set("myusername", "aeNg8aibai0oiloo7xiad1iaju1uch")
client.connect('127.0.0.1', 1883)
client.loop_forever()  # Start networking daemon

Dec

2021

How to fix Python ModuleNotFoundError: No module named ‘paho’

Problem:

When running your Python script, you see an error message like

Traceback (most recent call last):
  File "test.py", line 2, in <module>
    import paho.mqtt.client as mqtt
ModuleNotFoundError: No module named 'paho'

Solution:

Install the paho-mqtt package using

pip3 install paho-mqtt

pip install paho-mqtt

Dec

2021

How to compute resistor voltage divider ratio using Python

Use UliEngineering to compute the ratio of the voltage divider:

from UliEngineering.Electronics.VoltageDivider import *

# This prints 0.9578544061302683
print(voltage_divider_ratio("2.2k", "50k"))
# In other words, the output voltage of a 2.2kOhm / 50kOhm voltage divider
# equals 0.9578544061302683 times the input voltage.

You can install UliEngineering using pip such as:

sudo pip3 install UliEngineering

Dec

2021

How to replace host part of IPv6 address using Python

In our previous post Bitwise operation with IPv6 addresses and networks in Python we showed how to perform bitwise operations in Python using the ipaddress module. In this post we will use this previous work to replace just the host part of an IPv6 address, leaving the network part as-is – in other words, we will combine two IPv6 addresses together using a configurable network prefix length.

import ipaddress

def bitwise_and_ipv6(addr1, addr2):
    result_int = int.from_bytes(addr1.packed, byteorder="big") & int.from_bytes(addr2.packed, byteorder="big")
    return ipaddress.IPv6Address(result_int.to_bytes(16, byteorder="big"))

def bitwise_or_ipv6(addr1, addr2):
    result_int = int.from_bytes(addr1.packed, byteorder="big") | int.from_bytes(addr2.packed, byteorder="big")
    return ipaddress.IPv6Address(result_int.to_bytes(16, byteorder="big"))

def bitwise_xor_ipv6(addr1, addr2):
    result_int = int.from_bytes(addr1.packed, byteorder="big") ^ int.from_bytes(addr2.packed, byteorder="big")
    return ipaddress.IPv6Address(result_int.to_bytes(16, byteorder="big"))

def replace_ipv6_host_part(net_addr, host_addr, netmask_length=64):
    # Compute bitmasks
    prefix_network = ipaddress.IPv6Network(f"::/{netmask_length}")
    hostmask = prefix_network.hostmask # ffff:ffff:ffff:ffff:: for /64
    netmask = prefix_network.netmask # ::ffff:ffff:ffff:ffff for /64
    # Compute address
    net_part = bitwise_and_ipv6(net_addr, netmask)
    host_part = bitwise_and_ipv6(host_addr, hostmask)
    # Put together resulting IP
    return bitwise_or_ipv6(net_part, host_part)

# Usage example:
# IP address from which we take the network part ("prefix")
net_addr = ipaddress.IPv6Address("2a01:c22:6f71:9f00:8ce6:2eff:fe60:cc69")
# IP address from which we take the host part (suffix)
host_addr = ipaddress.IPv6Address("::dead:babe:cafe:0000")
print(replace_ipv6_host_part(net_addr, host_addr))

This prints

IPv6Address('2a01:c22:6f71:9f00:dead:babe:cafe:0')

Sep

2021

How to fix python ipaddress.IPv6Address AddressValueError: Unexpected ‘/’ in ‘…/64’

Problem:

When trying to parse an IPv6 network address in Python using code like

import ipaddress
ipaddress.IPv6Network("2a01:c23:c0bb:d00:8ce6:2eff:fe60:cc69/64")

you see an error message like

---------------------------------------------------------------------------
AddressValueError                         Traceback (most recent call last)
/tmp/ipykernel_154945/2602627019.py in <module>
----> 1 ipaddress.IPv6Address("2a01:c23:c0bb:d00:8ce6:2eff:fe60:cc69/64")

/usr/lib/python3.8/ipaddress.py in __init__(self, address)
   1836         addr_str = str(address)
   1837         if '/' in addr_str:
-> 1838             raise AddressValueError("Unexpected '/' in %r" % address)
   1839         self._ip = self._ip_int_from_string(addr_str)
   1840 

AddressValueError: Unexpected '/' in '2a01:c23:c0bb:d00:8ce6:2eff:fe60:cc69/64'

Solution 1: Remove the CIDR netmask (`/64`)

By just removing the slash and the part after it (the CIDR netmask).

Solution 1: Maybe you should use `IPv6Network` instead of `IPv6Address`

If you intend to parse the network, use ipaddress.IPv6Network but remember that this will discard all host bits. If you want to use IPv6Address or IPv6Network really depends on what you want to do with it – if you want to refer to a specific host, you almost always want to use IPv6Address.

import ipaddress
ipaddress.IPv6Network("2a01:c23:c0bb:d00:8ce6:2eff:fe60:cc69/64", strict=False)

Note that strict=False is added in order to prevent an exception due to host bits being set – see How to fix Python ipaddress.IPv6Network ValueError: … has host bits set

Problem:

Solution:

Problem:

Solution:

Problem:

Solution:

Problem:

Solution:

Example:

Problem:

Solution:

Problem:

Solution:

Problem:

Solution:

Example of how to connect with username & password:

Example of how to connect with username & password:

Problem:

Solution:

Problem:

Solution 1: Remove the CIDR netmask (/64)

Solution 1: Maybe you should use IPv6Network instead of IPv6Address

Search

Categories

Solution 1: Remove the CIDR netmask (`/64`)

Solution 1: Maybe you should use `IPv6Network` instead of `IPv6Address`