Python

What does sdist mean in Python?

sdist means source distribution. It is an archive (usually a .tar.gz) that contains all the source files and related data for a project in a single archive. A source distribution is what is downloaded from the PyPI package registry when you run pip install.

You can create the source distribution (sdist) for a project using

python setup.py sdist

 

Posted by Uli Köhler in Python

How to fix Python CFFI package not finding include file when running pip install

Problem:

When you install your package locally using python setup.py install, all include files are found and the package installs correctly

However you run pip install, the compiler is missing some include files

Solution:

Most likely, your include file does not end up in the sdist i.e. the source distribution .tar.gz file. In order to include it, create a MANIFEST.in file in the same folder as setup.py in order to tell setuptools what to include in addition to any file directly referenced in setup.py.

In my case, the include files are in the src folder so I add this to MANIFEST.in:

graft src/

Now, to check if that worked, run

python setup.py sdist

which will show you which files are included in the source distribution. Example:

C:\Users\ukoeh\cv_algorithms>python setup.py sdist
running sdist
running egg_info
writing cv_algorithms.egg-info\PKG-INFO
writing dependency_links to cv_algorithms.egg-info\dependency_links.txt
writing requirements to cv_algorithms.egg-info\requires.txt
writing top-level names to cv_algorithms.egg-info\top_level.txt
reading manifest file 'cv_algorithms.egg-info\SOURCES.txt'
reading manifest template 'MANIFEST.in'
adding license file 'LICENSE'
writing manifest file 'cv_algorithms.egg-info\SOURCES.txt'
running check
warning: check: missing meta-data: either (author and author_email) or (maintainer and maintainer_email) should be supplied

creating cv_algorithms-1.0.4
creating cv_algorithms-1.0.4\cv_algorithms
creating cv_algorithms-1.0.4\cv_algorithms.egg-info
creating cv_algorithms-1.0.4\src
creating cv_algorithms-1.0.4\test
copying files to cv_algorithms-1.0.4...
copying LICENSE -> cv_algorithms-1.0.4
copying MANIFEST.in -> cv_algorithms-1.0.4
copying README.md -> cv_algorithms-1.0.4
copying setup.py -> cv_algorithms-1.0.4
copying cv_algorithms\__init__.py -> cv_algorithms-1.0.4\cv_algorithms
copying cv_algorithms\_checks.py -> cv_algorithms-1.0.4\cv_algorithms
copying cv_algorithms\_ffi.py -> cv_algorithms-1.0.4\cv_algorithms
copying cv_algorithms\classification.py -> cv_algorithms-1.0.4\cv_algorithms
copying cv_algorithms\colorspace.py -> cv_algorithms-1.0.4\cv_algorithms
copying cv_algorithms\contours.py -> cv_algorithms-1.0.4\cv_algorithms
copying cv_algorithms\distance.py -> cv_algorithms-1.0.4\cv_algorithms
copying cv_algorithms\grassfire.py -> cv_algorithms-1.0.4\cv_algorithms
copying cv_algorithms\morphology.py -> cv_algorithms-1.0.4\cv_algorithms
copying cv_algorithms\neighbours.py -> cv_algorithms-1.0.4\cv_algorithms
copying cv_algorithms\popcount.py -> cv_algorithms-1.0.4\cv_algorithms
copying cv_algorithms\text.py -> cv_algorithms-1.0.4\cv_algorithms
copying cv_algorithms\thinning.py -> cv_algorithms-1.0.4\cv_algorithms
copying cv_algorithms\utils.py -> cv_algorithms-1.0.4\cv_algorithms
copying cv_algorithms.egg-info\PKG-INFO -> cv_algorithms-1.0.4\cv_algorithms.egg-info
copying cv_algorithms.egg-info\SOURCES.txt -> cv_algorithms-1.0.4\cv_algorithms.egg-info
copying cv_algorithms.egg-info\dependency_links.txt -> cv_algorithms-1.0.4\cv_algorithms.egg-info
copying cv_algorithms.egg-info\not-zip-safe -> cv_algorithms-1.0.4\cv_algorithms.egg-info
copying cv_algorithms.egg-info\requires.txt -> cv_algorithms-1.0.4\cv_algorithms.egg-info
copying cv_algorithms.egg-info\top_level.txt -> cv_algorithms-1.0.4\cv_algorithms.egg-info
copying src\common.hpp -> cv_algorithms-1.0.4\src
copying src\distance.cpp -> cv_algorithms-1.0.4\src
copying src\grassfire.cpp -> cv_algorithms-1.0.4\src
copying src\neighbours.cpp -> cv_algorithms-1.0.4\src
copying src\popcount.cpp -> cv_algorithms-1.0.4\src
copying src\thinning.cpp -> cv_algorithms-1.0.4\src
copying src\windows.cpp -> cv_algorithms-1.0.4\src
copying test\TestColorspace.py -> cv_algorithms-1.0.4\test
copying test\TestContours.py -> cv_algorithms-1.0.4\test
copying test\TestDistance.py -> cv_algorithms-1.0.4\test
copying test\TestGrassfire.py -> cv_algorithms-1.0.4\test
copying test\TestNeighbours.py -> cv_algorithms-1.0.4\test
copying test\TestPopcount.py -> cv_algorithms-1.0.4\test
copying test\TestThinning.py -> cv_algorithms-1.0.4\test
copying test\TestUtils.py -> cv_algorithms-1.0.4\test
copying test\__init__.py -> cv_algorithms-1.0.4\test
Writing cv_algorithms-1.0.4\setup.cfg
Creating tar archive
removing 'cv_algorithms-1.0.4' (and everything under i

After that, you need to re-release your package. See my cv_algorithms project for a complete example.

Posted by Uli Köhler in Python

How to fix Python ModuleNotFoundError: No module named ‘cv2’ on Windows

Problem:

You see an error message like the following one when running some Python code on Windows:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\ukoeh\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\cv_algorithms\__init__.py", line 5, in <module>
    from .text import *
  File "C:\Users\ukoeh\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\cv_algorithms\text.py", line 7, in <module>
    import cv2
ModuleNotFoundError: No module named 'cv2'

Solution:

Install the opencv-python package using

pip install opencv-python

and retry.

Posted by Uli Köhler in Python, Windows

How to fix Python CFFI error LNK2001: unresolved external symbol PyInit__…

Problem:

When trying to install your Python library using a C module on Windows, you see an error message like

LINK : error LNK2001: unresolved external symbol PyInit__cv_algorithms
build\temp.win-amd64-3.6\Release\src_cv_algorithms.cp36-win_amd64.lib : fatal error LNK1120: 1 unresolved externals
error: command 'C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\BIN\x86_amd64\link.exe' failed with exit status 1120

Solution:

On Windows, CFFI does not properly generate the.

You can easily fix that by adding a template function and including that only on windows: I suggest creating windows.cpp like this:

/**
 * This contains hacks to get the installation on Windows working.
 * This fixes error LNK2001: unresolved external symbol PyInit__cv_algorithms
 */
void PyInit__cv_algorithms(void) { }

Copy & paste the name of the function from the error message! The fix will not work if you use the wrong function name.

Example how to include that file only on Windows:

platform_src = ["src/windows.cpp"] if os.name == 'nt' else []

mod_cv_algorithms = Extension('cv_algorithms._cv_algorithms',
                         sources=['src/main.cpp'] + platform_src)

For a full example, see cv_algorithms where I first implemented this fix.

Posted by Uli Köhler in Python

Paramiko SSH client minimal example: How to connect to SSH server and execute command

This example shows how to use paramiko to connect to [email protected] using the SSH key stored in ~/.ssh/id_ed25519, execute the command ls and print its output. Since paramiko is a pure Python implementation of SSH, this does not require SSH clients to be installed.

import os.path
import paramiko

ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect("192.168.1.112", username="user",
            key_filename=os.path.join(os.path.expanduser('~'), ".ssh", "id_ed25519"))

# Example command:
stdin, stdout, stderr = ssh.exec_command("ls")
output = stdout.read()
print(output)

# Cleanup
ssh.close()

 

Posted by Uli Köhler in Paramiko, Python

Paramiko: How to execute command and get output as string

Using exec_command() in Paramiko returns a tuple (stdin, stdout, stderr). Most of the time, you just want to read stdout and ignore stdin and stderr. You can get the output the command by using stdout.read()(returns a string) or stdout.readlines() (returns a list of lines).

Example:

stdin, stdout, stderr = ssh.exec_command("ls")
output = stdout.read()
print(output)

Also see our full example: Paramiko SSH client minimal example: How to connect to SSH server and execute command

Posted by Uli Köhler in Paramiko, Python

Paramiko SSH client: How to connect using public key authentication

This example shows how to use paramiko to connect to [email protected] using the SSH key stored in ~/.ssh/id_ed25519 using Python:

import os.path
import paramiko

ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect("192.168.1.112", username="user",
            key_filename=os.path.join(os.path.expanduser('~'), ".ssh", "id_ed25519"))

Also see our full example: Paramiko SSH client minimal example: How to connect to SSH server and execute command

Posted by Uli Köhler in Paramiko, Python

How to fix Paramiko SSHException: Server … not found in known_hosts

Problem:

When trying to connect to a SSH server using paramiko using code like

ssh = paramiko.SSHClient()
ssh.connect("192.168.1.112")

you see an error message like

SSHException: Server '192.168.1.112' not found in known_hosts

Solution:

The simplest solution is to add

ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())

before the call to connect():

ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect("192.168.1.112")

This will automatically add unknown host keys to the known hosts store. Note that this to some extent defeats the purpose of verifying host keys and you should manually verify host keys if possible. However, in many applications, there is no-one there to verify host keys in an automated process and thus this solution is more practical. Just keep in mind its security implications. 

Posted by Uli Köhler in Paramiko, Python

How to fix ModuleNotFoundError: No module named ‘conans’

Problem:

When running conanfile.py or other Python scripts, you see an error message like

Traceback (most recent call last):
  File "/home/myuser/myproject/conanfile.py", line 1, in <module>
    from conans import ConanFile, CMake
ModuleNotFoundError: No module named 'conans'

Solution:

Install conan using one of the recommended methods such as the recommended install via pip

sudo pip3 install conan
Posted by Uli Köhler in C/C++, Python

How to fix bellows Error: Invalid value for “-D” / “–database”: File “/home/user/.config/bellows/app.db” does not exist.

Problem:

When running

bellows devices

you see this error message:

Error: Invalid value for "-D" / "--database": File "/home/uli/.config/bellows/app.db" does not exist.

Solution:

Just create the directory and create an empty file for the database:

mkdir -p ~/.config/bellows
touch ~/.config/bellows/app.db

Then retry running your original command like

bellows devices
Posted by Uli Köhler in Python, Zigbee

How to search pubmed entrez API with Python and filter results by metadata

If you want to apply more filters to search results of Pubmed than given in their web interface, you might want to use the entrez API.

The following example shows how you can sort alphabetically by the journal the articles originally appeared in.

I would recommend processing the data in the .json format.

import requests
import json

db = 'pubmed'
domain = 'https://www.ncbi.nlm.nih.gov/entrez/eutils'
nresults = 10
query = "depression"
retmode='json'

# standard query
queryLinkSearch = f'{domain}/esearch.fcgi?db={db}&retmax={nresults}&retmode={retmode}&term={query}'
response = requests.get(queryLinkSearch)
pubmedJson = response.json()

results = []

for paperId in pubmedJson["esearchresult"]["idlist"]:
    # metadata query
    queryLinkSummary = f'{domain}/esummary.fcgi?db={db}&id={paperId}&retmode={retmode}'
    results.append({'paperId': paperId, 'metadata': requests.get(queryLinkSummary).json()})
    
    # check the journalnames 
    # print(results[-1]["metadata"]["result"][paperId]["fulljournalname"])

resultsSorted = sorted(results, key=lambda x: x["metadata"]["result"][x["paperId"]]["fulljournalname"])

with open('resultsSorted.json', 'w') as f:
    json.dump(resultsSorted, f)
Posted by Joshua Simon in APIs, Python

How to set username & password in Paho-MQTT

Set username & password in Paho-MQTT using

client.username_pw_set("myusername", "aeNg8aibai0oiloo7xiad1iaju1uch")

You need to call that before calling connect()!

Example of how to connect with username & password:

client = mqtt.Client("mqtt-test") # client ID "mqtt-test"
client.on_connect = on_connect
client.on_message = on_message
client.username_pw_set("myusername", "aeNg8aibai0oiloo7xiad1iaju1uch")
client.connect('127.0.0.1', 1883)
client.loop_forever()  # Start networking daemon

 

Posted by Uli Köhler in MQTT, Python

How to fix Paho-MQTT result code 5

When you see result code 5 in paho-mqtt this means Unauthorized! Typically it means you don’t have the correct username and password set.

Set username & password using

client.username_pw_set("myusername", "aeNg8aibai0oiloo7xiad1iaju1uch")

Example of how to connect with username & password:

client = mqtt.Client("mqtt-test") # client ID "mqtt-test"
client.on_connect = on_connect
client.on_message = on_message
client.username_pw_set("myusername", "aeNg8aibai0oiloo7xiad1iaju1uch")
client.connect('127.0.0.1', 1883)
client.loop_forever()  # Start networking daemon

 

Posted by Uli Köhler in MQTT, Python

Python MQTT subscribe minimal example (Paho-MQTT)

#!/usr/bin/env python3
import paho.mqtt.client as mqtt

def on_connect(client, userdata, flags, rc):
    # This will be called once the client connects
    print(f"Connected with result code {rc}")
    # Subscribe here!
    client.subscribe("my-topic")

def on_message(client, userdata, msg):
    print(f"Message received [{msg.topic}]: {msg.payload}")

client = mqtt.Client("mqtt-test") # client ID "mqtt-test"
client.on_connect = on_connect
client.on_message = on_message
client.username_pw_set("myusername", "aeNg8aibai0oiloo7xiad1iaju1uch")
client.connect('127.0.0.1', 1883)
client.loop_forever()  # Start networking daemon

 

Posted by Uli Köhler in MQTT, Python

How to fix Python ModuleNotFoundError: No module named ‘paho’

Problem:

When running your Python script, you see an error message like

Traceback (most recent call last):
  File "test.py", line 2, in <module>
    import paho.mqtt.client as mqtt
ModuleNotFoundError: No module named 'paho'

Solution:

Install the paho-mqtt package using

pip3 install paho-mqtt

or

pip install paho-mqtt
Posted by Uli Köhler in MQTT, Python

How to compute resistor voltage divider ratio using Python

Use UliEngineering to compute the ratio of the voltage divider:

from UliEngineering.Electronics.VoltageDivider import *

# This prints 0.9578544061302683
print(voltage_divider_ratio("2.2k", "50k"))
# In other words, the output voltage of a 2.2kOhm / 50kOhm voltage divider
# equals 0.9578544061302683 times the input voltage.

You can install UliEngineering using pip such as:

sudo pip3 install UliEngineering

 

Posted by Uli Köhler in Electronics, Python

How to replace host part of IPv6 address using Python

In our previous post Bitwise operation with IPv6 addresses and networks in Python we showed how to perform bitwise operations in Python using the ipaddress module. In this post we will use this previous work to replace just the host part of an IPv6 address, leaving the network part as-is – in other words, we will combine two IPv6 addresses together using a configurable network prefix length.

import ipaddress

def bitwise_and_ipv6(addr1, addr2):
    result_int = int.from_bytes(addr1.packed, byteorder="big") & int.from_bytes(addr2.packed, byteorder="big")
    return ipaddress.IPv6Address(result_int.to_bytes(16, byteorder="big"))

def bitwise_or_ipv6(addr1, addr2):
    result_int = int.from_bytes(addr1.packed, byteorder="big") | int.from_bytes(addr2.packed, byteorder="big")
    return ipaddress.IPv6Address(result_int.to_bytes(16, byteorder="big"))

def bitwise_xor_ipv6(addr1, addr2):
    result_int = int.from_bytes(addr1.packed, byteorder="big") ^ int.from_bytes(addr2.packed, byteorder="big")
    return ipaddress.IPv6Address(result_int.to_bytes(16, byteorder="big"))

def replace_ipv6_host_part(net_addr, host_addr, netmask_length=64):
    # Compute bitmasks
    prefix_network = ipaddress.IPv6Network(f"::/{netmask_length}")
    hostmask = prefix_network.hostmask # ffff:ffff:ffff:ffff:: for /64
    netmask = prefix_network.netmask # ::ffff:ffff:ffff:ffff for /64
    # Compute address
    net_part = bitwise_and_ipv6(net_addr, netmask)
    host_part = bitwise_and_ipv6(host_addr, hostmask)
    # Put together resulting IP
    return bitwise_or_ipv6(net_part, host_part)

# Usage example:
# IP address from which we take the network part ("prefix")
net_addr = ipaddress.IPv6Address("2a01:c22:6f71:9f00:8ce6:2eff:fe60:cc69")
# IP address from which we take the host part (suffix)
host_addr = ipaddress.IPv6Address("::dead:babe:cafe:0000")
print(replace_ipv6_host_part(net_addr, host_addr))

This prints

IPv6Address('2a01:c22:6f71:9f00:dead:babe:cafe:0')

 

Posted by Uli Köhler in Networking, Python

How to fix python ipaddress.IPv6Address AddressValueError: Unexpected ‘/’ in ‘…/64’

Problem:

When trying to parse an IPv6 network address in Python using code like

import ipaddress
ipaddress.IPv6Network("2a01:c23:c0bb:d00:8ce6:2eff:fe60:cc69/64")

you see an error message like

---------------------------------------------------------------------------
AddressValueError                         Traceback (most recent call last)
/tmp/ipykernel_154945/2602627019.py in <module>
----> 1 ipaddress.IPv6Address("2a01:c23:c0bb:d00:8ce6:2eff:fe60:cc69/64")

/usr/lib/python3.8/ipaddress.py in __init__(self, address)
   1836         addr_str = str(address)
   1837         if '/' in addr_str:
-> 1838             raise AddressValueError("Unexpected '/' in %r" % address)
   1839         self._ip = self._ip_int_from_string(addr_str)
   1840 

AddressValueError: Unexpected '/' in '2a01:c23:c0bb:d00:8ce6:2eff:fe60:cc69/64'

Solution 1: Remove the CIDR netmask (/64)

By just removing the slash and the part after it (the CIDR netmask).

Solution 1: Maybe you should use IPv6Network instead of IPv6Address

If you intend to parse the network, use ipaddress.IPv6Network but remember that this will discard all host bits. If you want to use IPv6Address or IPv6Network really depends on what you want to do with it – if you want to refer to a specific host, you almost always want to use IPv6Address.

import ipaddress
ipaddress.IPv6Network("2a01:c23:c0bb:d00:8ce6:2eff:fe60:cc69/64", strict=False)

Note that strict=False is added in order to prevent an exception due to host bits being set – see How to fix Python ipaddress.IPv6Network ValueError: … has host bits set

Posted by Uli Köhler in Networking, Python

How to fix Python ipaddress.IPv6Network ValueError: … has host bits set

Problem:

When trying to parse an IPv6 network address in Python using code like

import ipaddress
ipaddress.IPv6Network("2a01:c23:c0bb:d00:8ce6:2eff:fe60:cc69/64")

you see an error message like

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
/tmp/ipykernel_154945/1312927855.py in <module>
      1 import ipaddress
----> 2 ipaddress.IPv6Network("2a01:c23:c0bb:d00:8ce6:2eff:fe60:cc69/64")

/usr/lib/python3.8/ipaddress.py in __init__(self, address, strict)
   2106         if packed & int(self.netmask) != packed:
   2107             if strict:
-> 2108                 raise ValueError('%s has host bits set' % self)
   2109             else:
   2110                 self.network_address = IPv6Address(packed &

ValueError: 2a01:c23:c0bb:d00:8ce6:2eff:fe60:cc69/64 has host bits set

Solution 1: Maybe you should use IPv6Address instead of IPv6Network

If you intend to parse the address and don’t care about the network, use ipaddress.IPv6Address but remember that the CIDR mask (e.g. /64)  needs to be removed. If you want to use IPv6Address or IPv6Network really depends on what you want to do with it – if you want to refer to a specific host, you almost always want to use IPv6Address.

import ipaddress
ipaddress.IPv6Address("2a01:c23:c0bb:d00:8ce6:2eff:fe60:cc69")

Solution 2: Use strict=False to let IPv6Network discard the host bits:

strict=False basically ignores this error

import ipaddress
ipaddress.IPv6Network("2a01:c23:c0bb:d00:8ce6:2eff:fe60:cc69/64", strict=False)

Note that the result will be

IPv6Network('2a01:c23:c0bb:d00::/64')

so the information in the host bits will be lost!

Solution 2: Remove the host bits manually

Since the host bits will be discarded anyway, you can just specify the IPv6 network with the correct netmask:

import ipaddress
ipaddress.IPv6Network("2a01:c23:c0bb:d00::/64")

 

Posted by Uli Köhler in Networking, Python

How to fix Pronterface/Printrun UnicodeDecodeError: ‘utf-8’ codec can’t decode byte … in position …: invalid continuation byte

Problem:

When starting Pronterface / Printrun on Linux using pronterfaceprintrunpython3 pronterface.py or similar, you see an error message like

Traceback (most recent call last):
  File "pronterface.py", line 62, in <module>
    app = PronterApp(False)
  File "/home/uli/dev/tools/Printrun/printrun/pronterface.py", line 2464, in __init__
    self.mainwindow = PronterWindow(self)
  File "/home/uli/dev/tools/Printrun/printrun/pronterface.py", line 170, in __init__
    self.parse_cmdline(sys.argv[1:])
  File "/home/uli/dev/tools/Printrun/printrun/pronsole.py", line 786, in parse_cmdline
    self.process_cmdline_arguments(args)
  File "/home/uli/dev/tools/Printrun/printrun/pronterface.py", line 1031, in process_cmdline_arguments
    pronsole.pronsole.process_cmdline_arguments(self, args)
  File "/home/uli/dev/tools/Printrun/printrun/pronsole.py", line 769, in process_cmdline_arguments
    self.load_default_rc()
  File "/home/uli/dev/tools/Printrun/printrun/pronsole.py", line 664, in load_default_rc
    self.load_rc(config)
  File "/home/uli/dev/tools/Printrun/printrun/pronsole.py", line 632, in load_rc
    for rc_cmd in rc:
  File "/usr/lib/python3.8/codecs.py", line 714, in __next__
    return next(self.reader)
  File "/usr/lib/python3.8/codecs.py", line 645, in __next__
    line = self.readline()
  File "/usr/lib/python3.8/codecs.py", line 558, in readline
    data = self.read(readsize, firstline=True)
  File "/usr/lib/python3.8/codecs.py", line 504, in read
    newchars, decodedbytes = self.decode(data, self.errors)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd3 in position 4: invalid continuation byte

Solution

When you see this error message, typically your pronsolerc is corrupted. In order to remove it,

rm -rf ~/.config/Printrun
Posted by Uli Köhler in 3D printing, Python