Python

How to get IP address for network interface in Python without any libraries

The following example code will get the IPv4 address for the local network interface eth0:

import subprocess
import re

def get_interface_ip(interface):
    """Get IP address for a given network interface"""
    command = f"ip addr show {interface}"
    output = subprocess.check_output(command, shell=True).decode()

    ip_pattern = r'inet\s+(\d+\.\d+\.\d+\.\d+)'
    match = re.search(ip_pattern, output)
    
    if match:
        ip_address = match.group(1)
        return ip_address
    else:
        return None

interface_name = "eth0"
ip_address = get_interface_ip(interface_name)

if ip_address:
    print(f"IP address of {interface_name}: {ip_address}")
else:
    print(f"No IP address found for {interface_name}")

Example output:

IP address of eth0: 192.168.178.221

 

Posted by Uli Köhler in Python

How to compute rolling/moving window average of a NumPy array

The best way to compute the moving average for a NumPy array is to use bn.move_mean()  from the bottleneck library. First, install bottleneck using

pip install bottleneck

Now import it using

import bottleneck as bn

Now you have to decide what to do with windows at the beginning of the array where not enough elements are present. The default behaviour is to set the result to NaN for those windows:

x = [1,2,3,4,5,6]
result = bn.move_mean(x, window=3)
# result = [nan, nan,  2.,  3.,  4.,  5.]

but you can also accept any window even if it has less than window elements.

x = [1,2,3,4,5,6]
result = bn.move_mean(x, window=3, min_count=1)
# result = [1. , 1.5, 2. , 3. , 4. , 5. ]

Another example with window=4:

x = [1,2,3,4,5,6]
result = bn.move_mean(x, window=4, min_count=1)
# result = array([1. , 1.5, 2. , 2.5, 3.5, 4.5])

 

Posted by Uli Köhler in Python

How to fix ImportError: Missing optional dependency ‘xlrd’. Install xlrd >= 1.0.0 for Excel support

Problem:

While trying to read a XLSX file using pandas , you see an error message such as

File /usr/lib/python3.10/importlib/__init__.py:126, in import_module(name, package)
    125         level += 1
--> 126 return _bootstrap._gcd_import(name[level:], package, level)

File :1050, in _gcd_import(name, package, level)

File :1027, in _find_and_load(name, import_)

File :1004, in _find_and_load_unlocked(name, import_)

ModuleNotFoundError: No module named 'xlrd'

During handling of the above exception, another exception occurred:

ImportError                               Traceback (most recent call last)
/home/uli/myproject/Analyze.ipynb Cell 3 in 2
     18 # Determine absolute path
     19 filepath = os.path.join(directory, filename)
...
--> 144         raise ImportError(msg)
    145     else:
    146         return None

ImportError: Missing optional dependency 'xlrd'. Install xlrd >= 1.0.0 for Excel support Use pip or conda to install xlrd.

Solution:

You can install xlrd using pip:

sudo pip install xlrd

 

Posted by Uli Köhler in pandas, Python

How to fix boto3 upload_fileobj TypeError: Strings must be encoded before hashing

Problem:

You are trying to use boto3′ upload_fileobj() to upload a file to S3 storage using code such as

# Create connection to Wasabi / S3
s3 = boto3.resource('s3',
    endpoint_url = 'https://minio.mydomin.com',
    aws_access_key_id = 'ACCESS_KEY',
    aws_secret_access_key = 'SECRET_KEY'
)
# Get bucket object
my_bucket = s3.Bucket('my-bucket')
# Upload string to file
with open("testtext.txt", "r") as f:
    my_bucket.upload_fileobj(f, "test.txt")

But when you try to run it, you see the following stacktrace:

Traceback (most recent call last):
  File "/home/uli/dev/MyProject/put.py", line 13, in <module>
    my_bucket.upload_fileobj(f, "test.txt")
  File "/usr/local/lib/python3.10/dist-packages/boto3/s3/inject.py", line 678, in bucket_upload_fileobj
    return self.meta.client.upload_fileobj(
  File "/usr/local/lib/python3.10/dist-packages/boto3/s3/inject.py", line 636, in upload_fileobj
    return future.result()
  File "/usr/local/lib/python3.10/dist-packages/s3transfer/futures.py", line 103, in result
    return self._coordinator.result()
  File "/usr/local/lib/python3.10/dist-packages/s3transfer/futures.py", line 266, in result
    raise self._exception
  File "/usr/local/lib/python3.10/dist-packages/s3transfer/tasks.py", line 139, in __call__
    return self._execute_main(kwargs)
  File "/usr/local/lib/python3.10/dist-packages/s3transfer/tasks.py", line 162, in _execute_main
    return_value = self._main(**kwargs)
  File "/usr/local/lib/python3.10/dist-packages/s3transfer/upload.py", line 758, in _main
    client.put_object(Bucket=bucket, Key=key, Body=body, **extra_args)
  File "/usr/local/lib/python3.10/dist-packages/botocore/client.py", line 530, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.10/dist-packages/botocore/client.py", line 933, in _make_api_call
    handler, event_response = self.meta.events.emit_until_response(
  File "/usr/local/lib/python3.10/dist-packages/botocore/hooks.py", line 416, in emit_until_response
    return self._emitter.emit_until_response(aliased_event_name, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/botocore/hooks.py", line 271, in emit_until_response
    responses = self._emit(event_name, kwargs, stop_on_response=True)
  File "/usr/local/lib/python3.10/dist-packages/botocore/hooks.py", line 239, in _emit
    response = handler(**kwargs)
  File "/usr/local/lib/python3.10/dist-packages/botocore/utils.py", line 3088, in conditionally_calculate_md5
    md5_digest = calculate_md5(body, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/botocore/utils.py", line 3055, in calculate_md5
    binary_md5 = _calculate_md5_from_file(body)
  File "/usr/local/lib/python3.10/dist-packages/botocore/utils.py", line 3068, in _calculate_md5_from_file
    md5.update(chunk)
TypeError: Strings must be encoded before hashing

Solution:

You need to open the file in binary mode  ("rb") . Instead of

with open("testtext.txt", "r") as f:

use

with open("testtext.txt", "rb") as f:

This will fix the issue.

Posted by Uli Köhler in Python, S3

How to make S3 HeadObject request using boto3

import boto3

# Create connection to S3
s3 = boto3.client('s3',
    endpoint_url = 'https://minio.mydomain.com',
    aws_access_key_id = 'VO5APZH2B2KS75GWORFQ',
    aws_secret_access_key = 'IBVCAVULO2CQTOQEE6VQ'
)

s3.head_object(
    Bucket='my-bucket',
    Key='folder/example-object.txt'
)

 

Posted by Uli Köhler in Python, S3

How to verify AWS Signature Version 4 implementations

You can use the Python botocore package which is a dependency of the boto3 AWS client in order to verify if your implementation produces correct HMAC signatures for a given string-to-sign.

In order to do this, we’ll use a fixed AmzDate i.e. timestamp and fixed (but random) access keys. The string to sign is also some random-ish string. The only thing that matters is that none of the random strings are empty and all values are the same for the verification path using botocore as they are for your own implementation.

After that, compare the output from the botocore implementation with your own custom implementation. While you might want to check your implementation with different values, in practice it works (maybe except for rare corner cases) if it works correctly for one string.

Verifying the outpt

from botocore.auth import SigV4Auth
from collections import namedtuple

# Data structures for isolated testing
Credentials = namedtuple('Credentials', ['access_key', 'secret_key'])
Request = namedtuple('Request', ['context'])

amzDate = "20130524T000000Z" # Fixed date for testing
signer = SigV4Auth(Credentials(
    access_key="GBWZ45MPRGGMO2JILBXA",
    secret_key="346NO6UJCAMHLHX4SMFA"
), "s3", "global")
signature = signer.signature("ThisStringWillBeSigned", Request(
    context={"timestamp": amzDate}
))
print(signature)

With the values given in this script, the output is

3be60989db53028ca485b46a07df9287a1731df74a234ea247a99febb7c2eb31

Verifying intermediary results

If the global result matches, you’re already finished. There is typically no need to check the intermediary results and input strings.

The SigV4Auth.signature() function doesn’t provide any way of accessing the intermediary results. However, we can just copy its source code to obtain the  relevant intermediaries and print those as hex:

secret_key="346NO6UJCAMHLHX4SMFA"
datestamp = "20130524"
region_name = "global"
service_name = "s3"
string_to_sign = "ThisStringWillBeSigned"

sign_input =  (f"AWS4{secret_key}").encode(), datestamp
k_date = signer._sign(*sign_input)
k_region = signer._sign(k_date, region_name)
k_service = signer._sign(k_region, service_name)
k_signing = signer._sign(k_service, 'aws4_request')
sign_result = signer._sign(k_signing, string_to_sign, hex=True)

print("Sign input: ", sign_input)
print("k_date: ", k_date.hex(), "of length: ", len(k_date))
print("k_region: ", k_region.hex(), "of length: ", len(k_region))
print("k_service: ", k_service.hex(), "of length: ", len(k_service))
print("k_signing: ", k_signing.hex(), "of length: ", len(k_signing))
print("sign_result: ", sign_result)

This prints:

Sign input:  (b'AWS4346NO6UJCAMHLHX4SMFA', '20130524')
k_date:  a788ed61da3106091ac303738fe248c3d391e851858d9b048d3fddf0494cac61 of length:  32
k_region:  90331d205578b73aeaf4ef9082cbb704111d29364dcae4d4405ddfefc4e6a8b0 of length:  32
k_service:  a0b2fb2efe1977349c647d28e86d373aaa67ca9f452c15c7cfbdb9a4fabd685b of length:  32
k_signing:  e02df2af0ce8890816c931c8e72168921f5f481dfbcaf92a35324b65fc322865 of length:  32
sign_result:  3be60989db53028ca485b46a07df9287a1731df74a234ea247a99febb7c2eb31

 

Posted by Uli Köhler in Python, S3

How to generate random AWS-like access key / secret key in Python

Typically, AWS access keys have a length of 20 characters from the following set: ABCDEFGHIJKLMNOPQRSTUVWXYZ234567 (also called uppercase base32 encoding)

The following Python code can be used to generate random AWS-like access keys & secret keys:

import base64
import secrets

def generate_random_base32_key(length=20):
    """
    Generate a random uppercase base32 string of the given length.
    """
    bytes_length = (length * 5) // 8
    random_bytes = secrets.token_bytes(bytes_length)
    base32_string = base64.b32encode(random_bytes).decode()[:length]
    return base32_string.upper()

# Usage example
random_access_key = generate_random_base32_key(20)
print(random_access_key)

Example output:

  • IBVCAVULO2CQTOQEE6VQ
  • 5ONXKHGNV3ILTX3BGJGA
  • 2B4QYQ2RFGWK2LSON4YQ
  • VO5APZH2B2KS75GWORFQ
  • SXCBFBDXLUNCJMCFM2SQ
Posted by Uli Köhler in Python, S3

How to automatically remove duplicate docker networks

docker-compose based setups with locally mounted volumes have very few common failure modes in practice. The most important ones are system upgrades to docker stopping all the services and duplicate networks with the same name preventing the startup of a service. Sometimes, docker-compose does not delete the old network properly, possibly due to unclean or unfinished shutdown procedures.

This will result in log messages such as

May 22 21:52:15 myserver docker-compose[2384642]: Removing network etherpad-mydomain_default
May 22 21:52:15 myserver docker-compose[2384642]: network etherpad-mydomain_default is ambiguous (2 matches found based on name)
May 22 21:52:16 myserver systemd[1]: etherpad-mydomain.service: Control process exited, code=exited, status=1/FAILURE

This simple script will find all duplicate network names and simply delete one of them.

#!/usr/bin/env python3
import subprocess
import json

already_seen_networks = set()

output = subprocess.check_output(["docker", "network", "ls", "--format", "{{json .}}"])
for line in output.decode("utf-8").split("\n"):
    line = line.strip()
    if not line: continue
    obj = json.loads(line.strip())
    id = obj["ID"]
    name = obj["Name"]
    if name in already_seen_networks:
        print(f"Detected duplicate network {name}. Removing duplicate network {id}...")
        subprocess.check_output(["docker", "network", "rm", id])

    already_seen_networks.add(name)

Just call this script without any arguments

python docker-remove-duplicate-networks.py

 

Posted by Uli Köhler in Docker, Python

How to compute the weight of a titanium or stainless steel rod using UliEngineering in Python

In the following example, we’ll compute the weight of a pure (i.e. grade 2) titanium rod of dimensions Ø10mm x 100mm and compare it to a 1.4404 steel rod of the same size.

In order to do that, we’ll use UliEngineering.Physics.Density containing pre-defined density values and UliEngineering.Math.Geometry.Cylinder to compute the volume of the rod.

In order to run this script, first install UliEngineering.

from UliEngineering.Math.Geometry.Cylinder import *
from UliEngineering.Physics.Density import *

weight_14404 = cylinder_weight_by_diameter(diameter=0.010, length=0.1, density=Densities['1.4404'])
weight_titanium = cylinder_weight_by_diameter(diameter=0.010, length=0.1, density=Densities['Titanium'])

print(f"Weight of 1.4404 rod: {weight_14404:.3n} kg")
print(f"Weight of Titanium rod: {weight_titanium:.3n} kg")

This prints:

Weight of 1.4404 rod: 0.0628 kg
Weight of Titanium rod: 0.0354 kg

 

Posted by Uli Köhler in Python

How to read KiCAD XML BOM using Python

This script reads an XML BOM (also called python-bom in kicad-cli) using beautifulsoup4 and – as an usage example, generates a RefDes to footprint name dictionary:

from bs4 import BeautifulSoup

def extract_footprints_by_ref_from_xml(filename):
    footprints_by_ref = {} # e.g. "C3" => "Capacitor_SMD:C_0603_1608Metric"
    
    with open(filename, "r") as infile:
        soup = BeautifulSoup(infile, features="xml")
        
    for component in soup.export.components.findChildren("comp"):
        refdes = component.attrs["ref"]
        footprints_by_ref[refdes] = component.footprint.text
    
    return footprints_by_ref

 

Posted by Uli Köhler in KiCAD, Python

How to iterate footprint pads using KiCAD pcbnew plugin Python API

Iterate the pads of a given pcbnew.FOOTPRINT object using

for pad in list(footprint.Pads()):
    # Example of what to do with [pad]
    print(f"{footprint.GetReference()} {pad.GetNetname()}")

Example of how to print all pads for every footprint on the board:

for footprint in list(pcbnew.GetBoard().GetFootprints()):
    for pad in list(footprint.Pads()):
        # Example of what to do with [pad]
        print(f"{footprint.GetReference()} {pad.GetNetname()}")

This will print, for example:

D1 Net-(D1-K)

 

Posted by Uli Köhler in KiCAD, Python

How to get all footprints on board using KiCAD plugin Python API

Use the following for loop to iterate all footprints in the current board:

for footprint in list(pcbnew.GetBoard().GetFootprints()):
    # Example of what to do with [footprint]
    print(footprint.GetReference())

 

Posted by Uli Köhler in KiCAD, Python

How to get all selected zones using KiCAD pcbnew plugin Python API

In our previous post, we showed how to get all selected objects using the KiCAD python API using

pcbnew.GetCurrentSelection()

You can simply filter these entries to obtain just a list of selected zones using either a for loop with inline filtering:

for selected_object in pcbnew.GetCurrentSelection():
    if type(selected_object).__name__ == 'ZONE':
        print(selected_object.GetReference())

or using a list comprehension:

selected_footprints: list[pcbnew.FOOTPRINT] = [
    zone for zone in pcbnew.GetCurrentSelection() if type(zone).__name__ == 'ZONE'
]

Complete plugin example:

#!/usr/bin/env python
import pcbnew
import os

class SimplePlugin(pcbnew.ActionPlugin):
    def defaults(self):
        self.name = "Plugin Name as shown in Pcbnew: Tools->External Plugins"
        self.category = "A descriptive category name"
        self.description = "A description of the plugin and what it does"
        self.show_toolbar_button = False # Optional, defaults to False
        self.icon_file_name = os.path.join(os.path.dirname(__file__), 'simple_plugin.png') # Optional, defaults to ""

    def Run(self):
        board: pcbnew.BOARD = pcbnew.GetBoard()
        footprints: list[pcbnew.FOOTPRINT] = board.GetFootprints()
        
        # TODO Do something useful with [board]
        for selected_object in pcbnew.GetCurrentSelection():
            print(selected_object)

SimplePlugin().register() # Instantiate and register to Pcbnew

Example output (excerpt):

D39
D32
D23
D37
D18
D34
D11
D15

 

Posted by Uli Köhler in KiCAD, Python

KiCAD minimal pcbnew plugin example

The following python script is more or less the minimal example of a KiCAD pcbnew plugin from our previous post How to show wx dialog message in KiCAD pcbnew plugin.

#!/usr/bin/env python
import pcbnew
import wx

class DialogExamplePlugin(pcbnew.ActionPlugin):
    def defaults(self):
        self.name = "Show dialog example"
        self.category = "A descriptive category name"
        self.description = "A description of the plugin and what it does"
        self.show_toolbar_button = False # Optional, defaults to False

    def Run(self):
        dlg = wx.MessageDialog(None, "Please select one or multiple footprints!\n...or use Ctrl+A to select everything.", "No footprints selected", wx.OK | wx.ICON_ERROR)
        dlg.ShowModal()
        dlg.Destroy()

DialogExamplePlugin().register() # Instantiate and register to Pcbnew

You can place this plugin, for example, in

~/.local/share/kicad/7.0/scripting/plugins/DialogExamplePlugin.py

Don’t forget to refresh the plugins from the pcbnew menu.

Posted by Uli Köhler in KiCAD, Python

How to show wx dialog message in KiCAD pcbnew plugin

When using KiCAD’s Python API for pcbnew, you can show a dialog by using the following snippet

# Show info dialog
dlg = wx.MessageDialog(None, "Please select one or multiple footprints!\n...or use Ctrl+A to select everything.", "No footprints selected", wx.OK | wx.ICON_ERROR)
dlg.ShowModal()
dlg.Destroy()

Note that you need to

import wx

at the top of your plugin.

This code will show the following dialog:

 

Complete plugin example:

#!/usr/bin/env python
import pcbnew
import wx

class DialogExamplePlugin(pcbnew.ActionPlugin):
    def defaults(self):
        self.name = "Show dialog example"
        self.category = "A descriptive category name"
        self.description = "A description of the plugin and what it does"
        self.show_toolbar_button = False # Optional, defaults to False

    def Run(self):
        dlg = wx.MessageDialog(None, "Please select one or multiple footprints!\n...or use Ctrl+A to select everything.", "No footprints selected", wx.OK | wx.ICON_ERROR)
        dlg.ShowModal()
        dlg.Destroy()

DialogExamplePlugin().register() # Instantiate and register to Pcbnew

You can place this plugin, for example, in

~/.local/share/kicad/7.0/scripting/plugins/DialogExamplePlugin.py

Don’t forget to refresh the plugins from the pcbnew menu.

Posted by Uli Köhler in KiCAD, Python

How to read KiCAD pick&place position file using pandas in Python

If you’ve exported a KiCAD pick & place position file using the GUI or the command line:

kicad-cli pcb export pos MyPCB.kicad_pcb --units mm -o MyPCB.pos

you can read it from within your Python script using pandas.read_table(...) like this:

import pandas as pd

pos = pd.read_table('KKS-Microcontroller-Board-R2.2.pos', delim_whitespace=True, names=["Ref", "Val", "Package", "PosX", "PosY", "Rot","Side"], comment="#")

Optionally, you can also index pos by the Ref column (which contains values such as C12D1 or U5):

pos.set_index("Ref", inplace=True)

You can also pack all that into a function:

def read_kicad_pos_file(filename):
    pos = pd.read_table(filename, delim_whitespace=True, names=["Ref", "Val", "Package", "PosX", "PosY", "Rot","Side"], comment="#")
    pos.set_index("Ref", inplace=True)
    return pos

If you’ve used .set_index(), you can access a component such as C13 using

pos.loc["C13"]

Example output:

Val                100nF_25V
Package    C_0603_1608Metric
PosX                187.1472
PosY               -101.8243
Rot                    180.0
Side                     top
Name: C1, dtype: object

 

Posted by Uli Köhler in KiCAD, pandas, Python

How to fix Python “match” statement: SyntaxError: invalid syntax

Problem:

Your Python script crashes with SyntaxError: invalid syntax at a match statement, e.g.:

  File "main.py", line 234
    match param:
          ^
SyntaxError: invalid syntax

Solution:

Match statement exist since Python 3.10 – you are using a Python 3.9 or earlier version. You either need to upgrade your Python to 3.10 or later, or rewrite the code and remove the match statement!

Posted by Uli Köhler in Python

How to compute Buck/Boost/LDO output voltage by feedback resistors using Python

You can use the UliEngineering library in order to compute the output voltage of your voltage converter.

First, you need to know Vfb, the feedback voltage of your converter IC, which you need to read from the datasheet. For this example, we’ll use the AP3012‘s feedback voltage of 1.25V.

from UliEngineering.Electronics.VoltageDivider import feedback_actual_voltage
from UliEngineering.EngineerIO import auto_format


# Compute voltage
output_voltage = feedback_actual_voltage("220k", "12.1k", vfb="1.25V")
# output_voltage == 23.97727272727273

# ... or format and print it
auto_format(feedback_actual_voltage, "220k", "12.1k", vfb="1.25V") #  Prints "24.0V"

 

Posted by Uli Köhler in Electronics, Python

How to install pip (Python) on pfSense

Once you have installed python on your pfSense, you might notice that it is missing pip:

[2.6.0-RELEASE][[email protected]]/root: pip
pip: Command not found.

and also python3.8 -m pip doesn’t work:

[2.6.0-RELEASE][[email protected]]/root: python3.8 -m pip
/usr/local/bin/python3.8: No module named pip

Installing it is rather easy, however:

python3.8 -m ensurepip

After that, you can run pip using

python3 -m pip

 

Posted by Uli Köhler in Networking, Python