How to fix MicroPython ‘ValueError: invalid I2C peripheral’

If you see the error message

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: invalid I2C peripheral

you are likely running code like this:

import machine

i2c = machine.I2C(machine.Pin(5), machine.Pin(4))

Solution

The MicroPython API has changed (source: forum). You need to use this syntax instead:

import machine

i2c = machine.I2C(-1, machine.Pin(5), machine.Pin(4))

-1 is the I2C ID that selects a specific peripheral. -1 selects a software I2C implementation which can work on most pins. See the MicroPython I2C class documentation for more details.

How to fix NodeJS MongoDB ‘Cannot read property ‘high_’ of null’

When encountering an error message like

TypeError: Cannot read property 'high_' of null
    at Long.equals (/home/uli/dev/NMUN/node_modules/bson/lib/bson/long.js:236:31)
    at nextFunction (/home/uli/dev/NMUN/node_modules/mongodb-core/lib/cursor.js:473:16)
    at Cursor.next (/home/uli/dev/NMUN/node_modules/mongodb-core/lib/cursor.js:763:3)
    at Cursor._next (/home/uli/dev/NMUN/node_modules/mongodb/lib/cursor.js:211:36)
    at nextObject (/home/uli/dev/NMUN/node_modules/mongodb/lib/operations/cursor_ops.js:192:10)
    at hasNext (/home/uli/dev/NMUN/node_modules/mongodb/lib/operations/cursor_ops.js:135:3)
    (...)

you likely have code like this:

const cursor = db.getCollection('mycollection').find({})
while (cursor.hasNext()) {
    const doc = cursor.next();
    // ... handle doc ...
}

The solution is quite simple: Since find(), cursor.hasNext() and cursor.next() all return Promises, you can’t use their results directly.

This example shows you how to do it properly using async/await:

const cursor = await db.getCollection('mycollection').find({})
while (await cursor.hasNext()) {
    const doc = await cursor.next();
    // ... handle doc ...
}

In order to do this remember that the function containing this code will need to be an async function. See the Mozilla documentation or google for Javascript async tutorial in order to learn about the details!

How to install MongoDB CE on Ubuntu in 1 minute

Run these shell commands on your Ubuntu computer to install the current MongoDB community edition and automatically start it (both instantly and on bootup)

sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 9DA31620334BD75D9DCB49F368818C72E52529D4
echo "deb [ arch=amd64 ] https://repo.mongodb.org/apt/ubuntu bionic/mongodb-org/4.0 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-4.0.list
sudo apt-get update
sudo apt-get install -y mongodb-org
sudo systemctl enable mongod
sudo systemctl start mongod

Source: Official MongoDB documentation

How to run X server using xserver-xorg-video-dummy driver on Ubuntu

When you need to have a dummy display on a headless server (i.e. without any physical monitor attached), first install the dummy driver package:

sudo apt install xserver-xorg-video-dummy

After that, save this config file as dummy-1920x1080.conf

Section "Monitor"
  Identifier "Monitor0"
  HorizSync 28.0-80.0
  VertRefresh 48.0-75.0
  # https://arachnoid.com/modelines/
  # 1920x1080 @ 60.00 Hz (GTF) hsync: 67.08 kHz; pclk: 172.80 MHz
  Modeline "1920x1080_60.00" 172.80 1920 2040 2248 2576 1080 1081 1084 1118 -HSync +Vsync
EndSection

Section "Device"
  Identifier "Card0"
  Driver "dummy"
  VideoRam 256000
EndSection

Section "Screen"
  DefaultDepth 24
  Identifier "Screen0"
  Device "Card0"
  Monitor "Monitor0"
  SubSection "Display"
    Depth 24
    Modes "1920x1080_60.00"
  EndSubSection
EndSection

Now you can start X.org

sudo X -config dummy-1920x1080.conf

The output will look like this:

X.Org X Server 1.19.6
Release Date: 2017-12-20
X Protocol Version 11, Revision 0
Build Operating System: Linux 4.4.0-138-generic x86_64 Ubuntu
Current Operating System: Linux ubuntu-s-1vcpu-2gb-fra1-01 4.15.0-45-generic #48-Ubuntu SMP Tue Jan 29 16:28:13 UTC 2019 x86_64
Kernel command line: BOOT_IMAGE=/boot/vmlinuz-4.15.0-45-generic root=LABEL=cloudimg-rootfs ro console=tty1 console=ttyS0
Build Date: 25 October 2018  04:11:27PM
xorg-server 2:1.19.6-1ubuntu4.2 (For technical support please see http://www.ubuntu.com/support) 
Current version of pixman: 0.34.0
        Before reporting problems, check http://wiki.x.org
        to make sure that you have the latest version.
Markers: (--) probed, (**) from config file, (==) default setting,
        (++) from command line, (!!) notice, (II) informational,
        (WW) warning, (EE) error, (NI) not implemented, (??) unknown.
(==) Log file: "/var/log/Xorg.0.log", Time: Sat Feb 23 17:48:07 2019
(++) Using config file: "dummy-1920x1080.conf"
(==) Using system config directory "/usr/share/X11/xorg.conf.d"

You need to keep this process running as long as you need the display.

Identifying the display number

If no other X server is running, this will, by default, use display number 0. Look for this line to identify the display in use:

(==) Log file: "/var/log/Xorg.0.log", Time: Sat Feb 23 17:48:07 2019

In said line, Xorg.0.log tells you that display 0 is in use, whereas Xorg.1.log tells you that display 1 is in use.

Starting your software

Now you can start your software that needs a graphical interface as follows (we use firefox, and display number 0, as an example):

DISPLAY=:0 firefox

Don’t forget the colon in DISPLAY=:0!

Starting X.org on a specific display number

If you want to start the X server on a specific display number, e.g. 7, because some other dummy server is running concurrently, use this command to start the X server:

sudo X :7 -config dummy-1920x1080.conf

Of course you’ll need to use DISPLAY=:7 firefox in order to start firefox using this config!

In case there is already an X server running on that display, you’ll see an error message like this:

(EE) 
Fatal server error:
(EE) Server is already active for display 7
        If this server is no longer running, remove /tmp/.X7-lock
        and start again.
(EE) 
(EE) 
Please consult the The X.Org Foundation support 
         at http://wiki.x.org
 for help. 
(EE) 

Look for Server is already active for display 7 to be sure you’re seeing the same error message!

How to download a file or directory from a LXC container

To download files, use

lxc file pull <container name>/<path>/<filename> <target directory>

To download directories, use

lxc file pull --recursive <container name>/<path>/<filename> <target directory>

Examples:

Download /root/myfile.txt from mycontainer to the current directory (.):

lxc file pull mycontainer/root/myfile.txt .

Download /root/mydirectory from mycontainer to the current directory (.):

lxc file pull -r mycontainer/root/mydirectory .

 

How to recover WordPress admin access if you only have FTP access

There are well-documented solutions to recover the wordpress admin access if you have MySQL, phpMyAdmin or shell access.

This solution shows you how to create a new admin user if you don’t know the admin username or password and you don’t have any form of MySQL or shell access (only FTP access is required).

Step 1: Identify your currently active theme

This can be done by looking at the source code of your homepage (go to your domain, then Ctrl+U to show the source) and then Ctrl+F-search for wp-content/themes. It will show hits like https://mydomain.de/wp-content/themes/twentyfifteen/style.css. This means the currently active theme is twentyfifteen.

Step 2: Create a new admin user using functions.php

Now open your FTP software (I recommend FileZilla) and find your wp-content folder. Inside wp-content, go to themes and then open the folder of your currently active theme.

If the currently active theme is not listed in your wp-content/themes folder, you might have the wrong wp-content folder. Check if there are any other folders around.

When you have found your theme folder, edit functions.php and, just after the first <?php, add this block of code:

function wpb_admin_account(){
    $user = 'newadmin';
    $pass = 'saiquae9shahZus6eeri3feNae8gie';
    $email = 'admin-email@example.org';
    if ( !username_exists( $user )  && !email_exists( $email ) ) {
        $user_id = wp_create_user( $user, $pass, $email );
        $user = new WP_User( $user_id );
        $user->set_role( 'administrator' );
    }
}
add_action('init','wpb_admin_account');

Be sure to replace the username, password and email! I recommend to use a new random password and a non-generic username! The username you enter here must.

Save the file and upload it to the server.

After that, goto your homepage and reload once (this will create the new user).

Then, try to login using your newly created user (go to e.g. https://my-domain.com/wp-admin to do so!). If it doesn’t work, check if you edited the functions.php for the correct theme and try to use a different username!

Step 3: Delete the code we just created

If you just leave in the code, this will create a potential security risk. So I recommend deleting it right away!

Also, be sure to delete any admin users you don’t need afterwards.

 

How to use Z0 (characteristic impedance of vacuum) constant in SciPy/NumPy

If you want to use the Z0 constant (characteristic impedance of free space) in Python, use this snippet:

import scipy.constants
Z0 = scipy.constants.physical_constants['characteristic impedance of vacuum'][0]

print(Z0) # 376.73031346177066

In contrast to other constants, Z0 is not available directly like scipy.constants.pi but you need to use the scipy.constants.physical_constants dict in order to access it.

How to backup Redmine using the Bitnami Docker image

In a previous post I detailed how to install Redmine on Linux using the excellent Bitnami docker image.

This post will teach you how to easily make an online backup of your Redmine installation. Note that automating the backup is not within the scope of this post.

We assume that the redmine is installed as shown in my previous post in /var/lib/redmine. and that you want to backup to my.backup.server:~/redmine-backup/ using rsync.

Backing up the Redmine data

This is pretty easy, as the data is all in just one directory. You can sync it using

rsync --checksum -Pavz /var/lib/redmine/redmine_data my.backup.server:~/redmine-backup/

Note that old versions of files in redmine_data will be overwritten, however files that are deleted locally will not be deleted on the backup server. To me, this seems like a good compromise between the ability to recover deleted files and the used storage space.

Backing up the Redmine database

This part is slightly more complicated, since we need to access the MariaDB server running in a different container. Important note: The container ID can change so it is not sufficient to just find the container ID once and then use it. You need to determine the appropriate ID each time you do a backup. See below on instructions how to do that.

Full command:

docker exec -it $(docker container ls | grep redmine_mariadb_1 | cut -d' ' -f1) mysqldump -uroot bitnami_redmine | xz -e9 -zc - > redmine-database-dump-$(date -I).sql.xz

Let’s break it down:

  • docker exec -it (container ID) (command): Run a command on a running docker container.
  • docker container ls | grep redmine_mariadb_1 | cut -d' ' -f1: Get the ID (first field of the output cut -d' ' -f1) of the running docker container named redmine_mariadb_1
  • mysqldump -uroot bitnami_redmine: This is run on the docker container and dumps the Redmine Database as SQL to stdout. No password is neccessary since the Bitnami MariaDB image allows access without any password.
  • xz -e9 -zc -: Takes the data from mysqldump from stdin (-), compresses it using maximum compression settings (-e9 -z) and writes the compressed data to stdout.
  • > redmine-database-dump-$(date -I).sql.xz: Writes the compressed data from xz into a file called redmine-database-dump-(current date).sql.xz in the current directory.

The resulting file is called e.g. redmine-database-dump-2019-02-01.sql.xz and it’s placed in the current directory. Ensure that you run the command in a suitable directory. Run it in /tmp if you don’t know which directory might be suitable.

Now we can rsync it to the server:

rsync --checksum -Pavz redmine-backup-*.sql.xz my.backup.server:~/redmine-backup/

Since the filename contains the current data, this approach will not overwrite old daily backups of the database, so you can restore your database very flexibly.

How to calculate STM32 bxCAN bit timings using Python

When I needed to configure some STM32 microcontrollers with nonstandard clock speeds to use the correct CAN bit timings.

While this script has been written for the STM32F413 series, it should be applicable to pretty much all STM32 MCUs with bxCAN – however, you need to check the details, especially from which clock the CAN is derived.

You need to know:

  • The frequency of the clock from which the CAN clock is derived (PCLK1 in my example: 48 MHz)
  • Your desired Baudrate: 1 MBaud in my example

Using an iterative trial and error approach, we will determine the appropriate values for BRP, TS1 and TS2 (these are specific sets of bits in the CAN):

#!/usr/bin/env python3
# Configuration. Change here
master_clock = 48e6 # PCLK1, or whatever CLK CAN is derived from
desired_baudrate = 1e6 # 1e6 = 1 MBaud, 500e3 = 500 kBaud
# Register values
brp = 5 # Baud rate prescaler
ts1 = 4 # (Length of time segment 1) - 1
ts2 = 1 # (Length of time segment 2) - 1

# Calculation. You usually don't need to change this.
bs1_segments = ts1 +1
bs2_segments = ts2 + 1
total_segments = 1 + bs1_segments + bs2_segments
bittime = 1. / desired_baudrate
master_time = 1. / master_clock
tq = (brp + 1) * master_time
total_time = total_segments * tq
effective_rate = 1. / total_time
sample_point = 1 - (bs2_segments / total_segments)

# Print results
print(f"Effective Baudrate: {effective_rate:.2f} Baud = {100. * effective_rate / desired_baudrate:.2f} % of desired baudrate")
print(f"Sample point: {100. * sample_point:.2f}  %")

When you run this script, you will see an output like this:

Effective Baudrate: 1000000.00 Baud = 100.00 % of desired baudrate
Sample point: 75.00  %

Of course you need to fill the master clock frequency and the desired baudrate at the top. You can start with the pre-configured values for BRP, TS1 and TS2.

In the end, you should achieve 100% of desired baudrate and a sample point that is 87.5% (80% to 90% is usually OK, 75% might not work with long cables. There might be a specific value dictated by your comms standard, e.g. CANOpen or J1939).

Change the values of for BRP, TS1 and TS2., re-running the script and observing the output each time. I recommend first changing BRP until you get close and then adjusting using this algorithm:

  • If your actual baudrate is too slow (i.e. < 100%), decrease BRP (larger step) and/or decrease TS1 + TS2
  • If your actual baudrate is too fast (i.e. > 100%), increase BRP (larger step) and/or increase TS1 + TS2
  • If your sample point percentage is too small, increase the ratio of TS1 to TS2
  • If your sample point percentage is too large, decrease the ratio of TS1 to TS2

Note that there might be master clock speeds and baudrates for which there is no ideal setting. Be sure to check whether a slightly different baudrate is still OK in your application (usually it’s not). If it is not, you need to find a way to use a different PCLK1 speed.

How to use custom themes with the Bitnami Redmine Docker image

In a previous post I detailed how to install Redmine on Linux using the excellent Bitnami docker image.

This post shows you how to install a custom theme like A1 (which I used successfully for more than 5 years) if you use the bitnami Docker image. We will assume that you installed redmine in /var/lib/redmine and your systemd service is called redmine.

Note: If you get any permission denied errors, try running the same command using sudo.

First, we need to create the themes directory.

sudo mkdir /var/lib/redmine/themes

The first thing we need to do is to copy the current (default) themes to that directory, since Redmine won’t be able to start up if the default theme isn’t available in the correct version.

In order to do this, we must first ensure that your container is running:

sudo systemctl start redmine

Now we can find out the container ID of the running redmine container:

uli:/var/lib/redmine$ docker container ps | grep redmine
ae4de10d0b41        bitnami/redmine:latest    "/app-entrypoint.sh …"   30 minutes ago      Up 30 minutes   0.0.0.0:3718->3000/tcp   redmine_redmine_1
c231d11c48e9        bitnami/mariadb:latest    "/entrypoint.sh /run…"   30 minutes ago      Up 30 minutes   3306/tcp redmine_mariadb_1

From these lines, you need to select the line that says redmine_redmine_1 at the end. The one that lists redmine_mariadb_1 at the end is the database container and we don’t need that one for this task.

From that line, copy the first column – this is the container ID – e.g. ae4de10d0b41 in this example.

Now we can copy the default theme folder:

docker cp ae4de10d0b41:/opt/bitnami/redmine/public/themes /var/lib/redmine/themes

Now copy your custom theme (e.g. the a1 folder) to /var/lib/redmine/themes.

The next step is to fix the permissions. The bitnami container uses the user with UID 1001, so we need to change the owner to that. Repeat this every time you changed something in the themes directory:

sudo chown -R 1001:1001 /var/lib/redmine/themes

At this point we need to edit the docker-compose config (in /var/lib/redmine/docker-compose.yml) to mount /var/lib/redmine/themes in the correct directory. This is pretty easy: Just add - '/var/lib/redmine-szalata/themes:/opt/bitnami/redmine/public/themes' to the volumes section of the redmine container.

The finished config file will look like this:

version: '2'
services:
  mariadb:
    image: 'bitnami/mariadb:latest'
    environment:
      - ALLOW_EMPTY_PASSWORD=yes
    volumes:
      - '/var/lib/redmine/mariadb_data:/bitnami'
  redmine:
    image: 'bitnami/redmine:latest'
    environment:
      - REDMINE_USERNAME=admin
      - REDMINE_PASSWORD=redmineadmin
      - REDMINE_EMAIL=me@gmail.com
      - SMTP_HOST=smtp.gmail.com
      - SMTP_PORT=25
      - SMTP_USER=me@gmail.com
      - SMTP_PASSWORD=yourGmailPassword
    ports:
      - '3718:3000'
    volumes:
      - '/var/lib/redmine/redmine_data:/bitnami'
      - '/var/lib/redmine-szalata/themes:/opt/bitnami/redmine/public/themes'
    depends_on:
      - mariadb

Now you can restart Redmine:

sudo systemctl restart redmine

and set your new theme by selecting it in Administration -> Settings -> Display.

Fixing ScanaStudio error while loading shared libraries: libftd2xx.so: cannot open shared object file on Ubuntu

Problem:

You want to start IkaLogic ScanaStudio, but you see the following error message:

./ScanaStudio: error while loading shared libraries: libftd2xx.so: cannot open shared object file: No such file or directory

Solution:

You need to download the D2XX drivers from the FTDI D2XX drivers page.

Look for your architecture in the columns (usually x64 (64 bit), this is the same as x86_64) and click the link in the Linux row.

This will download a file named something like libftd2xx-x86_64-1.4.8.gz.

Now you can extract the archive using

tar xvf libftd2xx-*

Now we can copy the shared object file to the IkaLogic ScanaStudio directory:

cp release/build/libftd2xx.so.* ~/Ikalogic/ScanaStudio/libftd2xx.so

Now you can start ScanaStudio:

cd ~/Ikalogic/ScanaStudio
./ScanaStudio

Puppeteer: Get text content / inner HTML of an element

Problem:

You want to use puppeteer to automate testing a webpage. You need to get either the text or the inner HTML of some element, e.g. of

<div id="mydiv">
</div>

on the page.

Solution:

# Get inner text
const innerText = await page.evaluate(() => document.querySelector('#mydiv').innerText);

# Get inner HTML
const innerHTML = await page.evaluate(() => document.querySelector('#mydiv').innerHTML);

Note that .innerText includes the text of sub-elements. You can use the complete DOM API inside page.evaluate(...). You can use any CSS selector as an argument for document.querySelector(...).

Fixing requests session TypeError: __init__() got an unexpected keyword argument ‘headers’

Problem:

You are trying to initialize a Python requests session using a custom set of HTTP headers like this:

s = requests.Session(headers={
    "User-Agent": "Mozilla/5.0 (Windows NT 6.2; WOW64; rv:34.0) Gecko/20100101 Firefox/34.0"
})

but you only see this stacktrace:

File "RequestsSession.py", line 30, in translate
    "User-Agent": "Mozilla/5.0 (Windows NT 6.2; WOW64; rv:34.0) Gecko/20100101 Firefox/34.0"
TypeError: __init__() got an unexpected keyword argument 'headers'

Solution:

You can’t use the headers=... argument in the requests.Session(...) constructor.

Use s.headers.update({...}) instead:

s = requests.Session()
s.headers.update({
    "User-Agent": "Mozilla/5.0 (Windows NT 6.2; WOW64; rv:34.0) Gecko/20100101 Firefox/34.0"
})

 

How to fix ModuleNotFoundError: No module named ‘google.cloud.iam’

Problem:

You want to run a Python script that uses one of the Google Cloud Python APIs but you get this error message:

ModuleNotFoundError: No module named 'google.cloud.iam'

Solution:

Reinstall any google cloud package using pip:

sudo pip install --upgrade google-cloud-storage

or

sudo pip3 install --upgrade google-cloud-storage

That will also reinstall the relevant google.cloud.iam module.

After that, re-run your script. If that didn’t work, try to install --upgrade some other google-cloud-* module, especially the modules you actually use in your script.

 

Fixing numpy.distutils.system_info.NotFoundError: No lapack/blas resources found on Ubuntu or Travis

Note: If you are on Windows, you can not install scipy using pip! Follow this guide instead: https://www.scipy.org/install.html. This blog post is only for Linux-based systems!

When building some of my libraries on Travis, I encountered this error during

sudo pip3 install numpy scipy --upgrade
numpy.distutils.system_info.NotFoundError: No lapack/blas resources

Solution

Install lapack and blas:

sudo apt-get -y install liblapack-dev libblas-dev

In most cases you will then get this error message:

error: library dfftpack has Fortran sources but no Fortran compiler found

Fix that by

sudo apt-get install -y gfortran

In Travis, you can do it like this in .travis.yml:

before_install:
    - sudo apt-get -y install liblapack-dev libblas-dev gfortran

Easily compute & visualize FFTs in Python using UliEngineering

UliEngineering is a mixed data analytics in Python – one of the utilities it provides is an easy-to-use package to compute FFTs. In contrast to other package, this library is oriented towards practical usecases and allows you to do the FFT in only one line of code! No knowledge of Math required.

Getting started

First, we’ll generate some test data. See this previous post for more details on how to generate sinusoid test data:

from UliEngineering.SignalProcessing.Simulation import *

# Generate test data: 100 Hz + 400 Hz tone
data = sine_wave(frequency=100.0, samplerate=1000, amplitude=1.) \
       + sine_wave(frequency=400.0, samplerate=1000, amplitude=0.5)

The test data consists of a 100 Hz sine wave plus a 400 Hz sine wave (with half the amplitude). That signal is sampled with a sample rate of 1000 Hz.

Now we can compute & visualize the FFT using matplotlib:

# Compute FFT. Use the same samplerate 
# NOTE: Window defaults to "blackman"!
from UliEngineering.SignalProcessing.FFT import compute_fft
fft = compute_fft(data, samplerate=1e3)

# Plot
from matplotlib import pyplot as plt
plt.style.use("ggplot")

plt.gcf().set_size_inches(10, 5) # Use (20, 10) to get a larger plot
plt.plot(fft.frequencies, fft.amplitudes)
plt.xlabel("Frequency")
plt.ylabel("Amplitude")


compute_fft(data, samplerate=1e3) returns a tuple (fftx, ffty). It performs an FFT the size of the input (i.e. since data is a length-1000 array, the FFT will be of size 1000).

fft.frequencies is an array of frequencies (in Hz), corresponding to the values in fft.amplitudes. You can also use fft.angles to get relative angles in degrees, but that is not being covered in this blogpost.

As you can see in the plot shown above, the maximum frequency that can be detected is always half the samplerate, i.e. for our samplerate of f_s = 1000\,\text{Hz} it is 500\,\text{Hz}. See this more math-centric FFT explanation if you want to know more details.

Internally, compute_fft() performs this computation:

2 \cdot \frac{\text{abs}\left(\text{FFT}(\text{data} \cdot \text{Window})\right)}{\text{len(data)}}
  • 2 is a correction factor that takes into account that we throw away the latter half of the raw FFT result (since we’re doing a real FFT)
  • \frac{1}{\text{len(data)}} normalizes the FFT results so they are indepedent of the data length (i.e. if you pass a longer sample of the same sine wave you will still get the same result
  • \text{abs}\left(\cdots\right) Converts the complex phase-aware result of the FFT to a spectrum which is easier to read & visualize.
  • Window (which defaults to blackman) is the window which is applied to the data to alleviate some mathematical effects at the beginning and the end of the dataset. See wikipedia on window functions for more details. UliEngineering currently offers this list of window functions:
    • blackman
    • bartlett
    • hamming
    • hanning
    • kaiser (Parameter is fixed to 2.0)
    • none

Selecting frequency ranges

Using the UliEngineering API, selecting a frequency range of the FFT is trivially easy: Just use fft[lowfreq:highfreq]. You can use fft[lowfreq:] to select everything starting from lowfreq or use fft[:highfreq] to select everything up to highfreq.

from UliEngineering.SignalProcessing.FFT import compute_fft
fft = compute_fft(data, samplerate=1e3)

# Select frequency range: 50 to 200 Hz
fft = fft[50.0:200.0]

# Plot
from matplotlib import pyplot as plt
plt.style.use("ggplot")

plt.gcf().set_size_inches(10, 5) # Use (20, 10) to get a larger plot
plt.plot(fft.frequencies, fft.amplitudes)
plt.xlabel("Frequency")
plt.ylabel("Amplitude")
plt.savefig("/ram/fft-frequency-range.svg")

Extract amplitude & angle at a certain frequency

By using [frequency], i.e. getitem operator with a single value, you get an FFTPoint() object containing the frequency, amplitude and relative angle for a given frequency. The library automatically selects the closest FFT bucket, so even if your FFT does not have a bucket for that specific frequency, you will get sensible results.

from UliEngineering.SignalProcessing.FFT import compute_fft
fft = compute_fft(data, samplerate=1e3)

# Show value at a certain frequency
print(fft[30]) # FFTPoint(frequency=30.0, value=2.91e-08, angle=0.0)
print(fft[100]) # FFTPoint(frequency=100.0, value=0.419, angle=0.0)

Short FFTs for long data

Using compute_fft(), if we have an extremely long data array, this means we’ll compute an extremely long FFT. In many cases, this is not desirable and you want to compute a fixed-size FFT (most often a power-of-two FFT, e.g. 1024, 2048, 4096 etc).

Let’s generate some long test data and assume we want to compute a size-1024 FFT on it

from UliEngineering.SignalProcessing.FFT import *
fftx, ffty = simple_serial_fft_reduce(data, samplerate=1e3, fftsize=1024)

Plotting the data like above yields

which looks almost exactly like our compute_fft() plot before – just as we expected.

simple_serial_fft_reduce() takes care of all the magic and normalization for us, including partitioning the data into overlapping chunks, adding the FFTs and properly normalizing the result.

The naming convention is significant here:

  • simple_...._reduce means this is a variant with sensible defaults (simple) for using a reduction function (default: sum) on multiple FFTs.
  • serial means the individual FFTs

Parallelizing the FFTs

If you have a huge dataset, you can use simple_parallel_fft_reduce() identically tosimple_serial_fft_reduce():

from UliEngineering.SignalProcessing.FFT import *
fftx, ffty = simple_parallel_fft_reduce(data, samplerate=1e3, fftsize=1024)

However in most cases you want to initialize the executor manually so you can re-use it later:

from UliEngineering.SignalProcessing.FFT import *
from concurrent.futures import ThreadPoolExecutor
executor = ThreadPoolExecutor() # No argument => use num_cpus threads
fftx, ffty = simple_parallel_fft_reduce(data, samplerate=1e3, fftsize=1024, executor=executor)

We can use a ThreadPoolExecutor() since scipy.fftpack (which UliEngineering uses to do the hard math) unlocks the Python GIL.

Note that due to the need to do a lot of housekeeping tasks, simple_parallel_fft_reduce() is much slower than simple_serial_fft_reduce() if you have a dataset so small that parallelization is not effective. My initial recommendation is to consider using the parallel variant if the total execution time of the serial variant is larger than 0.5\,s

Install:

sudo pip3 install git+https://github.com/ulikoehler/UliEngineering.git