Programming languages

PySerial minimal example: Copy data received from serial port to stdout

This example does not send data to the serial port but only copies data received from the serial port to stdout. Newlines received from the serial port are preserved.

#!/usr/bin/env python3
import serial
ser = serial.Serial("/dev/ttyUSB0", baudrate=115200)

try:
    while True:
        response = ser.read()
        if response:
            print(response.decode("iso-8859-1"), end="")
finally:
    ser.close()

By using iso-8859-1   decoding, we ensure that even binary bytes are decoded in some way and do not cause an exception.

Posted by Uli Köhler in 3D printing, Embedded, Python

Coyping a folder and renaming file extensions in Python

This script copies myfolder to myfolder2 recursively and renames all .txt files to .xml

#!/usr/bin/env python3
import os
import os.path
import re
import shutil

srcfolder = "myfolder"
dstfolder = "myfolder2"

for subdir, dirs, files in os.walk(srcfolder):
    for file in files:
        # Rename .txt to .xml
        if file.endswith(".txt"):
            dstfile_name = re.sub(r"\.txt$", ".xml", file) # Regex to replace .txt only at the end of the string
        else:
            dstfile_name = file
        # Compute destination path
        srcpath = os.path.join(subdir, file)
        dstpath_relative_to_outdir = os.path.relpath(os.path.join(subdir, dstfile_name), start=srcfolder)
        dstpath = os.path.join(dstfolder, dstpath_relative_to_outdir)
        # Create directory if not exists
        os.makedirs(os.path.dirname(dstpath), exist_ok=True)
        # Copy file (copy2 preserves metadata)
        shutil.copy2(srcpath, dstpath)

 

Posted by Uli Köhler in Python

How to zip folder recursively using Python (zipfile)

This script will zip the folder myfolder recursively to myzip. Note that empty directories will not be copied over to the ZIP.

#!/usr/bin/env python3
import zipfile
import os

# This folder 
folder = "myfolder"

with zipfile.ZipFile(f"{folder}.zip", "w") as outzip:
    for subdir, dirs, files in os.walk(folder):
        for file in files:
            # Read file
            srcpath = os.path.join(subdir, file)
            dstpath_in_zip = os.path.relpath(srcpath, start=folder)
            with open(srcpath, 'rb') as infile:
                # Write to zip
                outzip.writestr(dstpath_in_zip, infile.read())

 

Posted by Uli Köhler in Python

How to Serial.println() a std::string

In Arduino, if you have a std::string:

std::string str = "test";

you can’t directly print it – trying to do so leads to the following error messages:

src/main.cpp: In function 'void setup()':
src/main.cpp:122:22: error: no matching function for call to 'HardwareSerial::println(std::__cxx11::string&)'
   Serial.println(cert);
                      ^
In file included from /home/uli/.platformio/packages/[email protected]2/cores/esp32/Stream.h:26,
                 from /home/uli/.platformio/packages/[email protected]2/cores/esp32/Arduino.h:166,
                 from src/main.cpp:1:
/home/uli/.platformio/packages/[email protected]2/cores/esp32/Print.h:96:12: note: candidate: 'size_t Print::println(const __FlashStringHelper*)'
     size_t println(const __FlashStringHelper *);
            ^~~~~~~
/home/uli/.platformio/packages/[email protected]2/cores/esp32/Print.h:96:12: note:   no known conversion for argument 1 from 'std::__cxx11::string' {aka 'std::__cxx11::basic_string<char>'} to 'const __FlashStringHelper*'
/home/uli/.platformio/packages/[email protected]2/cores/esp32/Print.h:97:12: note: candidate: 'size_t Print::println(const String&)'
     size_t println(const String &s);

 

Solution:

You can use .c_str() to convert it to a NUL-terminated char* which can be printed directly:

Serial.println(str.c_str());

 

Posted by Uli Köhler in Arduino, C/C++, Embedded

How to convert Arduino String to std::string

If you have an Arduino String:

String arduinoStr = "test123";

you can easily convert it to a std::string by using:

std::string stdStr(arduinoStr.c_str(), arduinoStr.length());

The std::string constructor will copy the data, therefore you can de-allocate the Arduino String instance safely.

Posted by Uli Köhler in Arduino, C/C++

How to get filesize in LittleFS (ESP32/PlatformIO)

After you have initialized LittleFS (see ESP32 Filesystem initialization code example (LittleFS)) you can get the filesize by first opening the file, and then calling .size() on the opened file. Don’t forget to close the file afterwards.

auto file = LittleFS.open(filename, "r");
size_t filesize = file.size();
// Don't forget to clean up!
file.close();

Utility function to get the size of a file stored on LittleFS:

size_t LittleFSFilesize(const char* filename) {
  auto file = LittleFS.open(filename, "r");
  size_t filesize = file.size();
  // Don't forget to clean up!
  file.close();
  return filesize;
}

Example usage:

Serial.println(LittleFSFilesize("/cert.pem"));

 

Posted by Uli Köhler in C/C++, ESP8266/ESP32, PlatformIO

How to fix GCC elaborated-type-specifier for a scoped enum must not use the ‘class’ keyword

Problem:

In C++, you are declaring an enum class like this:

enum class Shape : uint8_t {
    Circle = 0,
    Square = 1
};

but when you try to compile the project, you see an error message like

include/Shape.hpp:4:6: warning: elaborated-type-specifier for a scoped enum must not use the 'class' keyword
 enum class Shape : uint8_t {
 ~~~~ ^~~~~
      -----

Solution:

The issue here is that you are deriving the enum class from another type –  uint8_t in this example –  but that type has not been declared.

So the solution is to #include the header where the type from which the enum class is inherited.

In our example – for the type uint8_t and for many other integral types such as int32_t , you can do that by adding

#include <cstdint>

before the enum class declaration.

Posted by Uli Köhler in C/C++, GCC errors

Python Lock / Mutex minimal example (threading.Lock)

This minimal example creates a threading.Lock and locks it using a with statement. This is the simplest way of using a Lock.

from threading import Lock

lock = Lock()

with lock:
    print("Holding the lock!")

 

Posted by Uli Köhler in Python

How to generate filename with date & time in Python

When storing real-time data from a Python script, it is often helpful to have a date & time in your filename, such as

mydata-2022-09-02_00-31-50-613015.csv

With this specific syntax we avoid special characters which are an issue on Windows operating systems, and we provide a lexically sortable filename

In Python you can do that using UliEngineering.Utils.generate_datetime_filename() from the UliEngineering library.

First, install the UliEngineering library using

pip install UliEngineering

Now you can generate your filename using

from UliEngineering.Utils.Date import *

filename = generate_datetime_filename()
# example: filename == 'data-2022-09-02_03-02-00-045587.csv'

or you can just open the file using with open():

with open(generate_datetime_filename(), "w") as outfile:
    # Example of what you can do with outfile
    outfile.write("test")

Without using UliEngineering

You can use this simplified version which does not support fractional seconds and will generate filenames like

data-2022-09-02_00-31-50.csv

Source code:

from datetime import datetime

def generate_datetime_filename(label="data", extension="csv", dt=None):
    if dt is None:
        dt = datetime.now()
    return f"{label}-{dt.year}-{dt.month:02d}-{dt.day:02d}_{dt.hour:02d}-{dt.minute:02d}-{dt.second:02d}.{extension}"
Posted by Uli Köhler in Python

Python minimal thread (threading.Thread) example

This is a really simple example of how to add an extra thread to your Python script, to run some task in the background:

from threading import Thread
import time

def extra_thread_function():
    while True:
        print("extra_thread is running")
        time.sleep(1)

extra_thread = threading.Thread(target=extra_thread_function)
extra_thread.start()

# TODO Your code for the main thread goes here!

# OPTIONAL: Wait for extra_thread to finish
extra_thread.join()

 

Posted by Uli Köhler in Python

Python minimal enum (enum.Enum) example

from enum import Enum

class Shape(Enum):
    Circle = 0
    Square = 1
    Hexagon = 2

# Usage example
print(Shape.Square) # prints Shape.Square

Based on the official enum package docs.

Posted by Uli Köhler in Python

How to fix STM32 error: expected constructor, destructor, or type conversion before __declspec(dllexport)

Problem:

When trying to compile a firmware for an STM32 microcontroller, you see a compiler error message like

myheader.h:23:12: error: expected constructor, destructor, or type conversion before ‘(’ token
   23 |  __declspec(dllexport) int myfunc(

Solution:

We previously explored the same problem for Linux platforms.

__declspec(dllexport) is a Windows-specific feature and not available on non-Windows platform such as the ARM embedded API platform (e.g. STM32). In order to fix it in a compatible way with both Windows and the STM32, add the following code either in a header that is included in every file containing __declspec(dllexport) or add it in each file where the error occurs:

#ifdef __ARM_EABI__
#define __declspec(v)
#endif

This will basically ignore any __declspec() call on the preprocessor level.

By using __ARM_EABI__ specfically, the definition will not trigger for ARM platforms for Windows.

Posted by Uli Köhler in C/C++, GCC errors, STM32

How to always get latest frame from OpenCV VideoCapture in Python

When working with OpenCV video capture, but when you only occasionally use images, you will get older images from the capture buffer.

This code example solves this issue by running a separate capture thread that continually saves images to a temporary buffer.

Therefore, you can always get the latest image from the buffer. The code is based on our basic example How to take a webcam picture using OpenCV in Python

video_capture = cv2.VideoCapture(0)

video_capture.set(cv2.CAP_PROP_FRAME_WIDTH, 1920)
video_capture.set(cv2.CAP_PROP_FRAME_HEIGHT, 1080)

if not video_capture.isOpened():
    raise Exception("Could not open video device")

class TakeCameraLatestPictureThread(threading.Thread):
    def __init__(self, camera):
        self.camera = camera
        self.frame = None
        super().__init__()
        # Start thread
        self.start()

    def run(self):
        while True:
            ret, self.frame = self.camera.read()

latest_picture = TakeCameraLatestPictureThread(video_capture)

Usage example:

# Convert latest image to the correct colorspace
rgb_img = cv2.cvtColor(latest_picture.frame, cv2.COLOR_BGR2RGB)
# Show
plt.imshow(rgb_img)

 

 

Posted by Uli Köhler in Audio/Video, OpenCV, Python

How to read ESP32 NVS value into std::string

In our previous post, we discussed How to get the length / size of NVS value on ESP32. Based on that, we can read a NVS value into a std::string.

Strategy

  1. Determine size of value in NVS
  2. Allocate temporary buffer of the determined size
  3. Read value from NVS into temporary buffer
  4. Create std::string from value
  5. Cleanup temporary buffer

Utility function to read NVS value as std::string

In case the key does not exist in NVS, this function will return the empty string ("").

#include <nvs.h>
#include <string>

std::string ReadNVSValueAsStdString(nvs_handle_t nvs, const char* key) {
    /**
     * Strategy:
     *  1. Determine size of value in NVS
     *  2. Allocate temporary buffer of determined size
     *  3. Read value from NVS into temporary buffer
     *  4. Create std::string from value
     *  5. Cleanup
     */
    // Step 1: Get size of key
    esp_err_t err;
    size_t value_size = 0;
    if((err = nvs_get_str(nvs, _key.c_str(), nullptr, &value_size)) != ESP_OK) {
        if(err == ESP_ERR_NVS_NOT_FOUND) {
            // Not found, no error
            return "";
        } else {
            printf("Failed to get size of NVS key %s: %s\r\n", key, esp_err_to_name(err));
            return;
        }
    }
    // Step 2: Allocate temporary buffer to read from
    char* buf = (char*)malloc(value_size);
    // Step 3: Read value into temporary buffer.
    esp_err_t err;
    if((err = nvs_get_str(nvs, _key.c_str(), buf, &value_size)) != ESP_OK) {
        // "Doesn't exist" has already been handled before, so this is an actual error.
        // We assume that the value did not change between reading the size (step 1) and now.
        // In case that assumption is value, this will fail with ESP_ERR_NVS_INVALID_LENGTH.
        // This is extremely unlikely in all usage scenarios, however.
        printf("Failed to read NVS key %s: %s\r\n", key, esp_err_to_name(err));
        free(buf);
        return "";
    }
    // Step 4: Make string
    std::string value = std::string(buf, value_size);
    // Step 5: cleanup
    free(buf);
    
    return value;
}

Usage example

This assumes that you have setup myNvs as we have shown in our previous post How to initialize NVS on ESP32

std::string value = ReadNVSValueAsStdString(myNvs, "MyKey");

C++17 optimizations

Starting from C++17, you can possibly create a std::string directly instead of using the temporary buffer, since there is an overload of .data() that returns a non-const pointer – so you can write directly to the std::string‘s buffer.

However, since my PlatformIO-based toolchain currently doesn’t support that, I have not written that code yet.

Posted by Uli Köhler in C/C++, ESP8266/ESP32

How to fix C/C++ ‘size_t’ does not name a type

Problem:

When compiling your C or C++ program. you see the following error message:

src/main.cpp:4:11: error: 'size_t' does not name a type
 size_t mySize = 32;

Solution:

In the file where the error occurs (in our example, that would be src/main.cpp), add the following line at or near the top. It must be added before the line where size_t is first used, and usually you would add that line after the last existing #include statements:

#include <stddef.h>

After adding that line, the error should be gone. Possibly you need to add said line to other files as well, so carefully check any compiler error messages if the error re-appears in other files.

Posted by Uli Köhler in C/C++, GCC errors

How to link Angular project dist directory to PlatformIO SPIFFS data directory

For tips how to make the Angular build small enough to fit into the SPIFFS image, see How to make Angular work with ESP32 SPIFFS / ESPAsyncWebserver

When you are building a PlatformIO image, you can easily make the dist/[project_name] directory from the Angular project directory appear in the SPIFFS image by using a symlink.

My config tells the server to serve from the www subdirectory.

server.serveStatic("/", SPIFFS, "/www/").setDefaultFile("index.html");

Therefore, we first need to create the data directory in the same directory where platformio.ini is located:

mkdir data

Now we can create a symlink from the angular dist directory to data/www, for example:

ln -s ../MyUI/dist/myui data/www

PlatformIO will automatically handle the symlink, if the directory exists.

Posted by Uli Köhler in Angular, ESP8266/ESP32, PlatformIO

How to make Angular work with ESP32 SPIFFS / ESPAsyncWebserver

The main issue when using Angular web UIs is that the resulting files get too large, hence building the filesystem image will fail with SPIFFS_write error(-10001): File system is full.

Using these tips, I could get an Angular PrimeNG app to fit into a 4MB flash ESP32 module without any custom partition table and without any other crazy hacks! Even the fonts & PrimeNG icons fit into the SPIFFS easily, with a total of only 380 kB of the approximately 1.5 MB being consumed.

File compression

The number one most important tip is that you can just gzip -9 the files from the angular dist directory and ESPAsyncWebserver will automatically handle decompressing them!

This is my platformio.ini:

[env:esp32dev]
platform = espressif32
platform_packages = framework-arduinoespressif32 @ https://github.com/espressif/arduino-esp32.git#2.0.3
board = esp32dev
framework = arduino
board_build.filesystem = littlefs
lib_deps =
    esphome/[email protected]^1.2.2
    esphome/[email protected]^2.1.0
    [email protected]
upload_speed = 460800
monitor_speed = 115200

This is my angular build script:

#!/bin/sh
ng build --aot --build-optimizer --optimization --progress --output-hashing none
gzip -9 dist/**/*

This is where I tell ESPAsyncWebserver (note that you should use the esphome fork) to serve files statically:

server.serveStatic("/", SPIFFS, "/www/").setDefaultFile("index.html");

Other tips

In order to make your life easier managing the data directory with both Angular files and other files, see How to link Angular project dist directory to PlatformIO SPIFFS data directory

You can use purgecss but compression works so well that it isn’t really worth both the risk of accidentally removing some CSS rules which you manually need to whitelist. Before discovering how well compression worked, I started to manually remove CSS rules from the PrimeNG theme file. This worked fine, but the SPIFFS still wasn’t small enough.

Often you can save space by deleting.

For example, primeicons.svg and primeicons.ttf are two different formats with the same content. Note that some (especially older, and some mobile) browsers don’t support all formats, hence it’s rather risky to remove them if you need to support multiple platforms.

Posted by Uli Köhler in Angular, ESP8266/ESP32, PlatformIO

How to remove hash from Angular “ng build” filenames

Angular generates filenames like Inter-Light.27083fa6375bb9ef.woff2 in the dist folder when building for production using ng build.

These hashes have the purpose of preventing the files from being cached, so if you remove the hash, you will need to find some other way of preventing caching

You can disable the hashes by using

--output-hashing none

as an argument to ng build.

Full ng build example:

ng build --aot --build-optimizer --optimization --progress --output-hashing none

 

Posted by Uli Köhler in Angular

How to use optional type in Angular / TypeScript

If you have a variable called ws of type WebSocket, you can not assign null or undefined to that variable

In order to make a variable that is either of the given type or undefined, use the following syntax:

ws?: WebSocket = undefined;

 

Posted by Uli Köhler in Angular, Javascript

How to turn DL3021 off via SCPI/LXI

In order to turn off the DL3021 electronic load (as in: disable any constant current/resistance/voltage/power function) via LXI, run the following command:

:SOURCE:INPUT:STATE Off

PyVISA example

For more info, see PyVISA Rigol DL3021 via LXI (TCP SCPI) example

#!/usr/bin/env python3

rm = pyvisa.ResourceManager()
inst = rm.open_resource("TCPIP0::192.168.178.112::INSTR")
# Query if instrument is present
# Prints e.g. "RIGOL TECHNOLOGIES,DL3021,DL3A204800938,00.01.05.00.01"
print(inst.query("*IDN?"))

# Turn electronic load off
inst.write(":SOURCE:INPUT:STATE Off")

 

Posted by Uli Köhler in Analog, Electronics, Networking, Python