Programming languages

How to construct C++ chrono::time_point from a milliseconds-since-epoch timestamp

#include <chrono>

size_t myMillisecondsSinceEpochTimestamp = 1680978404374;
std::chrono::time_point<std::chrono::system_clock> myChronoTimestamp(
    std::chrono::milliseconds(myMillisecondsSinceEpochTimestamp)
);

 

Posted by Uli Köhler in C/C++

heatshrink compression library: How to compress data from a buffer

This is a starting point for how to compress data from a buffer using heatshrink and write the data into an output buffer:

Note that this has not been thoroughly tested, but at least it doesn’t crash 🙂

// TODO Your code goes here
const size_t outbuf_size = filesize + 128;
char* outbuf = new char[outbuf_size];
heatshrink_encoder* encoder = heatshrink_encoder_alloc(10, 4);
size_t tosink = filesize;
size_t output_size = 0;
while(tosink > 0) {
    size_t sunk = 0;
    auto err = heatshrink_encoder_sink(encoder, reinterpret_cast<uint8_t*>(buf), (size_t)filesize, &sunk);
    if(err != HSER_SINK_OK) {
        std::cerr << "Error sinking data" << std::endl;
        break;
    }
    if(sunk == 0) {
        std::cerr << "No data sunk" << std::endl;
        break;
    }
    // Check how much has been sunk & update tosink
    tosink -= sunk;
    // Poll for output
    size_t polled = 0;
    auto err2 = heatshrink_encoder_poll(encoder, reinterpret_cast<uint8_t*>(outbuf + output_size), outbuf_size - output_size, &polled);
    if(err2 == HSER_POLL_ERROR_NULL || err2 == HSER_POLL_ERROR_MISUSE) {
        std::cerr << "Error polling data" << std::endl;
        break;
    }
    output_size += polled;
}
// Input data finished
auto err3 = heatshrink_encoder_finish(encoder);
// Poll for final output
// Poll for output
size_t polled = 0;
auto err2 = heatshrink_encoder_poll(encoder, reinterpret_cast<uint8_t*>(outbuf + output_size), outbuf_size - output_size, &polled);
if(err2 == HSER_POLL_ERROR_NULL || err2 == HSER_POLL_ERROR_MISUSE) {
    std::cerr << "Error finally polling data" << std::endl;
}
output_size += polled;

cout << "Original size: " << filesize << ", compressed size: " << output_size << endl;

 

Posted by Uli Köhler in C/C++, Embedded

How to read C++ binary file into a buffer (C++17 way)

#include <iostream>
#include <fstream>
#include <filesystem>

// Get size of file to know how much memory to allocate
std::uintmax_t filesize = std::filesystem::file_size("C037B221110.bin");

// Allocate buffer to hold file
char* buf = new char[filesize];
// Read file
std::ifstream fin("C037B221110.bin", std::ios::binary);
fin.read(buf, filesize);
if(!fin) {
    std::cerr << "Error reading file, could only read " << fin.gcount() << " bytes" << std::endl;
}
// Close file
fin.close();

 

Posted by Uli Köhler in C/C++

How to get filesize in C++ using C++17’s std::filesystem

Recent versions of C++ have a built-in cross-platform filesystem library which supports many useful operations such as getting the filesize.

#include <filesystem>

// Usage example:
std::filesystem::file_size("myfile.bin");

Full example:

#include <filesystem>
#include <iostream>

int main() {
    std::uintmax_t filesize = std::filesystem::file_size("myfile.bin");
    std::cout << "Filesize in bytes is " << filesize << std::endl;
    return 0;
}

 

Posted by Uli Köhler in C/C++

How to install pip (Python) on pfSense

Once you have installed python on your pfSense, you might notice that it is missing pip:

[2.6.0-RELEASE][[email protected]]/root: pip
pip: Command not found.

and also python3.8 -m pip doesn’t work:

[2.6.0-RELEASE][[email protected]]/root: python3.8 -m pip
/usr/local/bin/python3.8: No module named pip

Installing it is rather easy, however:

python3.8 -m ensurepip

After that, you can run pip using

python3 -m pip

 

Posted by Uli Köhler in Networking, Python

How to install python3 on pfSense

First, login to your pfSense as root using ssh.

Then use

pkg search python

to show which python versions are available. Output on pfSense 2.6.0:

frr7-pythontools-7.5.1_3       Provide configuration reload functionality for FRR
py38-gitpython-3.1.24          Python Git Library
python38-3.8.12_1              Interpreted object-oriented programming language

Now run e.g.

pkg install python38-3.8.12_1

On my pfSense, python3.8 was already installed.

Remember that in order to run python, you need to explicitly run python3.8, just running python or python3 won’t work!

Posted by Uli Köhler in Networking, Python

How to fix Angular “ng build” not producing index.html

Unless you have errors in your build (which is clearly visible from looking at the ng build output), the reason why ng build doesn’t produce an index.html is that the resulting bundle exceeds the maximum allowed size.

This is evident from the output such as

Error: bundle initial exceeded maximum budget. Budget 1.00 MB was not met by 3.73 kB with a total of 1.00 MB.

In order to fix it, either shrink your application by removing unneeded features or libraries or splitting off certain features into extra modules or, at least temporarily, increase the allowable budget.

In order to do that, open angular.json and look for the "budgets": {... section:

"budgets": [
  {
    "type": "initial",
    "maximumWarning": "500kb",
    "maximumError": "1mb"
  },
  {
    "type": "anyComponentStyle",
    "maximumWarning": "2kb",
    "maximumError": "4kb"
  }
],

Increase the maximumError value for "type": "initial" (the first entry in the list shown above).

For example, you could increase it from 1mb to 4mb to fix your build.

After that, run your ng build command again and index.html should be generated.

Be aware, however, that having huge JS bundles can really slow down your web application especially for mobile users.

Posted by Uli Köhler in Angular

How to fix PyVISA ValueError: Please install PyUSB to use this resource type

Problem:

When trying to run a Python script using PyVISA with a USB instrument, you see the following error message:

ValueError: Please install PyUSB to use this resource type.
No module named 'usb'

Solution:

On Linux distributions such as Ubuntu, you can fix this using

sudo apt-get -y install python3-usb

This will install the Python3 usb package and the libusb system packages.

Posted by Uli Köhler in Python

Angular HttpClient: Observable with data continous data stream using interval requests to API endpoint

The following example requests /api/data every second, creating an Observable where you can subscribe to the continous value stream.

import { HttpClient } from '@angular/common/http';
import { Injectable } from '@angular/core';
import { interval, mergeMap, Observable } from 'rxjs';

@Injectable({
  providedIn: 'root'
})
export class DataService {

  constructor(private http: HttpClient) { }

  continousDataStream(_interval=1000): Observable<any> {
    return interval(_interval).pipe(mergeMap(_ => {
        return this.http.get<any>(`/api/data`);
    }));
  }
}

 

Posted by Uli Köhler in Angular

How to fix NodeJS Error: EXDEV: cross-device link not permitted, rename …

Problem:

While trying to rename a file using fs.move() in NodeJS, you see an error message such as

Error: EXDEV: cross-device link not permitted, rename '/tmp/upload_8a8c71784abf9942b40c1359935b1997' -> 'myfile.pdf'

Solution:

This error occurs because you are moving a file from one drive (/tmp in our example above) to another drive (the current directory in our example above). On most platforms, this can’t be done using a simple rename operation.

Instead, fs.copyFile() the file.

For example, instead of

await fs.move('/tmp/upload_8a8c71784abf9942b40c1359935b1997', 'myfile.pdf')

first, copy the file:

await fs.copyFile('/tmp/upload_8a8c71784abf9942b40c1359935b1997', 'myfile.pdf')

and then – if desired – remove the source file to mirror the behaviour of os.move() as closely as possible.

await fs.remove('/tmp/upload_8a8c71784abf9942b40c1359935b1997', 'myfile.pdf')
await fs.unlink('/tmp/upload_8a8c71784abf9942b40c1359935b1997')

 

Posted by Uli Köhler in NodeJS

How to get filesize in NodeJS / TypeScript using Promisese

First, import stat from the NodeJS standard library:

import { stat } from "node:fs/promises";

Async-await style

// Async-await style:
const statResult = await stat("myfile.pdf");
const fileSizeInBytes = statResult.size;

Promise.then() style

stat("myfile.pdf").then(statResult => {
    const fileSizeInBytes = statResult.size;
    // TODO your code goes here
});

 

Posted by Uli Köhler in NodeJS, Typescript

How to format axis as dB (decibel) using matplotlib

In our previous post Matplotlib custom SI-prefix unit tick formatter we showed how to format a matplotlib Y axis with a custom unit with SI prefixes.

Similarly, we can use UliEngineering.Math.Decibel in order to format plots (most notably plots with a logarithmic Y axis) as decibels:

from UliEngineering.Math.Decibel import *
import matplotlib.ticker as mtick

def decibel_formatter(v0=1.0, unit='dB'):
    def format_value(value, pos=None):
        dB = value_to_dB(value, v0=v0)
        return f'{dB:.0f} {unit}'
    return format_value

# Usage example:
plt.gca().set_yscale("log") # Optional

plt.gca().yaxis.set_major_formatter(mtick.FuncFormatter(decibel_formatter()))

 

Posted by Uli Köhler in Python

How to fix erpc ValueError: The generated shim code version … is different to the rest of eRPC code

Problem:

When trying to import your erpc project generated Python code using e.g.

from erpc_myproject import *

You see the following error message:

Traceback (most recent call last)
Cell In [1], line 1
----> 1 from erpc_myproject import *

File ./erpc_myproject/__init__.py:13
     11     version = "unknown"
     12 if version != "1.9.1":
---> 13     raise ValueError("The generated shim code version (1.9.1) is different to the rest of eRPC code (%s). \
     14 Install newer version by running \"python setup.py install\" in folder erpc/erpc_python/." % repr(version))
     16 from . import common
     17 from . import client

ValueError: The generated shim code version (1.9.1) is different to the rest of eRPC code ('unknown'). Install newer version by running "python setup.py install" in folder erpc/erpc_python/.

Solution:

Either you have not installed the erpc Python library (if the error message lists ... different to the rest of eRPC code ('unknown')) or you have installed the wrong version (e.g. ... (1.9.1) is different to the rest of eRPC code ('1.10.0')).

If you have not installed erpc at all, simply use

pip install erpc

and retry running your script.

If you have installed the wrong version, you have two options:

Option 1 (preferred): Re-generate the code

Just use the original command (some erpcgen call) you’ve used to re-generate the code using the currently installed version.

Option 2: Install the correct version

For this, you need to determine what the correct version is. Let’s consider the following error message:

ValueError: The generated shim code version (1.9.1) is different to the rest of eRPC code ('1.10.0'). Install newer version by running "python setup.py install" in folder erpc/erpc_python/.

From this message, we can read that the shim code version is 1.9.1 whereas you have 1.10.0 installed. Therefore, in order to make it work, we need to install erpc version 1.9.1.

Install it using

pip install -U erpc==1.9.1

and then retry your command. If you are using jupyter notebooks or similar, you need to restart your kernel to load the new library!

Posted by Uli Köhler in Embedded, Python

How to fix ESP32 error: ‘ESP_LOGE’ was not declared in this scope

Problem:

When trying compile your ESP32 project, you see an error message such as

.pio/libdeps/esp32dev/HumanESPHTTP/src/QueryURLParser.cpp:29:9: error: 'ESP_LOGE' was not declared in this scope
         ESP_LOGE("Query URL parser", "parsing URL");
         ^~~~~~~~

Solution:

At the top of the file where the error occurs (QueryURLParser.cpp in this example), add the following line:

#include <esp_log.h>

 

 

Posted by Uli Köhler in C/C++, ESP8266/ESP32

How to iterate all IP addresses in network using Python

The following code will iterate over all IP addresses in the given network, i.e. 192.168.1.0 ... 192.168.1.254:

import ipaddress

network = ipaddress.ip_network('192.168.1.0/24')

for ip in network:
    print(ip)

The following variant will iterate over all IP addresses in this network except the broadcast IP address 192.168.1.255 and the network IP address 192.168.1.0:

import ipaddress

network = ipaddress.ip_network('192.168.1.0/24')

for ip in network:
    # Ignore e.g. 192.168.1.0 and 192.168.1.255
    if ip == network.broadcast_address or ip == network.network_address:
        continue
    print(ip)

 

Posted by Uli Köhler in Python

How to fix pip fatal error: portaudio.h: No such file or directory

Problem:

While trying to install a package using pip, you see the following error message:

x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -fPIC -I/usr/local/include -I/usr/include -I/usr/include/python3.10 -c src/pyaudio/device_api.c -o build/temp.linux-x86_64-3.10/src/pyaudio/device_api.o
src/pyaudio/device_api.c:9:10: fatal error: portaudio.h: No such file or directory
    9 | #include "portaudio.h"
      |          ^~~~~~~~~~~~~
compilation terminated.
error: command '/usr/bin/x86_64-linux-gnu-gcc' failed with exit code 1
[end of output]

Solution:

On Ubuntu, you need to install the portaudio19-dev package:

sudo apt -y install portaudio19-dev

 

Posted by Uli Köhler in Python

How to fix ESP32 Last error reported from esp-tls: 0x8008

Problem:

While trying to use TLS such as MQTTS or HTTPS on the ESP32, you see an error message like

E (333183) MQTT_CLIENT: mqtt_message_receive: transport_read() error: errno=119
[328153][E][MyMQTT.cpp:80] log_error_if_nonzero(): [MQTT] Last error reported from esp-tls: 0x8008
E (333191) MQTT_CLIENT: mqtt_process_receive: mqtt_message_receive() returned -1

Solution:

0x8008 means  ESP_ERR_ESP_TLS_TCP_CLOSED_FIN. In other words, a TCP connection had been established successfully but unexpectedly, the connection has been closed by the server.

This is often caused by the server software crashing, or restarting in some way. When a server process is terminated, the operating system will cleanup after it and close all connections.

In order to debug the issue, start by checking the log of your server message and/or system log to check for unintended crashes. If that doesn’t help, it’s sometimes helpful to packet capture the communication between the ESP32 and the server. You can also write a software script doing the same communication with the server as the ESP32. This will often allow you to try out changes much more easily than on the microcontroller and observe what’s happening using a debugger.

Posted by Uli Köhler in C/C++, Embedded, ESP8266/ESP32, Networking

How to fix ESP32 Last error reported from esp-tls: 0x8001

Problem:

While trying to use TLS such as MQTTS or HTTPS on the ESP32, you see an error message like

[MQTT] Last error reported from esp-tls: 0x8001

Solution:

0x8001 means ESP_ERR_ESP_TLS_CANNOT_RESOLVE_HOSTNAME. In other words, the ESP32 is unable to resolve the hostname of the host you’re trying to connect to using DNS.

Typically, this is a DNS problem, so check the DNS settings of your network. Also check if the ESP32 has a correct DNS server set – for example, if the ESP32 has 0.0.0.0 as a DNS server, this explains why it isn’t able to resolve the hostname.

Sometimes this issue is also caused by the hostname not existing at all (i.e. there is no DNS entry for that hostname). You can easily check this by resolving the hostname you’re trying to connect

Posted by Uli Köhler in C/C++, Embedded, ESP8266/ESP32, Networking

How I fixed WordPress category page title: Category <span>…</span>

One one of my WordPress sites, the title of category pages was literally

Category <span>My category</span>

Searching through the theme source code didn’t yield a solution quickly, so I added this custom JS using a plugin:

jQuery(document).ready(function( $ ){
    if(jQuery("h1.page-title").text().includes("Category: <span>")) {
        var input = jQuery("h1.page-title").text();
        var output = /<span>(.*)<\/span>/g.exec(input)[1];
        jQuery("h1.page-title").text(output);
    }
});

This will just replace the title by the content of the span tag. You might need to adjust the CSS classes for your theme.

Posted by Uli Köhler in Javascript, Wordpress

How to compute voltage divider output voltage using Python

You can use the UliEngineering library to compute the output voltage of a voltage divider by its top resistor, bottom resistor and its input voltage:

from UliEngineering.Electronics.VoltageDivider import *

# Voltage divider with 1MΩ and 150kΩ resistors & 24V input voltage
print(voltage_divider_voltage("1MΩ", "150kΩ", "24V")) # Print 3.1304347826086953

You can also directly format this voltage using auto_format():

from UliEngineering.EngineerIO import auto_format

# Print formatted as volts
print(auto_format(voltage_divider_voltage, "1MΩ", "150kΩ", "24V")) # Prints "3.13 V"
Posted by Uli Köhler in Electronics, Python