Programming languages

How to search pubmed entrez API with Python and filter results by metadata

If you want to apply more filters to search results of Pubmed than given in their web interface, you might want to use the entrez API.

The following example shows how you can sort alphabetically by the journal the articles originally appeared in.

I would recommend processing the data in the .json format.

import requests
import json

db = 'pubmed'
domain = 'https://www.ncbi.nlm.nih.gov/entrez/eutils'
nresults = 10
query = "depression"
retmode='json'

# standard query
queryLinkSearch = f'{domain}/esearch.fcgi?db={db}&retmax={nresults}&retmode={retmode}&term={query}'
response = requests.get(queryLinkSearch)
pubmedJson = response.json()

results = []

for paperId in pubmedJson["esearchresult"]["idlist"]:
    # metadata query
    queryLinkSummary = f'{domain}/esummary.fcgi?db={db}&id={paperId}&retmode={retmode}'
    results.append({'paperId': paperId, 'metadata': requests.get(queryLinkSummary).json()})
    
    # check the journalnames 
    # print(results[-1]["metadata"]["result"][paperId]["fulljournalname"])

resultsSorted = sorted(results, key=lambda x: x["metadata"]["result"][x["paperId"]]["fulljournalname"])

with open('resultsSorted.json', 'w') as f:
    json.dump(resultsSorted, f)
Posted by Joshua Simon in APIs, Python

How to set username & password in Paho-MQTT

Set username & password in Paho-MQTT using

client.username_pw_set("myusername", "aeNg8aibai0oiloo7xiad1iaju1uch")

You need to call that before calling connect()!

Example of how to connect with username & password:

client = mqtt.Client("mqtt-test") # client ID "mqtt-test"
client.on_connect = on_connect
client.on_message = on_message
client.username_pw_set("myusername", "aeNg8aibai0oiloo7xiad1iaju1uch")
client.connect('127.0.0.1', 1883)
client.loop_forever()  # Start networking daemon

 

Posted by Uli Köhler in MQTT, Python

How to fix Paho-MQTT result code 5

When you see result code 5 in paho-mqtt this means Unauthorized! Typically it means you don’t have the correct username and password set.

Set username & password using

client.username_pw_set("myusername", "aeNg8aibai0oiloo7xiad1iaju1uch")

Example of how to connect with username & password:

client = mqtt.Client("mqtt-test") # client ID "mqtt-test"
client.on_connect = on_connect
client.on_message = on_message
client.username_pw_set("myusername", "aeNg8aibai0oiloo7xiad1iaju1uch")
client.connect('127.0.0.1', 1883)
client.loop_forever()  # Start networking daemon

 

Posted by Uli Köhler in MQTT, Python

Python MQTT subscribe minimal example (Paho-MQTT)

#!/usr/bin/env python3
import paho.mqtt.client as mqtt

def on_connect(client, userdata, flags, rc):
    # This will be called once the client connects
    print(f"Connected with result code {rc}")
    # Subscribe here!
    client.subscribe("my-topic")

def on_message(client, userdata, msg):
    print(f"Message received [{msg.topic}]: {msg.payload}")

client = mqtt.Client("mqtt-test") # client ID "mqtt-test"
client.on_connect = on_connect
client.on_message = on_message
client.username_pw_set("myusername", "aeNg8aibai0oiloo7xiad1iaju1uch")
client.connect('127.0.0.1', 1883)
client.loop_forever()  # Start networking daemon

 

Posted by Uli Köhler in MQTT, Python

How to fix Python ModuleNotFoundError: No module named ‘paho’

Problem:

When running your Python script, you see an error message like

Traceback (most recent call last):
  File "test.py", line 2, in <module>
    import paho.mqtt.client as mqtt
ModuleNotFoundError: No module named 'paho'

Solution:

Install the paho-mqtt package using

pip3 install paho-mqtt

or

pip install paho-mqtt
Posted by Uli Köhler in MQTT, Python

FreeRTOS task queue minimal example

This is how you create and use a task queue in FreeRTOS:

Global declaration

Declare the structure of a task (I recommend to use a task type enum class in order to keep the flexibility of using multiple task types:

#include <freertos/queue.h>

enum class I2CTaskType : uint8_t {
    MyTaskType = 0
};

struct I2CTask {
    I2CTaskType type;
    // Parameters
    int16_t value;
};
static QueueHandle_t i2cTaskQueue;

Initialization code

Call this once, before using it:

// Create task queue
i2cTaskQueue = xQueueCreate(8 /* Number of queue slots */, sizeof(I2CTask));

In the thread processing the queue

if (xQueueReceive(i2cTaskQueue, (void *)&task, portMAX_DELAY /* Wait infinitely for new tasks */) == pdTRUE) {
    if(task.type == I2CTaskType::MyTaskType) {
        // TODO process task
        Serial.printf("My task type: %d\r\n", task.value);
    }
}

How to add a task to the queue

void AddTask(int16_t val) {
    I2CTask task;
    task.type = I2CTaskType::MyTaskType;
    task.value = val;
    xQueueSend(i2cTaskQueue, (void*)&task, 10 / portTICK_PERIOD_MS /* timeout */);
}

 

Posted by Uli Köhler in C/C++, Embedded, FreeRTOS, PlatformIO

How to fix Arduino error: ‘size_t’ has not been declared

Problem:

In Arduino, you see an error message like

src/main.cpp:7:48: error: 'size_t' has not been declared
void MyFunction(size_t size);

Solution:

Include stddef.h where size_t is declared:

#include <stddef.h>

Add this line to the top of the file where the error occured.

Posted by Uli Köhler in Arduino, C/C++, Embedded, PlatformIO

How to fix Arduino I2C Wire error: call of overloaded ‘begin(const int&, const int&, int)’ is ambiguous

Problem:

You are trying to call Wire.begin() for I2C using

Wire.begin(Pin_I2C_SDA, Pin_I2C_SCL, 400000);

but you see an error message like

src/MyI2C.cpp: In function 'void MyInitI2C()':
src/NyI2C.cpp:139:47: error: call of overloaded 'begin(const int&, const int&, int)' is ambiguous
     Wire.begin(Pin_I2C_SDA,Pin_I2C_SCL, 400000);
                                               ^
In file included from include/MyIO.hpp:2:0,
                 from src/MyI2C.cpp:2:
/home/uli/.platformio/packages/framework-arduinoespressif32/libraries/Wire/src/Wire.h:79:10: note: candidate: bool TwoWire::begin(int, int, uint32_t)
     bool begin(int sda=-1, int scl=-1, uint32_t frequency=0); // returns true, if successful init of i2c bus
          ^
/home/uli/.platformio/packages/framework-arduinoespressif32/libraries/Wire/src/Wire.h:80:10: note: candidate: bool TwoWire::begin(uint8_t, int, int, uint32_t)
     bool begin(uint8_t slaveAddr, int sda=-1, int scl=-1, uint32_t frequency=0);

Solution:

This happens with specific versions of the Arduino framework. For me it happened specifically when upgrading to arduino-esp32 version 2.0.1.

You need to explicitly cast the third argument (400000) to uint32_t in order to tell the compiler which of the two functions you want to call:

Wire.begin(Pin_I2C_SDA, Pin_I2C_SCL, 400000);
Posted by Uli Köhler in Arduino, C/C++, Embedded, PlatformIO

How to compute resistor voltage divider ratio using Python

Use UliEngineering to compute the ratio of the voltage divider:

from UliEngineering.Electronics.VoltageDivider import *

# This prints 0.9578544061302683
print(voltage_divider_ratio("2.2k", "50k"))
# In other words, the output voltage of a 2.2kOhm / 50kOhm voltage divider
# equals 0.9578544061302683 times the input voltage.

You can install UliEngineering using pip such as:

sudo pip3 install UliEngineering

 

Posted by Uli Köhler in Electronics, Python

Example of how to do std::enable_if<...> with multiple conditions

Simply use && like this:

std::enable_if<CONDITION1::value && CONDITION2::value, T>

Full example:

template<class T, class I>
constexpr typename std::enable_if<std::is_enum<T>::value && std::is_integral<I>::value, I>::type operator|(T lhs, I rhs) 
{
    return static_cast<typename std::underlying_type<T>::type>(lhs) | rhs;
}
Posted by Uli Köhler in C/C++

How to replace host part of IPv6 address using Python

In our previous post Bitwise operation with IPv6 addresses and networks in Python we showed how to perform bitwise operations in Python using the ipaddress module. In this post we will use this previous work to replace just the host part of an IPv6 address, leaving the network part as-is – in other words, we will combine two IPv6 addresses together using a configurable network prefix length.

import ipaddress

def bitwise_and_ipv6(addr1, addr2):
    result_int = int.from_bytes(addr1.packed, byteorder="big") & int.from_bytes(addr2.packed, byteorder="big")
    return ipaddress.IPv6Address(result_int.to_bytes(16, byteorder="big"))

def bitwise_or_ipv6(addr1, addr2):
    result_int = int.from_bytes(addr1.packed, byteorder="big") | int.from_bytes(addr2.packed, byteorder="big")
    return ipaddress.IPv6Address(result_int.to_bytes(16, byteorder="big"))

def bitwise_xor_ipv6(addr1, addr2):
    result_int = int.from_bytes(addr1.packed, byteorder="big") ^ int.from_bytes(addr2.packed, byteorder="big")
    return ipaddress.IPv6Address(result_int.to_bytes(16, byteorder="big"))

def replace_ipv6_host_part(net_addr, host_addr, netmask_length=64):
    # Compute bitmasks
    prefix_network = ipaddress.IPv6Network(f"::/{netmask_length}")
    hostmask = prefix_network.hostmask # ffff:ffff:ffff:ffff:: for /64
    netmask = prefix_network.netmask # ::ffff:ffff:ffff:ffff for /64
    # Compute address
    net_part = bitwise_and_ipv6(net_addr, netmask)
    host_part = bitwise_and_ipv6(host_addr, hostmask)
    # Put together resulting IP
    return bitwise_or_ipv6(net_part, host_part)

# Usage example:
# IP address from which we take the network part ("prefix")
net_addr = ipaddress.IPv6Address("2a01:c22:6f71:9f00:8ce6:2eff:fe60:cc69")
# IP address from which we take the host part (suffix)
host_addr = ipaddress.IPv6Address("::dead:babe:cafe:0000")
print(replace_ipv6_host_part(net_addr, host_addr))

This prints

IPv6Address('2a01:c22:6f71:9f00:dead:babe:cafe:0')

 

Posted by Uli Köhler in Networking, Python

How to add preprocessor build flags in PlatformIO

In order to define a preprocessor build flag in PlatformIO, add it to build_flags in platformio.ini, prefixing it with -D:

build_flags = -DQUICKSPI_DEBUG_WRITES

or, in order to define it to a specific value, use

build_flags = -DQUICKSPI_DEBUG_WRITES=1

 

 

Posted by Uli Köhler in C/C++, PlatformIO

How to add FreeRTOS task (“thread”) to any PlatformIO project

Most PlatformIO default configurations already have FreeRTOS enabled – they just don’t use it.

In order to start a new FreeRTOS “thread” (called task in FreeRTOS-speak), first add these includes:

#include <freertos/FreeRTOS.h>
#include <freertos/task.h>

Now add the task function and handle:

TaskHandle_t myTaskHandle;
void MyTask( void * parameter )
{
    for(;;)
    {
       // TODO Task code goes here
    }
    // if you ever exit the loop, this is here to clean up the resources
    vTaskDelete( NULL );
}

then start the task using this code once, for example in your main function:

// Start MyTask thread
xTaskCreate(
    MyTask, // Task function
    "MyTask", // Name
    10000, // Stack size
    NULL, // Parameter
    1, // Priority
    &myTaskHandle);

Also see our new post on how to use xTaskCreateStatic() to use statically allocated instead of dynamically allocated stack memory for the task: FreeRTOS task with static stack memory (xTaskCreateStatic) example

Posted by Uli Köhler in C/C++, Electronics, Embedded, FreeRTOS, PlatformIO

How to read 8-bit I2C register using Arduino Wire library: A minimal example

The following code demonstrates how to read a register that is 1 byte (8 bits) long over I2C. It will work with almost all I2C devices like EEPROMs, ADCs and others, provided you have the correct. Note that some devices like the LAN9303 have a slightly different addressing scheme or other peculiarities. In my opinion, it’s most efficient to just try out the standard way of reading a register and start from there.

Note that this code does not implement error handling for the sake of simplicity. Additionally, we wait for data using delay() instead of Wire.available(). This is a minimal example so it creates minimal confusion for the reader. We will provide a full example with error handling in a followup post.

const uint8_t SLAVE_I2C_ADDRESS = 0b1010;
const uint16_t SLAVE_I2C_REGISTER_ADDRESS = 0x50;

Wire.beginTransmission(SLAVE_I2C_ADDRESS);
Wire.write(SLAVE_I2C_REGISTER_ADDRESS);
Wire.endTransmission();
Wire.requestFrom(SLAVE_I2C_ADDRESS, 1); // This register is 8 bits = 1 byte long
delay(2); // Wait for data to be available
// Read directly into an uint8_t
uint8_t buf = (uint8_t)Wire.read();
// Print register value
Serial.printf("Register value: %02x\r\n", buf);

Also see:

Posted by Uli Köhler in Arduino, C/C++, Electronics, Embedded, PlatformIO

How to read 16-bit I2C register using Arduino Wire library: A minimal example

The following code demonstrates how to read a register that is 2 bytes (16 bits) long over I2C. It will work with almost all I2C devices like EEPROMs, ADCs and others, provided you have the correct. Note that some devices like the LAN9303 have a slightly different addressing scheme or other peculiarities. In my opinion, it’s most efficient to just try out the standard way of reading a register and start from there.

Note that this code does not implement error handling for the sake of simplicity. Additionally, we wait for data using delay() instead of Wire.available(). This is a minimal example so it creates minimal confusion for the reader. We will provide a full example with error handling in a followup post.

Option 1: Reading the register into an uint16_t (recommended)

const uint8_t SLAVE_I2C_ADDRESS = 0b1010;
const uint16_t SLAVE_I2C_REGISTER_ADDRESS = 0x50;

Wire.beginTransmission(SLAVE_I2C_ADDRESS);
Wire.write(SLAVE_I2C_REGISTER_ADDRESS);
Wire.endTransmission();
Wire.requestFrom(SLAVE_I2C_ADDRESS, 2); // This register is 16 bits = 2 bytes long
delay(5); // Wait for data to be available
// Read directly into an uint32_t
uint16_t buf;
Wire.readBytes((uint8_t*)&buf, 2);
// Print register value
Serial.printf("Register value: %04x\r\n", __builtin_bswap16(buf));

For an explanation on why we need __builtin_bswap16(), see How to print 16-bit uint16_t as four hex digits in Arduino

Option 2: Reading the register into an uint8_t array

const uint8_t SLAVE_I2C_ADDRESS = 0b1010;
const uint16_t SLAVE_I2C_REGISTER_ADDRESS = 0x50;

Wire.beginTransmission(SLAVE_I2C_ADDRESS);
Wire.write(SLAVE_I2C_REGISTER_ADDRESS);
Wire.endTransmission();
Wire.requestFrom(SLAVE_I2C_ADDRESS, 2); // This register is 16 bits = 2 bytes long
delay(5); // Wait for data to be available
// Read into a 2-byte buffer
uint8_t buf[2];
Wire.readBytes(buf, 2);
// Print register value
Serial.printf("Register value: %02x%02x\r\n", buf[0], buf[1]);

Also see:

Posted by Uli Köhler in Arduino, C/C++, Electronics, Embedded, PlatformIO

How to read 32-bit I2C register using Arduino Wire library: A minimal example

The following code demonstrates how to read a register that is 4 bytes (32 bits) long over I2C. It will work with almost all I2C devices like EEPROMs, ADCs and others, provided you have the correct. Note that some devices like the LAN9303 have a slightly different addressing scheme or other peculiarities. In my opinion, it’s most efficient to just try out the standard way of reading a register and start from there.

Note that this code does not implement error handling for the sake of simplicity. Additionally, we wait for data using delay() instead of Wire.available(). This is a minimal example so it creates minimal confusion for the reader. We will provide a full example with error handling in a followup post.

Option 1: Reading the register into an uint32_t (recommended)

const uint8_t SLAVE_I2C_ADDRESS = 0b1010;
const uint16_t SLAVE_I2C_REGISTER_ADDRESS = 0x50;

Wire.beginTransmission(SLAVE_I2C_ADDRESS);
Wire.write(SLAVE_I2C_REGISTER_ADDRESS);
Wire.endTransmission();
Wire.requestFrom(SLAVE_I2C_ADDRESS, 4); // This register is 32 bits = 4 bytes long
delay(5); // Wait for data to be available
// Read directly into an uint32_t
uint32_t buf;
size_t actually_read = Wire.readBytes((uint8_t*)&buf, 4);
// Print register value
Serial.printf("Register value: %08lx\r\n", __builtin_bswap32(buf));

For an explanation on why we need __builtin_bswap32(), see How to print 32-bit uint32_t as eight hex digits in Arduino

Option 2: Reading the register into an uint8_t array

const uint8_t SLAVE_I2C_ADDRESS = 0b1010;
const uint16_t SLAVE_I2C_REGISTER_ADDRESS = 0x50;

Wire.beginTransmission(SLAVE_I2C_ADDRESS);
Wire.write(SLAVE_I2C_REGISTER_ADDRESS);
Wire.endTransmission();
Wire.requestFrom(SLAVE_I2C_ADDRESS, 4); // This register is 32 bits = 4 bytes long
delay(5); // Wait for data to be available
// Read into a 4-byte buffer
uint8_t buf[4];
size_t actually_read = Wire.readBytes(buf, 4);
// Print register value
Serial.printf("Register value: %02x%02x%02x%02x\r\n", buf[0], buf[1], buf[2], buf[3]);

Also see:

Posted by Uli Köhler in Arduino, C/C++, Electronics, Embedded, PlatformIO

How to access LAN9303 registers over I2C using Arduino / PlatformIO

The LAN9303 has some peculiarities when accessing its registers. This post will not cover indirect register access but only access to the registers which are directly accessible over I2C. Note that the prerequisite for this is to configure the LAN9303 into a mode where management over the I2C slave interface is enabled. See How to get the LAN9303 into I2C managed mode for more info on how to do that.

The main point to take away is that you do not write the register offset from the datasheet (such as 0x50 for the Chip ID and revision register) in the I2C address byte but the address divided by 4 (0x50 >> 2 == 0x14). This is evidenced by figure 8-8 from the datasheet, copyright Microchip, listing the address byte as A[9:2] as opposed to the standard A[7:0]:

 

Example on how to access the register at offset 0x50 (Chip ID and revision register) in Arduino using the Wire library:

const uint8_t LAN9303_I2C_ADDRESS = 0b1010;
const uint16_t LAN9303_CHIPID_REV_Register = 0x50;

Wire.beginTransmission(LAN9303_I2C_ADDRESS);
Wire.write(LAN9303_CHIPID_REV_Register >> 2);
Wire.endTransmission();
Wire.requestFrom(LAN9303_I2C_ADDRESS, 4); // This register is 32 bits = 4 bytes long
delay(5); // Wait for data to be available

// Read directly into an uint8_t
uint32_t buf;
size_t actually_read = Wire.readBytes((uint8_t*)&buf, 4);
// Check if we have received all 4 bytes
if(actually_read != 4) {
    Serial.println("Did not read enough bytes");
}

// Print register value
Serial.printf("LAN9303 Chip ID and revision: %08lx\r\n", __builtin_bswap32(buf));

This will print

LAN9303 Chip ID and revision: 93030001

in other words: Chip ID = 0x9303, revision = 0x0001

 

Posted by Uli Köhler in Arduino, C/C++, Electronics, Embedded, Networking, PlatformIO

Teensy 4.x Quadrature Encoder minimal example in PlatformIO/Arduino

This example uses PlatformIO and the Teensy-4.x-Quad-Encoder-Library to implement a hardware quadrature encoder on pins 0 and 1. Remember that the teensy will be destroyed if you use 5V encoder signals. You can only use 3.3V signals!

#include <Arduino.h>
#include <QuadEncoder.h>

QuadEncoder encoder(1, 0, 1, 0);

void setup() {
    Serial.begin(115200);
    encoder.setInitConfig();
    encoder.init();
}

void loop() {
    int32_t position = encoder.read();
    // Compute position in mm and print it
    constexpr float mm_per_count = 0.0012;
    float mm = position * mm_per_count;
    Serial.printf("Position: %+3.4f (count %ld)\r\n", mm, position);
    // Print every 50ms
    delay(50);
}
[env:teensy41]
platform = teensy
board = teensy41
framework = arduino
monitor_speed = 115200
lib_deps =
    git+https://github.com/mjs513/Teensy-4.x-Quad-Encoder-Library

 

Posted by Uli Köhler in C/C++, Electronics, Embedded, PlatformIO, Teensy

How to use HAL_GPIO_Init() in modern C++ (STM32)

In modern C++, you can directly initialize structs like a GPIO_InitTypeDef, making the code much prettier and less prone to errors. The following example configures PA8 of a STM32 in alternate function 1 mode (TIM1 output).

GPIO_InitTypeDef pinInit = {
  .Pin = GPIO_PIN_8,
  .Mode = GPIO_MODE_AF_PP,
  .Pull = GPIO_NOPULL,
  .Speed = GPIO_SPEED_FREQ_VERY_HIGH,
  .Alternate = GPIO_AF1_TIM1
};
HAL_GPIO_Init(GPIOA, &pinInit);

As opposed to the old, much more verbose way of doing that:

GPIO_InitTypeDef pinInit;
pinInit.Pin = GPIO_PIN_8;
pinInit.Mode = GPIO_MODE_AF_PP;
pinInit.Pull = GPIO_NOPULL;
pinInit.Speed = GPIO_SPEED_FREQ_VERY_HIGH;
pinInit.Alternate = GPIO_AF1_TIM1;
HAL_GPIO_Init(GPIOA, &pinInit);

 

 

Posted by Uli Köhler in C/C++, Electronics, STM32