Programming languages

Implementing STM32 DFU bootloader firmware upgrade in Marlin using M997

Marlin implements the M997 command which is intended to switch the mainboard into a firmware upgrade mode.

All STM32 variants have an integrated (hard-coded – so no need to flash it yourself) bootloader that is notoriously difficult to active.

However, Marlin implements M997 on the STM32 as just a reboot:

void flashFirmware(const int16_t) { HAL_reboot(); }

This only works if you are using a custom bootloader on your board – however, it does not use the STM32 integrated bootloader.

The following code was tested on the STM32F446 (BigTreeTech Octopus V1) but should work on any STM32 variant. It is based on previous work by Dave Hyland on StackOverflow. Replace the default flashFirmware() function in Marlin/src/HAL/STM32/HAL.cpp

void flashFirmware(const int16_t) {
    HAL_RCC_DeInit();
    HAL_DeInit();

    __HAL_REMAPMEMORY_SYSTEMFLASH();

    // arm-none-eabi-gcc 4.9.0 does not correctly inline this
    // MSP function, so we write it out explicitly here.
    //__set_MSP(*((uint32_t*) 0x00000000));
    __ASM volatile ("movs r3, #0\nldr r3, [r3, #0]\nMSR msp, r3\n" : : : "r3", "sp");

    ((void (*)(void)) *((uint32_t*) 0x00000004))();

    // This will never be executed
    HAL_reboot();
}

Note that we left a HAL_reboot() call as a safeguard at the end, just in case the previous calls fail.

On my BigTreeTech Octopus V1 (STM32F446), by using this code, you can successfully enter the integrated DFU bootloader.

Also see our previous posts on how to use the STM32 in DFU bootloader mode:

Posted by Uli Köhler in 3D printing, C/C++, STM32

How to install NodeJS 16.x LTS on Ubuntu in 1 minute

Run these shell commands on your Ubuntu computer to install NodeJS 14.x:

curl -sL https://deb.nodesource.com/setup_16.x | sudo -E bash -
sudo apt-get install -y nodejs

Instead of setup_16.x you can also choose other versions like setup_14.x. However, using this method, you can’t install multiple versions of NodeJS in parallel.

Source: Official nodesource documentation

Posted by Uli Köhler in Linux, NodeJS

How to fix python ipaddress.IPv6Address AddressValueError: Unexpected ‘/’ in ‘…/64’

Problem:

When trying to parse an IPv6 network address in Python using code like

import ipaddress
ipaddress.IPv6Network("2a01:c23:c0bb:d00:8ce6:2eff:fe60:cc69/64")

you see an error message like

---------------------------------------------------------------------------
AddressValueError                         Traceback (most recent call last)
/tmp/ipykernel_154945/2602627019.py in <module>
----> 1 ipaddress.IPv6Address("2a01:c23:c0bb:d00:8ce6:2eff:fe60:cc69/64")

/usr/lib/python3.8/ipaddress.py in __init__(self, address)
   1836         addr_str = str(address)
   1837         if '/' in addr_str:
-> 1838             raise AddressValueError("Unexpected '/' in %r" % address)
   1839         self._ip = self._ip_int_from_string(addr_str)
   1840 

AddressValueError: Unexpected '/' in '2a01:c23:c0bb:d00:8ce6:2eff:fe60:cc69/64'

Solution 1: Remove the CIDR netmask (/64)

By just removing the slash and the part after it (the CIDR netmask).

Solution 1: Maybe you should use IPv6Network instead of IPv6Address

If you intend to parse the network, use ipaddress.IPv6Network but remember that this will discard all host bits. If you want to use IPv6Address or IPv6Network really depends on what you want to do with it – if you want to refer to a specific host, you almost always want to use IPv6Address.

import ipaddress
ipaddress.IPv6Network("2a01:c23:c0bb:d00:8ce6:2eff:fe60:cc69/64", strict=False)

Note that strict=False is added in order to prevent an exception due to host bits being set – see How to fix Python ipaddress.IPv6Network ValueError: … has host bits set

Posted by Uli Köhler in Networking, Python

How to fix Python ipaddress.IPv6Network ValueError: … has host bits set

Problem:

When trying to parse an IPv6 network address in Python using code like

import ipaddress
ipaddress.IPv6Network("2a01:c23:c0bb:d00:8ce6:2eff:fe60:cc69/64")

you see an error message like

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
/tmp/ipykernel_154945/1312927855.py in <module>
      1 import ipaddress
----> 2 ipaddress.IPv6Network("2a01:c23:c0bb:d00:8ce6:2eff:fe60:cc69/64")

/usr/lib/python3.8/ipaddress.py in __init__(self, address, strict)
   2106         if packed & int(self.netmask) != packed:
   2107             if strict:
-> 2108                 raise ValueError('%s has host bits set' % self)
   2109             else:
   2110                 self.network_address = IPv6Address(packed &

ValueError: 2a01:c23:c0bb:d00:8ce6:2eff:fe60:cc69/64 has host bits set

Solution 1: Maybe you should use IPv6Address instead of IPv6Network

If you intend to parse the address and don’t care about the network, use ipaddress.IPv6Address but remember that the CIDR mask (e.g. /64)  needs to be removed. If you want to use IPv6Address or IPv6Network really depends on what you want to do with it – if you want to refer to a specific host, you almost always want to use IPv6Address.

import ipaddress
ipaddress.IPv6Address("2a01:c23:c0bb:d00:8ce6:2eff:fe60:cc69")

Solution 2: Use strict=False to let IPv6Network discard the host bits:

strict=False basically ignores this error

import ipaddress
ipaddress.IPv6Network("2a01:c23:c0bb:d00:8ce6:2eff:fe60:cc69/64", strict=False)

Note that the result will be

IPv6Network('2a01:c23:c0bb:d00::/64')

so the information in the host bits will be lost!

Solution 2: Remove the host bits manually

Since the host bits will be discarded anyway, you can just specify the IPv6 network with the correct netmask:

import ipaddress
ipaddress.IPv6Network("2a01:c23:c0bb:d00::/64")

 

Posted by Uli Köhler in Networking, Python

How to fix Pronterface/Printrun UnicodeDecodeError: ‘utf-8’ codec can’t decode byte … in position …: invalid continuation byte

Problem:

When starting Pronterface / Printrun on Linux using pronterfaceprintrunpython3 pronterface.py or similar, you see an error message like

Traceback (most recent call last):
  File "pronterface.py", line 62, in <module>
    app = PronterApp(False)
  File "/home/uli/dev/tools/Printrun/printrun/pronterface.py", line 2464, in __init__
    self.mainwindow = PronterWindow(self)
  File "/home/uli/dev/tools/Printrun/printrun/pronterface.py", line 170, in __init__
    self.parse_cmdline(sys.argv[1:])
  File "/home/uli/dev/tools/Printrun/printrun/pronsole.py", line 786, in parse_cmdline
    self.process_cmdline_arguments(args)
  File "/home/uli/dev/tools/Printrun/printrun/pronterface.py", line 1031, in process_cmdline_arguments
    pronsole.pronsole.process_cmdline_arguments(self, args)
  File "/home/uli/dev/tools/Printrun/printrun/pronsole.py", line 769, in process_cmdline_arguments
    self.load_default_rc()
  File "/home/uli/dev/tools/Printrun/printrun/pronsole.py", line 664, in load_default_rc
    self.load_rc(config)
  File "/home/uli/dev/tools/Printrun/printrun/pronsole.py", line 632, in load_rc
    for rc_cmd in rc:
  File "/usr/lib/python3.8/codecs.py", line 714, in __next__
    return next(self.reader)
  File "/usr/lib/python3.8/codecs.py", line 645, in __next__
    line = self.readline()
  File "/usr/lib/python3.8/codecs.py", line 558, in readline
    data = self.read(readsize, firstline=True)
  File "/usr/lib/python3.8/codecs.py", line 504, in read
    newchars, decodedbytes = self.decode(data, self.errors)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd3 in position 4: invalid continuation byte

Solution

When you see this error message, typically your pronsolerc is corrupted. In order to remove it,

rm -rf ~/.config/Printrun
Posted by Uli Köhler in 3D printing, Python

How to center-align all columns in pandas XLSX export

In our previous post How to center-align column in pandas XLSX export we showed how to center-align one specific column in a XLSX export.

In this post, we’ll show how to center all columns in a pandas DataFrame XLSX export. Note that the variants with and without index shown here differ only in the column index computation. Pandas automatically centers the index column, so we don’t need to do that explicitly.

For export with index

# Center all columns
for column_idx in range(len(df.columns)):
    for cell in sheet[openpyxl.utils.cell.get_column_letter(column_idx + 1)]:
        cell.alignment = Alignment(horizontal="center")

For export without index

# Center all columns
for column_idx in range(len(df.columns)):
    for cell in sheet[openpyxl.utils.cell.get_column_letter(column_idx + 1)]:
        cell.alignment = Alignment(horizontal="center")

Full example:

import pandas as pd
import openpyxl.utils.cell
from openpyxl.styles.alignment import Alignment

df = pd.DataFrame([
    {"a": 1.0},
    {"a": 2.0},
    {"a": 3.0},
    {"a": 4.0},
    {"a": 5.0},
])
with pd.ExcelWriter("out.xlsx", engine="openpyxl") as writer:
    sheet_name = "Bool"
    # Export DataFrame content
    df.to_excel(writer, sheet_name=sheet_name)
    # Align to center
    sheet = writer.sheets[sheet_name]
    # Center all columns
    for column_idx in range(len(df.columns) + 1):
        for cell in sheet[openpyxl.utils.cell.get_column_letter(column_idx + 1)]:
            cell.alignment = Alignment(horizontal="center")

The table will look like this with the centering enabled:

whereas the table is right-aligned with only the title centered by Pandas with the centering code disabled: – Note that the index column is centered automatica

Posted by Uli Köhler in OpenPyXL, pandas, Python

How to center-align column in pandas XLSX export

Note: Want to align all columns in the export? See our followup post How to center-align all columns in pandas XLSX export

When exporting a DataFrame in pandas, often you want to center a column in order to make the resulting table visually more appealing.

In this post we’ll show how to do this using the openpyxl engine.

If you want to center column B, add this code within the pd.ExcelWriter with: block:

for cell in sheet["B"]:
    cell.alignment = Alignment(horizontal="center")

Full example:

import pandas as pd
from openpyxl.styles.alignment import Alignment

df = pd.DataFrame([
    {"a": 1.0},
    {"a": 2.0},
    {"a": 3.0},
    {"a": 4.0},
    {"a": 5.0},
])
with pd.ExcelWriter("out.xlsx", engine="openpyxl") as writer:
    sheet_name = "Bool"
    # Export DataFrame content
    df.to_excel(writer, sheet_name=sheet_name)
    # Align to center
    sheet = writer.sheets[sheet_name]
    # Align every cell in column "B" horizontally
    for cell in sheet["B"]:
        cell.alignment = Alignment(horizontal="center")

The table will look like this with the centering enabled:

whereas the table is right-aligned with only the title centered by Pandas with the centering code disabled:

Posted by Uli Köhler in OpenPyXL, pandas, Python

How center text horizontally using Alignment in OpenPyXL

Based on our previous OpenPyXL minimal XLSX write example, this code will generate a XLSX with a number in cell A1 that is right-aligned automatically:

from openpyxl import Workbook
wb = Workbook()
sheet = wb["Sheet"] # This sheet is created by default
# Add content to sheet
sheet["A1"] = 4.5
sheet["A1"].alignment = Alignment(horizontal="center")
# Save
wb.save("openpyxl-test.xlsx")

How to add centered alignment

In order to align the cell horizontally, we just need to

from openpyxl.styles.alignment import Alignment

sheet["A1"].alignment = Alignment(horizontal="center")

Full example

from openpyxl import Workbook
from openpyxl.styles.alignment import Alignment
wb = Workbook()
sheet = wb["Sheet"] # This sheet is created by default
# Add content to sheet
sheet["A1"] = 4.5
sheet["A1"].alignment = Alignment(horizontal="center")
# Save
wb.save("openpyxl-test.xlsx")

This will look like this:

Posted by Uli Köhler in OpenPyXL, Python

Pandas XLSX export with background color based on cell value

This example script uses openpyxl to set a cell’s background color to red for False cells or to green for True cells. Note that we convert the boolean values to "True"or "False" strings based on the method we outlined in our previous post How to write Pandas bool column as True/False instead of 1/0 to XLSX. For more details on how to just set the background color in OpenPyXL, see our post on How to set cell background color in OpenPyXL.

import pandas as pd
from openpyxl.styles import PatternFill

df = pd.DataFrame([
    {"a": True},
    {"a": False},
    {"a": True},
    {"a": False},
    {"a": False},
])
df["a"] = df["a"].map({True: "True", False: "False"})
with pd.ExcelWriter("out.xlsx", engine="openpyxl") as writer:
    sheet_name = "Bool"
    # Export DataFrame content
    df.to_excel(writer, sheet_name=sheet_name)
    # Set backgrund colors depending on cell values
    sheet = writer.sheets[sheet_name]
    for cell, in sheet[f'B2:B{len(df) + 1}']: # Skip header row, process as many rows as there are DataFrames
        value = df["a"].iloc[cell.row - 2] # value is "True" or "False"
        cell.fill = PatternFill("solid", start_color=("5cb800" if value == "True" else 'ff2800'))

The output from this script looks like this:

Posted by Uli Köhler in OpenPyXL, Python

How to get OpenPyXL column letter by index

If you want to access OpenPyXL columns using indices instead of letters, use

import openpyxl.utils.cell

openpyxl.utils.cell.get_column_letter(idx)

Note that idx is 1-based: index 0 is not a valid argument & index 1 yields column name A 

For example:

openpyxl.utils.cell.get_column_letter(3) # returns "C"

 

Posted by Uli Köhler in OpenPyXL, Python

How to set cell background color in OpenPyXL

In order to set a cell’s background color in OpenPyXL, use PatternFill with fill_type="solid" and start_color="your_desired_color"  and no end_color. The following example will set a cell’s background to green:

sheet["A1"].fill = PatternFill("solid", start_color="5cb800")

Full example based on our OpenPyXL minimal XLSX write example

from openpyxl import Workbook
wb = Workbook()
sheet = wb["Sheet"] # This sheet is created by default
# Add content to sheet
sheet["A1"] = "This is cell A1"
sheet["A1"].fill = PatternFill("solid", start_color="5cb800")
# Save
wb.save("openpyxl-test.xlsx")

Posted by Uli Köhler in OpenPyXL, Python

OpenPyXL minimal XLSX write example

This serves a minimal example of how to write an XLSX file using OpenPyXL:

from openpyxl import Workbook
wb = Workbook()
sheet = wb["Sheet"] # This sheet is created by default
# Add content to sheet
sheet["A1"] = "This is cell A1"
# Save
wb.save("openpyxl-test.xlsx")

Posted by Uli Köhler in OpenPyXL, Python

How to write Pandas bool column as True/False instead of 1/0 to XLSX

Problem:

When writing a pandas XLSX, bool-type columns are shown are 0 for False or 1 for True.

df = pd.DataFrame([
    {"a": True},
    {"a": False},
    {"a": True},
    {"a": False},
    {"a": False},
])

with pd.ExcelWriter("out.xlsx", engine="xlsxwriter") as writer:
    df.to_excel(writer)

Solution:

Map the column to a string column with your desired value before exporting:

df["a"] = df["a"].map({True: "True", False: "False"})

Full example code:

df = pd.DataFrame([
    {"a": True},
    {"a": False},
    {"a": True},
    {"a": False},
    {"a": False},
])
df["a"] = df["a"].map({True: "True", False: "False"})
with pd.ExcelWriter("out.xlsx", engine="xlsxwriter") as writer:
    df.to_excel(writer)

 

Posted by Uli Köhler in pandas, Python

How to implement 1MHz interrupt in PlatformIO / Arduino on STM32

In our previous post Minimal STM32 HardwareTimer PlatformIO / Arduino timer interrupt blink example we showed how to use HardwareTimer to blink the onboard LED of our STM32F407 board using a timer interrupt.

In this post, we’ll provide an example of how to use HardwareTimer and have a really fast interrupt which runs at 1 MHz – in other words: one million times per second.

#include <Arduino.h>

HardwareTimer timer(TIM1);
bool ledOn = false;

void OnTimer1Interrupt() {
    ledOn = !ledOn;
    digitalWrite(PC13, ledOn ? HIGH : LOW);
}

void setup() {
    pinMode(PC13, OUTPUT);
    // Configure timer
    timer.setPrescaleFactor(21); // Set prescaler to 21 => timer frequency = 168/21 = 8 MHz (from prediv'd by 1 clocksource of 168 MHz)
    timer.setOverflow(8); // Set ARR to 8 => timer frequency = 1 MHz
    timer.attachInterrupt(OnTimer1Interrupt);
    timer.refresh(); // Make register changes take effect
    timer.resume(); // Start timre
}

void loop() {
}

Note that when running such a quick interrupt, you can’t do all too much within the interrupt before the next time the interrupt will run.

Posted by Uli Köhler in C/C++, PlatformIO, STM32

How to use STM32 _Msk and _Pos definitions to read and write registers

The STM32 HAL contains definitions like TIM_CR1_CKD_Msk or TIM_CR1_CKD_Pos which you can use to make it easier to read or write parts of a register.

Reading a part of a register

uint32_t ckd = (TIM1->CR1 & TIM_CR1_CKD_Msk) >> TIM_CR1_CKD_Pos;

Writing a part of a register

uint32_t new_ckd_value = TIM_CLOCKDIVISION_DIV4; // example
TIM1->CR1 &= TIM_CR1_CKD_Msk; // Clear bits
TIM1->CR1 |= new_ckd_value << TIM_CR1_CKD_Pos; // Set bits

 

Posted by Uli Köhler in C/C++, STM32

How to create pandas pd.DataFrame from list of dicts

You can create a pandas pd.DataFrame from a list of dicts (lst) using the pd.DataFrame(lst) constructor.

a = {"x": 1, "y": 2}
b = {"x": 3, "y": 2}
c = {"x": 6, "y": 9}
df = pd.DataFrame([a,b,c])

df will look like this:

Posted by Uli Köhler in pandas, Python

How to identify the latest npm package version

You can use

npm info [package name] version | head -n1

to print just the latest package version, for example:

npm info @angular/cli version | head -n1

The head -n1 part is neccessary to suppress unwanted npm info messages. Just running the command without head -n1 :

npm info @angular/cli version

prints the following output:

12.2.6
npm notice 
npm notice New minor version of npm available! 7.21.1 -> 7.24.0
npm notice Changelog: https://github.com/npm/cli/releases/tag/v7.24.0
npm notice Run npm install -g [email protected] to update!
npm notice

I typically use the node:latest docker container instead of a local npm to find the latest version of an npm package:

docker run -it --rm node:latest npm info meshcommander version | head -n1

At the time of writing this article, the command prints

0.9.0-d

 

Posted by Uli Köhler in NodeJS

How to get size of compressed file using JSZip

In JSZip, you can get the file size of a file compressed inside a ZIP archive by first decompressing it using zip.file(...).async(...) into an ArrayBuffer and then using .byteLength to get the size of the buffer.

zip.file("filename.txt").async("ArrayBuffer").then(function(data) {
    var fileSize = data.byteLength;
    // TODO Your code goes here
})

Full example

This is based on our post on How to uncompress file inside ZIP archive using HTML5 & JSZip which reads a local file using a HTML5 file input. This example prints the file size of every file inside the ZIP archive:

<html>
<body>
    <input type="file" id="myfile" onchange="onMyfileChange(this)" />

    <script src="https://unpkg.com/[email protected]/dist/jszip.js" type="text/javascript"></script>
    <script type="text/javascript">
        function onMyfileChange(fileInput) {
            if(fileInput.files[0] == undefined) {
                return ;
            }

            var filename = fileInput.files[0].name;
            var reader = new FileReader();
            reader.onload = function(ev) {
                JSZip.loadAsync(ev.target.result).then(function(zip) {
                    zip.file("word/document.xml").async("ArrayBuffer").then(function(data) {
                        console.log("word/document.xml is", data.byteLength, "bytes long");
                    })
                }).catch(function(err) {
                    console.error("Failed to open", filename, " as ZIP file:", err);
                })
            };
            reader.onerror = function(err) {
                console.error("Failed to read file", err);
            }
            reader.readAsArrayBuffer(fileInput.files[0]);
        }
    </script>
</body>
</html>

Example output

word/document.xml is 88369 bytes long

 

Posted by Uli Köhler in Javascript

How to list files inside a ZIP archive using JSZip

In JSZip, you can list files inside a ZIP archive using zip.fileswhich is an Object with key = filename and value = file metadata object.

JSZip.loadAsync(fileContent).then(function(zip) {
    for(let [filename, file] of Object.entries(zip.files)) {
        // TODO Your code goes here
        console.log(filename);
    }
}).catch(function(err) {
    console.error("Failed to open", filename, " as ZIP file:", err);
})

Full example

This is based on our post on How to read local ZIP in HTML5/Javascript using JSZip

<html>
<body>
    <input type="file" id="myfile" onchange="onMyfileChange(this)" />

    <script src="https://unpkg.com/[email protected]/dist/jszip.js" type="text/javascript"></script>
    <script type="text/javascript">
        function onMyfileChange(fileInput) {
            if(fileInput.files[0] == undefined) {
                return ;
            }

            var filename = fileInput.files[0].name;
            // var filesize = fileInput.files[0].size;
            var reader = new FileReader();
            reader.onload = function(ev) {
                JSZip.loadAsync(ev.target.result).then(function(zip) {
                    console.log(zip)
                }).catch(function(err) {
                    console.error("Failed to open", filename, " as ZIP file");
                })
            };
            reader.onerror = function(err) {
                console.error("Failed to read file", err);
            }
            reader.readAsArrayBuffer(fileInput.files[0]);
        }
    </script>
</body>
</html>

 

 

Posted by Uli Köhler in Javascript

How to read a file inside a ZIP archive as string using JSZip

Want to read as an ArrayBuffer instead? Read How to read a file inside a ZIP archive as ArrayBuffer using JSZip

In JSZip, you can read a compressed file as JavaScript string using

zip.file("filename.txt").async("string").then(function(data) {
    // data is a string
    // TODO your code goes here, this is just an example
    console.log(data);
})

Full example

This is based on our post on How to uncompress file inside ZIP archive using HTML5 & JSZip which reads a local file using a HTML5 file input. You can upload any .docx file for testing:

<html>
<body>
    <input type="file" id="myfile" onchange="onMyfileChange(this)" />

    <script src="https://unpkg.com/[email protected]/dist/jszip.js" type="text/javascript"></script>
    <script type="text/javascript">
        function onMyfileChange(fileInput) {
            if(fileInput.files[0] == undefined) {
                return ;
            }

            var filename = fileInput.files[0].name;
            var reader = new FileReader();
            reader.onload = function(ev) {
                JSZip.loadAsync(ev.target.result).then(function(zip) {
                    zip.file("word/document.xml").async("string").then(function(data) {
                        // data is a string
                        // TODO Your code goes here!
                        console.log(data)
                    })
                }).catch(function(err) {
                    console.error("Failed to open", filename, " as ZIP file:", err);
                })
            };
            reader.onerror = function(err) {
                console.error("Failed to read file", err);
            }
            reader.readAsArrayBuffer(fileInput.files[0]);
        }
    </script>
</body>
</html>

 

Posted by Uli Köhler in Javascript