How to get into a MicroPython REPL using USB in 10 seconds

On Linux, use picocom:

picocom /dev/ttyUSB0 -b115200

and press enter to get a REPL (or press the RESET button on the board if that does not work).

You should now see the

>>>

prompt.

Don’t have picocom installed?

sudo apt -y install picocom

Got permission denied errors?

sudo usermod -a -G dialout $USER

then logout and log back in or reboot!

Posted by Uli Köhler in MicroPython

How to flash MicroPython to your ESP32 board in 30 seconds

This script does all the neccessary steps to flash MicroPython to your ESP32 board on Linux. Run this script with your board plugged in via USB:

#!/bin/sh
# Call without arguments
# Download esptool
git clone https://github.com/espressif/esptool.git -b v2.8
cd esptool
# Erase flash. Press the reset button while Connecting.
python3 esptool.py --chip esp32 --port /dev/ttyUSB* erase_flash
# Download firmware
wget -qO micropython.bin https://micropython.org/resources/firmware/esp32-idf4-20191220-v1.12.bin
# Upload firmware to board
python3 esptool.py --chip esp32 --port /dev/ttyUSB* --baud 460800 write_flash -z 0x1000 micropython.bin

Remember to press the RESET button on the board if the script is telling you Connecting.... Not sure which button it is? Just press either one, and if the output doesn’t show Chip is ESP32 within 2 seconds, press the other one.

This script was built for my ESP-WROOM-32 board.

Now you can try to get into a REPL using USB.

Posted by Uli Köhler in Embedded, MicroPython

How to fix Nextcloud upgrade Migration step ‘…’ is unknown

Problem:

When upgrading your Nextcloud instance, you see an error message like

Migration step 'OCA\Files\Migration\Version11301Date20191205150729' is unknown

Solution:

This error occurs because you just copied over multiple versions of the apps directory, which resulted in

First, make a backup of your apps directory (located inside your Nextcloud folder).

Then delete your current apps directory and replace it by the apps directory from the new version of Nextcloud you want to upgrade to.

After that, retry the upgrade.

Posted by Uli Köhler in Nextcloud

How to fix NextCloud OnlyOffice MixedContent or ‘Refused to frame ‘http://…’ because it violates the following Content Security Policy directive: “frame-src https://…”.

Problem:

In reverse-proxy setups  forwarding requests to OnlyOffice like our reference setup there you might encounter issues like

Refused to frame 'http://onlyoffice.mydomain.com/' because it violates the following Content Security Policy directive: "frame-src https://onlyoffice.mydomain.com/".

Solution:

Just add

proxy_set_header X-Forwarded-Proto $scheme;

directly after your proxy_pass clause in your nginx config, then run sudo service nginx reload.

The reason for this issue is that OnlyOffice thinks it’s being loaded using HTTP, but the Nextcloud page prevents insecure content from being loaded.

Using a proxy other than nginx? Just ensure that every proxied request (i.e. every request directed towards the OnlyOffice instance) has the X-Forwarded-Proto header set to the protocol of the original request – which should be https.

Posted by Uli Köhler in Nextcloud, nginx

How to setup OnlyOffice using docker-compose, systemd and nginx

In this setup we show how to setup OnlyOffice using nginx as a reverse proxy, docker-compose to run and configure the OnlyOffice image and systemd to automatically start and restart the OnlyOffice instance. Running it in a reverse proxy configuration allows you to have other domains listening on the same IP address and have a central management of Let’s Encrypt SSL certificates.

We will setup the instance in /opt/onlyoffice on port 2291.

Save this file as /opt/onlyoffice/docker-compose.yml and don’t forget to change JWT_SECRET to a random password!

version: '3'
services:
  onlyoffice-documentserver:
    image: onlyoffice/documentserver:latest
    restart: always
    environment:
      - JWT_ENABLED=true
      - JWT_SECRET=Shei9AifuZ4ze7udahG2seb3aa6ool
    ports:
      - 2291:80
    volumes:
      - ./onlyoffice/data:/var/www/onlyoffice/Data
      - ./onlyoffice/lib:/var/lib/onlyoffice
      - ./onlyoffice/logs:/var/log/onlyoffice
      - ./onlyoffice/db:/var/lib/postgresql

Now we can create the systemd service. I created it using TechOverflow’s docker-compose systemd .service generator. Save it in /etc/systemd/system/OnlyOffice.service:

[Unit]
Description=OnlyOffice
Requires=docker.service
After=docker.service

[Service]
Restart=always
User=root
Group=docker
# Shutdown container (if running) when unit is stopped
ExecStartPre=/usr/local/bin/docker-compose -f /opt/onlyoffice/docker-compose.yml down -v
# Start container when unit is started
ExecStart=/usr/local/bin/docker-compose -f /opt/onlyoffice/docker-compose.yml up
# Stop container when unit is stopped
ExecStop=/usr/local/bin/docker-compose -f /opt/onlyoffice/docker-compose.yml down -v

[Install]
WantedBy=multi-user.target

Now we can enable & start the service using

sudo systemctl enable OnlyOffice.service
sudo systemctl start OnlyOffice.service

Now let’s create the nginx config in /etc/nginx/sites-enabled/OnlyOffice.conf. Obviously, you’ll have to modify at least the

 server {
    server_name onlyoffice.mydomain.com;

    access_log /var/log/nginx/onlyoffice.access_log;
    error_log /var/log/nginx/onlyoffice.error_log info;

    location / {
        proxy_pass http://127.0.0.1:2291;
        proxy_http_version 1.1;
        proxy_read_timeout 3600s;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "Upgrade";
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $remote_addr;
        proxy_set_header X-Forwarded-Proto $scheme;
        add_header X-Frontend-Host $host;
    }

    listen 80;
}

Check the validity of the nginx config using

sudo nginx -t

and unless it fails, reload nginx using

sudo service nginx reload

Now I recommend to use certbot to enable TLS encryption on your domain. You should be familiar with these steps already ; my approach is to sudo apt -y install python-certbot-nginx, then certbot --nginx --staging to first obtain a staging certificate to avoid being blocked if there are any issues and after you have obtained the staging certificate use certbot --nginx and Renew & replace cert. After that, run sudo service nginx reload and check if you domain works with HTTPS. You should always choose redirection to HTTPS if certbot asks you.

Posted by Uli Köhler in Docker, nginx

How to fix Nextcloud Refused to send form data to /login/v2/grant because it violates the following Content Security Policy directive: form-action ‘self’

Problem:

When trying to connect using the Nextcloud client, during the Flow v2 authorization step where you open a page in the browser to authorize the client, you see an error message in the JS console like

Refused to send form data to 'http://nextcloud.mydomain.com/login/v2/grant' because it violates the following Content Security Policy directive: "form-action 'self'".

Solution:

Add

'overwriteprotocol' => 'https',

after this line:

'version' => '18.0.0.10',

in your nextcloud/config/config.php.

Posted by Uli Köhler in Nextcloud, PHP

Auto-generate nginx forwarding configs using this script

A major hassle for me when setting up lots of docker-based services on a machine is to setup the individual nginx configs to forward the domains to the correct services.

TechOverflow provides a simple way to automatically generate nginx configs for a single domain to configure port forwarding to a specific port.

wget -qO- https://techoverflow.net/scripts/generate-nginx-docker-config.sh | sudo bash /dev/stdin service.mydomain.com 1234

Note: This script was tested on Ubuntu 18.04 and is regularly used by myself and others. However, if used incorrectly or in case there is a major bug, it might damage your webserver configuration, so be sure to be prepared to fix any issues that might arise. This script is provided without any warranty, expressed or implied.

Remember to replace service.mydomain.com by your domain and 1234 by the local port your docker service is listening to.

The script will generate /etc/nginx/sites-enabled/[domain].conf.

Run

sudo nginx -t

to check if the config syntax is OK, and only if that succeeds, run

sudo service nginx reload

Now your domain should be online and I recommend running

sudo certbot --nginx

 

Posted by Uli Köhler in nginx

How to read IDF diabetes statistics in Python using Pandas

The International Diabetes Foundation provides a Data portal with various statistics related to diabetes.

In this post we’ll show how to read the Diabetes estimates (20-79 y) / People with diabetes, in 1,000s data export in CSV format using pandas.

First download IDF (people-with-diabetes--in-1-000s).csv from the data page.

Now we can parse the CSV file:

import pandas as pd

# Download at https://www.diabetesatlas.org/data/en/indicators/1/
df = pd.read_csv("IDF (people-with-diabetes--in-1-000s).csv")
# Parse year columns to obtain floats and multiply by thousands factor. Pandas fails to parse values like "12,345.67"
for column in df.columns:
    try:
        int(column)
        df[column] = df[column].apply(lambda s: None if s == "-" else float(s.replace(",", "")) * 1000)
    except:
        pass

As you can see in the postprocessing step, the number of diabetes patients are given in 1000s in the CSV, so we multiply them by 1000 to obtain the actual numbers.

If you want to modify the data columns (i.e. the columns referring to year), you can use this simple template:

for column in df.columns:
    try:
        int(column) # Will raise ValueError() if column is not a year number
        # Whatever you do here will only be applied to year columns
        df[column] = df[column] * 0.75 # Example on how to modify a column
        # But note that if your code raises an Exception, it will be ignored!
    except:
        pass

Let’s plot some data:

regions = df[df["Type"] == "Region"] # Only regions, not individual countries

from matplotlib import pyplot as plt
plt.style.use("ggplot")
plt.gcf().set_size_inches(20,4)
plt.ylabel("Diabetes patients [millions]")
plt.xlabel("Region")
plt.title("Diabetes patients in 2019 by region")
plt.bar(regions["Country/Territory"], regions["2019"] / 1e6)

Note that if you use a more recent dataset than the version I’m using the 2019 column might not exist in your CSV file. Choose an appropriate column in that case.

Posted by Uli Köhler in Bioinformatics, Python

How to repair docker-compose MariaDB instances (aria_chk -r)

Problem:

You are trying to run a MariaDB container using docker-compose. However, the database container doesn’t start up and you see error messages like these in the logs:

[ERROR] mysqld: Aria recovery failed. Please run aria_chk -r on all Aria tables and delete all aria_log.######## files
[ERROR] Plugin 'Aria' init function returned error.
[ERROR] Plugin 'Aria' registration as a STORAGE ENGINE failed.
....
[ERROR] Could not open mysql.plugin table. Some plugins may be not loaded
[ERROR] Failed to initialize plugins.
[ERROR] Aborting

Solution:

The log messages already tell you what to do – but they don’t tell you how to do it:

Aria recovery failed. Please run aria_chk -r on all Aria tables and delete all aria_log.######## files

First, backup the entire MariaDB data directory: Check onto which host directory the data directory (/var/lib/mysql) of the container is mapped and copy the entire directory to a backup space. This is important in case the repair process fails.

Now let’s run aria_chk -r to check and repair MySQL table files.

docker-compose run my-db bash -c 'aria_chk -r /var/lib/mysql/**/*'

Replace my-db by the name of your database container. This will attempt to repair a lot of non-table-files as well but aria_chk will happily ignore those.

Now we can delete the log files:

docker-compose run my-db bash -c 'rm /var/lib/mysql/aria_log.*'

Again, replace my-db by the name of your database container.

Posted by Uli Köhler in Databases, Docker

Parsing World Population Prospects (WPP) XLSX data in Python

The United Nations provides the Word Population Prospects (WPP) dataset on geographic and age distribution of mankind as downloadable XLSX files.

Reading these files in Python is rather easy. First we have to find out how many rows to skip. For the 2019 WPP dataset this value is 16 since row 17 contains all the column headers. The number of rows to skip might be different depending on the dataset. We’re using WPP2019_POP_F07_1_POPULATION_BY_AGE_BOTH_SEXES.xlsx in this example.

We can use Pandas read_excel() function to import the dataset in Python:

import pandas as pd

df = pd.read_excel("WPP2019_INT_F03_1_POPULATION_BY_AGE_ANNUAL_BOTH_SEXES.xlsx", skiprows=16, na_values=["..."])

This will take a few seconds until the large dataset has been processed. Now we can check if skiprows=16 is the correct value. It is correct if pandas did recognize the column names correctly:

>>> df.columns
Index(['Index', 'Variant', 'Region, subregion, country or area *', 'Notes',
       'Country code', 'Type', 'Parent code', 'Reference date (as of 1 July)',
       '0-4', '5-9', '10-14', '15-19', '20-24', '25-29', '30-34', '35-39',
       '40-44', '45-49', '50-54', '55-59', '60-64', '65-69', '70-74', '75-79',
       '80-84', '85-89', '90-94', '95-99', '100+'],
      dtype='object')

Now let’s filter for a country:

russia = df[df["Region, subregion, country or area *"] == 'Russian Federation']

This will show us the population data for multiple years in 5-year intervals from 1950 to 2020. Now let’s filter for the most recent year:

russia.loc[russia["Reference date (as of 1 July)"].idxmax()]

This will show us a single dataset:

Index                                                 3255
Variant                                          Estimates
Region, subregion, country or area *    Russian Federation
Notes                                                  NaN
Country code                                           643
Type                                          Country/Area
Parent code                                            923
Reference date (as of 1 July)                         2020
0-4                                                9271.69
5-9                                                9350.92
10-14                                              8174.26
15-19                                              7081.77
20-24                                               6614.7
25-29                                              8993.09
30-34                                              12543.8
35-39                                              11924.7
40-44                                              10604.6
45-49                                              9770.68
50-54                                              8479.65
55-59                                                10418
60-64                                              10073.6
65-69                                              8427.75
70-74                                              5390.38
75-79                                              3159.34
80-84                                              3485.78
85-89                                              1389.64
90-94                                              668.338
95-99                                              102.243
100+                                                 9.407
Name: 3254, dtype: object
​

How can we plot that data? First, we need to select all the columns that contain age data. We’ll do this by manually inserting the name of the first such column (0-4) into the following code and assuming that there are no columns after the last age column:

>>> df.columns[df.columns.get_loc("0-4"):]
Index(['0-4', '5-9', '10-14', '15-19', '20-24', '25-29', '30-34', '35-39',
       '40-44', '45-49', '50-54', '55-59', '60-64', '65-69', '70-74', '75-79',
       '80-84', '85-89', '90-94', '95-99', '100+'],
      dtype='object')

Now let’s select those columns from the russia dataset:

most_recent_russia = russia.loc[russia["Reference date (as of 1 July)"].idxmax()]
age_columns = df.columns[df.columns.get_loc("0-4"):]

russian_age_data = most_recent_russia[age_columns]

Let’s have a look at the dataset:

>>> russian_age_data
0-4      9271.69
5-9      9350.92
10-14    8174.26
15-19    7081.77
20-24     6614.7
25-29    8993.09
30-34    12543.8
35-39    11924.7
40-44    10604.6
45-49    9770.68
50-54    8479.65
55-59      10418
60-64    10073.6
65-69    8427.75
70-74    5390.38
75-79    3159.34
80-84    3485.78
85-89    1389.64
90-94    668.338
95-99    102.243
100+       9.407

That looks useable, note however that the values are in thousands, i.e. we have to multiply the values by 1000 to obtain the actual estimates of the population. Let’s plot it:

from matplotlib import pyplot as plt
plt.style.use("ggplot")

plt.title("Age composition of the Russian population (2020)")
plt.ylabel("People in age group [Millions]")
plt.xlabel("Age group")
plt.gcf().set_size_inches(15,5)
# Data is given in thousands => divide by 1000 to obtain millions
plt.plot(russian_age_data.index, russian_age_data.as_matrix() / 1000., lw=3)

The finished plot will look like this:

Here’s our finished script:

#!/usr/bin/env python3
import pandas as pd
df = pd.read_excel("WPP2019_POP_F07_1_POPULATION_BY_AGE_BOTH_SEXES.xlsx", skiprows=16)
# Filter only russia
russia = df[df["Region, subregion, country or area *"] == 'Russian Federation']

# Filter only most recent estimate (1 row)
most_recent_russia = russia.loc[russia["Reference date (as of 1 July)"].idxmax()]
# Retain only value columns
age_columns = df.columns[df.columns.get_loc("0-4"):]
russian_age_data = most_recent_russia[age_columns]

# Plot!
from matplotlib import pyplot as plt
plt.style.use("ggplot")

plt.title("Age composition of the Russian population (2020)")
plt.ylabel("People in age group [Millions]")
plt.xlabel("Age group")
plt.gcf().set_size_inches(15,5)
# Data is given in thousands => divide by 1000 to obtain millions
plt.plot(russian_age_data.index, russian_age_data.as_matrix() / 1000., lw=3)

# Export as SVG
plt.savefig("russian-demographics.svg")

 

 

Posted by Uli Köhler in Bioinformatics, Data science, Python

How to automatically cleanup your docker registry instance

Quick install

This quick-install script works if you are running the docker registry image using docker-compose and the service in docker-compose.yml is called registry. I recommend to use our example on how to install the docker registry for Gitlab (not yet available).

Run this in the directory where docker-compose.yml  is located!

wget -qO- https://techoverflow.net/scripts/install-registry-autocleanup.sh | sudo bash

Need an explanation (or not using docker-compose)?

Docker registry instances will store every version of every image you push to them, so especially if you are in a continous integration environment you might want to do periodic cleanups that delete all images without a tag.

The command to do that is

registry garbage-collect /etc/docker/registry/config.yml -m

You can use a systemd service like

[Unit]
Description=registry-gc

[Service]
Type=oneshot
ExecStart=/usr/local/bin/docker-compose exec -T registry bin/registry garbage-collect /etc/docker/registry/config.yml -m
WorkingDirectory=/opt/my-registry

and a timer like

[Unit]
Description=registry-gc

[Timer]
OnCalendar=daily
Persistent=true

[Install]
WantedBy=timers.target

to run the command daily. You need to adjust both the WorkingDirectory and the exact docker-compose exec command to suit your needs.

Copy both files to /etc/systemd/system and enable the timer using

sudo systemctl enable registry-gc.timer

and you can run it manually at any time using

sudo systemctl start registry-gc.service
Posted by Uli Köhler in Docker

How to fix Gitlab Runner ‘dial unix /var/run/docker.sock: connect: permission denied’

Problem:

In your Gitlab build jobs that use docker you see error messages like

Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post http://%2Fvar%2Frun%2Fdocker.sock/v1.40/auth: dial unix /var/run/docker.sock: connect: permission denied

Solution:

usermod -a -G docker gitlab-runner

to give the user running the jobs permission to access docker resources then restart the server/VM on which the runner is installed !

Still doesn’t work? Check if you have installed docker correctly. We recommend to use our automated install script, see How to install docker and docker-compose on Ubuntu in 30 seconds.

Posted by Uli Köhler in Docker

How to automatically remove docker images that are not associated to a container daily

Note: This will not only remove docker images without a tag but all docker images not associated to a running or stopped container. See our previous post How to automatically cleanup (prune) docker images daily in case this is not the desired behaviour.

docker image prune provides an easy way to remove “unused” docker images from a system and hence fixes or significantly delays docker eating up all your disk space on e.g. automated disk space.

I created a systemd-timer based daily image removal routine using TechOverflow’s Simple systemd timer generator.

Quick install using

wget -qO- https://techoverflow.net/scripts/install-cleanup-docker-all.sh | sudo bash

This is the script which automatically creates & installs both systemd config files.

#!/bin/sh
# This script installs automated docker cleanup.
# onto systemd-based systems.
# See https://techoverflow.net/2020/02/04/how-to-remove-all-docker-images-that-are-not-associated-to-a-container/
# for details on what images are removed.
# It requires that docker is installed properly

cat >/etc/systemd/system/PruneDockerAll.service <<EOF
[Unit]
Description=PruneDockerAll

[Service]
Type=oneshot
ExecStart=/bin/bash -c "docker image ls --format '{{.ID}}' | xargs docker image rm ; true"
WorkingDirectory=/tmp
EOF

cat >/etc/systemd/system/PruneDockerAll.timer <<EOF
[Unit]
Description=PruneDockerAll

[Timer]
OnCalendar=daily
Persistent=true

[Install]
WantedBy=timers.target
EOF

# Enable and start service
systemctl enable PruneDockerAll.timer && sudo systemctl start PruneDockerAll.timer

 

To view the logs, use

journalctl -xfu PruneDockerAll.service

To view the status, use

sudo systemctl status PruneDockerAll.timer

To immediately cleanup your docker images, use

sudo systemctl start PruneDockerAll.service
Posted by Uli Köhler in Docker

How to remove all docker images that are not associated to a container

In our previous post we showed how to prune docker images to free up space on your hard drive. However, this approach will not remove images that have tags (i.e. names) associated with them.

Often you want to remove all images that are not required by one of the containers (both running and stopped containers).

This is pretty easy:

docker image ls --format '{{.ID}}' | xargs docker image rm

This command will list all image IDs using docker image ls --format '{{.ID}}' and run docker image rm for every image ID.

Since docker image rm will fail for images that are associated to either a running or a stopped container (hence that image won’t be deleted), this will only delete those images that are not associated to any container.

In case you get a lot of those error messages as output from the command:

Error response from daemon: conflict: unable to delete 1f9cfa8dc305 (cannot be forced) - image is being used by running container 22a27af7d595
Error response from daemon: conflict: unable to delete 9af515ad5c74 (must be forced) - image is being used by stopped container 2ebcbd936841

don’t worry, that’s fine, that just means one of your images is associated to a container and hence won’t be deleted.

Posted by Uli Köhler in Docker

How to read TSV (tab-separated values) in C++

This minimal example show your how to read & parse a tab-separated values (TSV) file in C++. We use boost::algorithm::split to split each line into its tab-separated components.

#include <fstream>
#include <iostream>
#include <string>
#include <vector>
#include <boost/algorithm/string.hpp>

using namespace std;
using namespace boost::algorithm;

int main(int argc, char** argv) {
    ifstream fin("test.tsv");
    string line;
    while (getline(fin, line)) {
        // Split line into tab-separated parts
        vector<string> parts;
        split(parts, line, boost::is_any_of("\t"));
        // TODO Your code goes here!
        cout << "First of " << parts.size() << " elements: " << parts[0] << endl;
    }
    fin.close();
}

 

 

Posted by Uli Köhler in C/C++

C++ read file line by line minimal example

This minimal example reads a file line-by-line using std::getline and prints out each line on stdout.

#include <fstream>
#include <iostream>
#include <string>

using namespace std;

int main(int argc, char** argv) {
    ifstream fin("test.tsv");
    string line;
    while (getline(fin, line)) {
        // TODO Your code goes here. This is just an example!
        cout << line << endl;
    }
    fin.close();
}

 

Posted by Uli Köhler in C/C++

How to fix bitnami mariadb ‘mkdir: cannot create directory ‘/bitnami/mariadb’: Permission denied’

Problem:

You are trying to run a docker bitnami/mariadb container but when you try to start it up, you see an error message like

mariadb_1   | mkdir: cannot create directory '/bitnami/mariadb': Permission denied

Solution:

bitnami containers mostly are non-root-containers, hence you need to adjust the permissions for the data directory mapped onto the host.

First, find out what directory your /bitnami is mapped to on the host. For example, for

services:
     mariadb:
         image: 'bitnami/mariadb:latest'
         environment:
             - ALLOW_EMPTY_PASSWORD=yes
         volumes:
             - '/var/lib/my_docker/mariadb_data:/bitnami'

it is mapped to /var/lib/my_docker/mariadb_data.

Now chown this directory to 1001:1001 since the image is using UID 1001 as the user running the command:

sudo chown -R 1001:1001 [directory]

for example

sudo chown -R 1001:1001 /var/lib/my_docker/mariadb_data

 

Posted by Uli Köhler in Docker