Uli Köhler

How to fix ‘Configuring tzdata’ interactive input when building Docker images

Often, when installing deb packages in your Dockerfile, some packages will install tzdata as a dependency.

The tzdata installer will try to interactively prompt you for your location using

Configuring tzdata
------------------

Please select the geographic area in which you live. Subsequent configuration
questions will narrow this down by presenting a list of cities, representing
the time zones in which they are located.

 1. Africa      4. Australia  7. Atlantic  10. Pacific  13. Etc
 2. America     5. Arctic     8. Europe    11. SystemV
 3. Antarctica  6. Asia       9. Indian    12. US
Geographic area:

This will stall your image build.

In order to fix that, we’ll need to make the tzdata prompt non-interactive.

The preferred method is to add

ENV DEBIAN_FRONTEND=noninteractive

before the first RUN statements in your Dockerfile.

Alternatively you can run the apt install or apt-get install command using DEBIAN_FRONTEND=noninteractive:

RUN DEBIAN_FRONTEND=noninteractive apt install -y tzdata

This will automatically select a default configuration for tzdata.

Posted by Uli Köhler in Container, Docker

How to install NodeJS 12.x on Ubuntu in 1 minute

Run these shell commands on your Ubuntu computer to install NodeJS 12.x:

curl -sL https://deb.nodesource.com/setup_12.x | sudo -E bash -
sudo apt-get install -y nodejs

Instead of setup_12.x you can also choose other versions like setup_8.x. However, using this method, you can’t install multiple versions of NodeJS in parallel.

Source: Official nodesource documentation

Posted by Uli Köhler in Linux, NodeJS

What is a ‘headless’ program or application?

If you’re a backend programmer, you most likely have encountered the term of headless applications (like headless Java or headless Chromium) many times. But what does headless actually mean?

headless means it runs without a graphical user interface (GUI). In most cases, headless applications are command line applications or applications that are interfaced from a programming language.

When you open your browser on your Desktop or Laptop, you’ll see the browser window pop up. This window is a graphical user interface (GUI). Unless you are living under a rock, you have had your fair share of experience with GUIs and know how convenient they can be sometimes.

However, especially in the professional IT world, command line user interfaces are also common – mostly, because they can be used on servers and local PCs with a screen alike, and they are much easier to automate than GUIs.

If you have worked in application development for some time, you might know that GUI applications require a huge number of incredibly complex libraries to run. On Linux that includes the X11 server (X11 is the system that displays and arranges all the windows and other graphical features on your screen), some utility libraries for X11 and possibly some libraries to be able to create the high-level user interfaces in use in most applications today: While high-level GUIs consist of buttons and text inputs, low-level GUIs consist of pixel, colored areas and callback functions (from which the buttons and text inputs are built of).

As said before, headless applications run without a GUI. Under the right circumstances, this has two main advantages:

  • You don’t have to spend as much time developing a complex graphical user interface
  • You can easily run it on a server
Why can you run headless applications on a server but not normal GUI applications?

Firstly, as we said before, GUIs require a large number of complex libraries and software infrastructure to be present on a server. On many servers (like most Linux servers), this software is not installed by default – because installing it would eat up valuable resources like system memory (RAM), hard drive space and the system would be hard to maintain.

Contrary to common belief, running a GUI infrastructure on Linux (i.e. X server plus some utilities) does not require a screen or a dedicated graphics card. See our post on How to run X server using xserver-xorg-video-dummy driver on Ubuntu for an example on how to accomplish this.

Secondly, GUIs are hard to monitor, maintain and automate:

Imagine you have 25 servers running the same application on each of them. If that were a GUI application, you would have to look at 25 windows displaying the state of the application – not to mention the time spent to make all the windows display on your local screen reliably – or the development time spent to make the GUI

Using headless applications you can instead easily automate the task of monitoring the applications and therefore just display a summary on your local screen. Since you don’t have to spend time writing and maintaining GUIs, you can also spend your time more wisely.

Posted by Uli Köhler in Technologies

How to iterate all days of year using Python

To iterate all days in a year using Python you can use this function snippet:

from collections import namedtuple
from calendar import monthrange

Date = namedtuple("Date", ["year", "month", "day"])

def all_dates_in_year(year=2019):
    for month in range(1, 13): # Month is always 1..12
        for day in range(1, monthrange(year, month)[1] + 1):
            yield Date(year, month, day)

Usage example:

for date in all_dates_in_year(2019):
    print(date)

This will print (excerpt):

Date(year=2019, month=1, day=1)
Date(year=2019, month=1, day=2)
Date(year=2019, month=1, day=3)
[...]
Date(year=2019, month=2, day=27)
Date(year=2019, month=2, day=28)
Date(year=2019, month=3, day=1)
[...]
Date(year=2019, month=12, day=29)
Date(year=2019, month=12, day=30)
Date(year=2019, month=12, day=31)

You can also install my Python3 library UliEngineering using sudo pip3 install -U UliEngineering and then use all_dates_in_year() by importing it like this:

from UliEngineering.Utils.Date import *

 

Also see How to get number of days in month in Python

 

Posted by Uli Köhler in Python

How to get number of days in month in Python

If you want to find out how many days a month using Python use this snippet:

from calendar import monthrange

num_days = monthrange(2019, 2)[1] # num_days = 28

print(num_days) # Prints 28

The calendar module automatically takes into account leap years.

You can also use this function snippet:

from calendar import monthrange

def number_of_days_in_month(year=2019, month=2):
    return monthrange(year, month)[1]

# Usage example:
print(number_of_days_in_month(2019, 2)) # Prints 28

You can also install my Python3 library UliEngineering using sudo pip3 install -U UliEngineering and then use number_of_days_in_month() by importing it like this:

from UliEngineering.Utils.Date import *

 

Posted by Uli Köhler in Python

How to get unit/resolution of NumPy np.datetime64 object

NumPy’s datetime64 objects are represented as 64 bit integers (that’s what the 64 in the name means).

In order to find out what the resolution (e.g. us, ns etc) first install my Python 3 UliEngineering library using

sudo pip3 install -U UliEngineering

and then use datetime64_unit() from UliEngineering.Utils.NumPy like this:

from UliEngineering.Utils.NumPy import *

my_datetime = datetime64_now()
print(datetime64_resolution(my_datetime)) # Prints "us"
Posted by Uli Köhler in Python

How to get unit/resolution of NumPy np.timedelta64 object

In order to get the unit (e.g. us, ms, ns, …) of a np.timedelta64 object like

my_timedelta = np.timedelta64(625, 'us') # Unit is 'us'

first install my Python 3 UliEngineering library using

sudo pip3 install -U UliEngineering

and then use timedelta64_unit() from UliEngineering.Utils.NumPy like this:

import numpy as np
from UliEngineering.Utils.NumPy import *

my_timedelta = np.timedelta64(625, 'us')
print(timedelta64_resolution(my_timedelta)) # Prints "us"
Posted by Uli Köhler in Python

How to convert numpy timedelta (np.timedelta64) object to integer

If you have a NumPy np.timedelta64 object like

import numpy as np
my_timedelta = np.timedelta64(625, 'us')

a common task is to convert it to an integer.

This is often as easy as

my_timedelta.astype(int) # = 625, type: np.int64

This will give you the number (which is always an integer!) stored in the np.timedelta64 object, however it will ignore the unit  (e.g. us i.e. microseconds in our example).

To get the unit of the timedelta, first install UliEngineering using sudo pip3 install -U UliEngineering which you can then use like this:

import numpy as np
from UliEngineering.Utils.NumPy import *

my_timedelta = np.timedelta64(625, 'us')
print(timedelta64_resolution(my_timedelta)) # Prints "us"
Posted by Uli Köhler in Python

How to create a new numpy timedelta (np.timedelta64) object

There are two basic ways to create a new np.timedelta64 object:

Construct directly:

import numpy as np
my_timedelta = np.timedelta64(625, 'us')

Note that the constructor will only work with ints, i.e. np.timedelta64(625.5, 'us') won’t work. See

Construct as a difference between two np.datetime64 objects:

See How to get current datetime as numpy datetime (np.datetime64) for more details on how to get the current datetime as np.datetime64.

For any two np.datetime64 instances, we can compute the difference using

from datetime import datetime
import numpy as np

# There will be a minimal time difference between those two
dt1 = np.datetime64(datetime.now())
dt2 = np.datetime64(datetime.now())

my_timedelta = dt2 - dt1
print(my_timedelta) # For me, this prints "16 microseconds"
Posted by Uli Köhler in Python

How to get current datetime as numpy datetime (np.datetime64)

The simplest way of getting the current time as np.datetime64 object is to combine datetime.datetime.now() and the np.datetime64() constructor:

from datetime import datetime
import numpy as np

dt_now = np.datetime64(datetime.now())

print(dt_now)

Or use this function for convenience:

from datetime import datetime
import numpy as np

def datetime64_now():
    return np.datetime64(datetime.now())

dt_now = datetime64_now()
print(dt_now)

datetime64_now() is also available on my library UliEngineering.
Install using sudo pip3 install -U UliEngineering, then use like this:

from UliEngineering.Utils.NumPy import datetime64_now

dt_now = datetime64_now()
print(dt_now)
Posted by Uli Köhler in Python

How to list attributes of a Python object without __dict__

Usually you can list all attributes and functions of a Python object using __dir__:

from collections import namedtuple
nt = namedtuple("Foo", [])
print(nt.__dict__) # Prints attributes and functions of nt.

This can be very useful to find out e.g. which functions you can call on an object:

However for some objects like datetime.datetime this doesn’t work. Attempting to run

from datetime import datetime
dt = datetime.now()
print(dt.__dict__)

will result in

Traceback (most recent call last):
  File "test.py", line 3, in <module>
AttributeError: 'datetime.datetime' object has no attribute '__dict__'

So how can you find out which attributes your datetime.datetime object has and what function you can call on it?

Use dir():

from datetime import datetime
dt = datetime.now()
print(dir(dt))

This will print e.g.

['__add__', '__class__', '__delattr__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__ne__', '__new__', '__radd__', '__reduce__', '__reduce_ex__', '__repr__', '__rsub__', '__setattr__', '__sizeof__', '__str__', '__sub__', '__subclasshook__', 'astimezone', 'combine', 'ctime', 'date', 'day', 'dst', 'fold', 'fromordinal', 'fromtimestamp', 'hour', 'isocalendar', 'isoformat', 'isoweekday', 'max', 'microsecond', 'min', 'minute', 'month', 'now', 'replace', 'resolution', 'second', 'strftime', 'strptime', 'time', 'timestamp', 'timetuple', 'timetz', 'today', 'toordinal', 'tzinfo', 'tzname', 'utcfromtimestamp', 'utcnow', 'utcoffset', 'utctimetuple', 'weekday', 'year']

 

Posted by Uli Köhler in Python

How I fixed Nextcloud PHP Fatal error: Class contains 1 abstract method and must therefore be declared abstract

 

Recently my Nextcloud 16 instance (running via PHP 7.2 FPM on Nginx) return HTTP status 500 (internal server error) when trying to access it.

Analyzin the webserver log revealed this error message:

FastCGI sent in stderr: "PHP message: PHP Fatal error:  Class OC\Authentication\Token\PublicKeyToken contains 1 abstract method and must therefore be declared abstract or implement the remaining methods (OC\Authentication\Token\IToken::setExpires) in /var/sites/nextcloud.mydomain.com/lib/private/Authentication/Token/PublicKeyToken.php on line 47" while reading response header from upstream, client: 2003:ea:b747:ee00:7491:f492:480:57a9, server: nextcloud.mydomain.com, request: "PROPFIND /remote.php/dav/files/admin/ HTTP/1.1", upstream: "fastcgi://unix:/var/run/php/php7.2-fpm.sock:", host: "nextcloud.mydomain.com"

After some research, I found out that – at least in my case – the issue could be fixed by flushing the PHP opcache:

sudo service php7.2-fpm reload

After that, nextcloud started to work properly again.

Posted by Uli Köhler in PHP

Minimal argparse example in Python

Minimal example with one positional parameter (param):

#!/usr/bin/env python3
import argparse

if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("param")
    args = parser.parse_args()
    # Your code goes here!
    print(args.param)

Minimal example with one argument (-p, --param):

#!/usr/bin/env python3
import argparse

if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("-p", "--param")
    args = parser.parse_args()
    # Your code goes here!
    print(args.param)

 

Posted by Uli Köhler in Python

Minimal example for iterable classes in Python

Here’s a minimal example for a custom class that is iterable similar to a generator:

class MyIterable(object):
    def __init__(self):
        pass # ... Your code goes here!

    def __iter__(self):
        # Usually there is no need to modify this!
        return self

    def __next__(self):
        # To indicate no further values, use: raise StopIteration
        return "test" # Return the next value here. Return n

Usage example:

it = MyIterable()
print(next(it)) # Prints "test"
print(next(it)) # Prints "test"

In case you want to read more about how to create your own iterators, I recommend this tutorial.

Posted by Uli Köhler in Python

How to convert Celsius/Fahrenheit/Kelvin temperatures in Python using UliEngineering

In this example, we’ll use the UliEngineering library to convert between the three most commonly used units of temperature: Celsius (°C), Fahrenheit (°F) and Kelvin (K)

In order to install UliEngineering (a Python 3 library) run:

sudo pip3 install -U UliEngineering

We can now use the following functions from the UliEngineering.Physics.Temperature package to convert between the units of temperature:

  • celsius_to_kelvin(temperature)
  • kelvin_to_celsius(temperature)
  • fahrenheit_to_kelvin(temperature)
  • fahrenheit_to_celsius(temperature)

Tip: You can pass both numbers (like 120) or strings (like 120 °C or 0.01 °F) to most UliEngineering functions. SI prefixes like k and m are automatically decoded.

Example:

from UliEngineering.EngineerIO import auto_print
from UliEngineering.Physics.Temperature import *

# Convert to celsius and store in variable
temp = fahrenheit_to_celsius("120 °F") # temp = 48.88888888888897

# Automatically format value & print. Prints "48.9 °C"
auto_print(fahrenheit_to_celsius, "120 °F")

Additionally, UliEngineering provides normalize_temperature_celsius(temp, default_unit="°C") which takes a string and automatically recognizes the unit (if there is no unit given, the default_unit is assumed).

Examples:

  • normalize_temperature_celsius("120 °C") == 120.0
  • normalize_temperature_celsius("120") == 120.0
  • normalize_temperature_celsius(120) == 120.0
  • normalize_temperature_celsius("120 °F") == 48.88888888888897
  • normalize_temperature_celsius("120 K") == -153.14999999999998
  • auto_print(normalize_temperature_celsius, "120 K") prints -153.15 °C

Note that while °K is recognized by UliEngineering‘s functions, K is correctly used without a degree symbol.

Also there is normalize_temperature(temp, default_unit="°C") which is equivalent to normalize_temperature_celsius() except it returns the temperature in Kelvin instead of degrees Celsius.

Posted by Uli Köhler in Python

Computing crystal load capacitance using Python & UliEngineering

When you implement a crystal oscillator circuit, one task that is both essential and often overlooked is how to compute the load capacitors of the crystal.

Tip: In many cases it is easier to use an integrated crystal oscillator for which you don’t have to care about load capacitors. Crystals are usually recommendable for low-power applications and high-volume manufacturing where oscillators are too expensive.

In this example, we’ll compute the load capacitors for a Abracon ABLS-16.000MHZ-B4-T (specified with a load capacitance of 18 pF according to the datasheet).

It is paramount to understand that 18 pF load capacitance specification does not mean that you can use 18 pF capacitors or 2x 9 pF capacitors. You need to actually calculate the correct values.

You need the following information:

  • Load capacitance of the crystal, specified in the crystal’s datasheet.
  • An estimate of the stray capacitance, i.e. the capacitance of the PCB traces from the crystal to the microcontroller. If you don’t know this, I recommend to use 2 pF
  • The pin capacitance of the microcontroller (or whatever device you want to connect your device to) as specified in the microcontroller’s datasheet. Often the capacitance of oscillator pins is not the same as for other pins and is hence specified separately. 3 pF is a good first estimate for most modern ICs. The value must be specified per pin!

In order to install UliEngineering (a Python 3 library) run:

sudo pip3 install -U UliEngineering

Now we can compute the correct load capacitors using:

from UliEngineering.EngineerIO import auto_print
from UliEngineering.Electronics.Crystal import *

# Print load capacitors: prints "29.0 pF"
auto_print(load_capacitors, cload="18 pF", cpin="3 pF", cstray="2 pF")

In this case, you need to have two load capacitors of 29.0 pF each.

You should use the closest available value of load capacitor, but always check the resulting frequency if you have high clock accuracy requirements.

Tip: You can pass both numbers (like 18e-12) or strings (like 18 pF or 0.018 nF) to most UliEngineering functions. SI prefixes like p and n are automatically decoded

If you want to use the value programmatically, call load_capacitors() directly:

from UliEngineering.EngineerIO import auto_print
from UliEngineering.Electronics.Crystal import *

# Coompute the load capacitor value (for both of the two load caps): 
load_caps = load_capacitors(cload="18 pF", cpin="3 pF", cstray="2 pF")
# load_caps = 2.9e-11

 

Posted by Uli Köhler in Electronics, Python

Computing resistor power dissipation in Python using UliEngineering

In this example we’ll calculate the power dissipation of a 1 kΩ resistor with a constant current of 30 mA flowing through it, using our science and engineering Python library UliEngineering.

In order to install UliEngineering (a Python3-only library), run

sudo pip3 install -U UliEngineering

Now we can compute the resistor power dissipation using power_dissipated_in_resistor_by_current()

from UliEngineering.EngineerIO import auto_print
from UliEngineering.Electronics.Resistors import *
# Just compute the value:
power = power_dissipated_in_resistor_by_current("1 kΩ", "30 mA") # power = 0.9

# Print value: prints: prints "900 mW"
auto_print(power_dissipated_in_resistor_by_current, "1 kΩ", "30 mA")

Since the result is 900 mW, you can deduce that you need to use a resistor with a power rating of at least one Watt.

Note that you can pass both numbers (like 0.03) or strings (like 30 mA or 0.03 A) to most UliEngineering functions. SI prefixes like k and M are automatically decoded

If you know the voltage across the resistor, you can use power_dissipated_in_resistor_by_voltage(). Let’s assume there is 1V dropped across the resistor:

from UliEngineering.EngineerIO import auto_print
from UliEngineering.Electronics.Resistors import *
# Just compute the value:
power = power_dissipated_in_resistor_by_voltage("1 kΩ", "30 mA") # power = 0.001

# Print value: prints: prints "1.00 mW"
auto_print(power_dissipated_in_resistor_by_voltage, "1 kΩ", "30 mA")

So in this case the power dissipation is extremely low – only 1.00 mW – and won’t matter for most practical applications.

In many cases, you can also pass NumPy arrays to UliEngineering functions:

from UliEngineering.EngineerIO import format_value
from UliEngineering.Electronics.Resistors import *
import numpy as np

# Compute the value:
resistors = np.asarray([100, 500, 1000]) # 100 Ω, 500 Ω, 1 kΩ
power = power_dissipated_in_resistor_by_voltage(resistors, "30 mA") # power = 0.001

# power = [9.0e-06 1.8e-06 9.0e-07]

 

Posted by Uli Köhler in Electronics, Python

What does ‘lsusb’ do on Linux?

lsusb lists all the USB devices to your computer.

Example output:

Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 003: ID 8087:0a2b Intel Corp. 
Bus 001 Device 002: ID 04f2:b5e8 Chicony Electronics Co., Ltd 
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

Each line represents a single USB device – but an USB device might also be internal to the computer and not connected to a physical USB port. For example, Bus 001 Device 003: ID 8087:0a2b Intel Corp. is an internal Bluetooth modem in my notebook.

Explanation:

  • Bus 001: The device is connected to the first USB bus (001) on the system
  • Device 003: In the aforementioned bus, the device has been assigned the ID 003.
  • ID 8087:0a2b: This is the ID assigned to that device. The first part (8087) is the Vendor ID (VID). The second part (0a2b) is the Product ID (PID). These IDs are assigned by the USB Implementer’s forum. This ID will always stay the same for any given device. If you buy a USB product multiple times, all of them will have the same ID.
  • Intel Corp.: This is the description of the device. Often Linux can show you more information, but in this case it’s just the name of the manufacturer.
Posted by Uli Köhler in Linux

How to fix ‘error: option –rednose not recognized’

Problem:

You want to test a Python package e.g. using python setup.py test.  However, you see the following error message:

usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
   or: setup.py --help [cmd1 cmd2 ...]
   or: setup.py --help-commands
   or: setup.py cmd --help

error: option --rednose not recognized

Solution:

Rednose is a Python package that you need to install:

sudo pip3 install rednose

or, if you don’t have pip3, try

sudo pip install rednose
Posted by Uli Köhler in Python