Programming languages

How to install NodeJS 12.x on Ubuntu in 1 minute

Run these shell commands on your Ubuntu computer to install NodeJS 12.x:

curl -sL https://deb.nodesource.com/setup_12.x | sudo -E bash -
sudo apt-get install -y nodejs

Instead of setup_12.x you can also choose other versions like setup_8.x. However, using this method, you can’t install multiple versions of NodeJS in parallel.

Source: Official nodesource documentation

Posted by Uli Köhler in Linux, NodeJS

How to iterate all days of year using Python

To iterate all days in a year using Python you can use this function snippet:

from collections import namedtuple
from calendar import monthrange

Date = namedtuple("Date", ["year", "month", "day"])

def all_dates_in_year(year=2019):
    for month in range(1, 13): # Month is always 1..12
        for day in range(1, monthrange(year, month)[1] + 1):
            yield Date(year, month, day)

Usage example:

for date in all_dates_in_year(2019):
    print(date)

This will print (excerpt):

Date(year=2019, month=1, day=1)
Date(year=2019, month=1, day=2)
Date(year=2019, month=1, day=3)
[...]
Date(year=2019, month=2, day=27)
Date(year=2019, month=2, day=28)
Date(year=2019, month=3, day=1)
[...]
Date(year=2019, month=12, day=29)
Date(year=2019, month=12, day=30)
Date(year=2019, month=12, day=31)

You can also install my Python3 library UliEngineering using sudo pip3 install -U UliEngineering and then use all_dates_in_year() by importing it like this:

from UliEngineering.Utils.Date import *

 

Also see How to get number of days in month in Python

 

Posted by Uli Köhler in Python

How to get number of days in month in Python

If you want to find out how many days a month using Python use this snippet:

from calendar import monthrange

num_days = monthrange(2019, 2)[1] # num_days = 28

print(num_days) # Prints 28

The calendar module automatically takes into account leap years.

You can also use this function snippet:

from calendar import monthrange

def number_of_days_in_month(year=2019, month=2):
    return monthrange(year, month)[1]

# Usage example:
print(number_of_days_in_month(2019, 2)) # Prints 28

You can also install my Python3 library UliEngineering using sudo pip3 install -U UliEngineering and then use number_of_days_in_month() by importing it like this:

from UliEngineering.Utils.Date import *

 

Posted by Uli Köhler in Python

How to get unit/resolution of NumPy np.datetime64 object

NumPy’s datetime64 objects are represented as 64 bit integers (that’s what the 64 in the name means).

In order to find out what the resolution (e.g. us, ns etc) first install my Python 3 UliEngineering library using

sudo pip3 install -U UliEngineering

and then use datetime64_unit() from UliEngineering.Utils.NumPy like this:

from UliEngineering.Utils.NumPy import *

my_datetime = datetime64_now()
print(datetime64_resolution(my_datetime)) # Prints "us"
Posted by Uli Köhler in Python

How to get unit/resolution of NumPy np.timedelta64 object

In order to get the unit (e.g. us, ms, ns, …) of a np.timedelta64 object like

my_timedelta = np.timedelta64(625, 'us') # Unit is 'us'

first install my Python 3 UliEngineering library using

sudo pip3 install -U UliEngineering

and then use timedelta64_unit() from UliEngineering.Utils.NumPy like this:

import numpy as np
from UliEngineering.Utils.NumPy import *

my_timedelta = np.timedelta64(625, 'us')
print(timedelta64_resolution(my_timedelta)) # Prints "us"
Posted by Uli Köhler in Python

How to convert numpy timedelta (np.timedelta64) object to integer

If you have a NumPy np.timedelta64 object like

import numpy as np
my_timedelta = np.timedelta64(625, 'us')

a common task is to convert it to an integer.

This is often as easy as

my_timedelta.astype(int) # = 625, type: np.int64

This will give you the number (which is always an integer!) stored in the np.timedelta64 object, however it will ignore the unit  (e.g. us i.e. microseconds in our example).

To get the unit of the timedelta, first install UliEngineering using sudo pip3 install -U UliEngineering which you can then use like this:

import numpy as np
from UliEngineering.Utils.NumPy import *

my_timedelta = np.timedelta64(625, 'us')
print(timedelta64_resolution(my_timedelta)) # Prints "us"
Posted by Uli Köhler in Python

How to create a new numpy timedelta (np.timedelta64) object

There are two basic ways to create a new np.timedelta64 object:

Construct directly:

import numpy as np
my_timedelta = np.timedelta64(625, 'us')

Note that the constructor will only work with ints, i.e. np.timedelta64(625.5, 'us') won’t work. See

Construct as a difference between two np.datetime64 objects:

See How to get current datetime as numpy datetime (np.datetime64) for more details on how to get the current datetime as np.datetime64.

For any two np.datetime64 instances, we can compute the difference using

from datetime import datetime
import numpy as np

# There will be a minimal time difference between those two
dt1 = np.datetime64(datetime.now())
dt2 = np.datetime64(datetime.now())

my_timedelta = dt2 - dt1
print(my_timedelta) # For me, this prints "16 microseconds"
Posted by Uli Köhler in Python

How to get current datetime as numpy datetime (np.datetime64)

The simplest way of getting the current time as np.datetime64 object is to combine datetime.datetime.now() and the np.datetime64() constructor:

from datetime import datetime
import numpy as np

dt_now = np.datetime64(datetime.now())

print(dt_now)

Or use this function for convenience:

from datetime import datetime
import numpy as np

def datetime64_now():
    return np.datetime64(datetime.now())

dt_now = datetime64_now()
print(dt_now)

datetime64_now() is also available on my library UliEngineering.
Install using sudo pip3 install -U UliEngineering, then use like this:

from UliEngineering.Utils.NumPy import datetime64_now

dt_now = datetime64_now()
print(dt_now)
Posted by Uli Köhler in Python

How to list attributes of a Python object without __dict__

Usually you can list all attributes and functions of a Python object using __dir__:

from collections import namedtuple
nt = namedtuple("Foo", [])
print(nt.__dict__) # Prints attributes and functions of nt.

This can be very useful to find out e.g. which functions you can call on an object:

However for some objects like datetime.datetime this doesn’t work. Attempting to run

from datetime import datetime
dt = datetime.now()
print(dt.__dict__)

will result in

Traceback (most recent call last):
  File "test.py", line 3, in <module>
AttributeError: 'datetime.datetime' object has no attribute '__dict__'

So how can you find out which attributes your datetime.datetime object has and what function you can call on it?

Use dir():

from datetime import datetime
dt = datetime.now()
print(dir(dt))

This will print e.g.

['__add__', '__class__', '__delattr__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__ne__', '__new__', '__radd__', '__reduce__', '__reduce_ex__', '__repr__', '__rsub__', '__setattr__', '__sizeof__', '__str__', '__sub__', '__subclasshook__', 'astimezone', 'combine', 'ctime', 'date', 'day', 'dst', 'fold', 'fromordinal', 'fromtimestamp', 'hour', 'isocalendar', 'isoformat', 'isoweekday', 'max', 'microsecond', 'min', 'minute', 'month', 'now', 'replace', 'resolution', 'second', 'strftime', 'strptime', 'time', 'timestamp', 'timetuple', 'timetz', 'today', 'toordinal', 'tzinfo', 'tzname', 'utcfromtimestamp', 'utcnow', 'utcoffset', 'utctimetuple', 'weekday', 'year']

 

Posted by Uli Köhler in Python

How I fixed Nextcloud PHP Fatal error: Class contains 1 abstract method and must therefore be declared abstract

 

Recently my Nextcloud 16 instance (running via PHP 7.2 FPM on Nginx) return HTTP status 500 (internal server error) when trying to access it.

Analyzin the webserver log revealed this error message:

FastCGI sent in stderr: "PHP message: PHP Fatal error:  Class OC\Authentication\Token\PublicKeyToken contains 1 abstract method and must therefore be declared abstract or implement the remaining methods (OC\Authentication\Token\IToken::setExpires) in /var/sites/nextcloud.mydomain.com/lib/private/Authentication/Token/PublicKeyToken.php on line 47" while reading response header from upstream, client: 2003:ea:b747:ee00:7491:f492:480:57a9, server: nextcloud.mydomain.com, request: "PROPFIND /remote.php/dav/files/admin/ HTTP/1.1", upstream: "fastcgi://unix:/var/run/php/php7.2-fpm.sock:", host: "nextcloud.mydomain.com"

After some research, I found out that – at least in my case – the issue could be fixed by flushing the PHP opcache:

sudo service php7.2-fpm reload

After that, nextcloud started to work properly again.

Posted by Uli Köhler in PHP

Minimal argparse example in Python

Minimal example with one positional parameter (param):

#!/usr/bin/env python3
import argparse

if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("param")
    args = parser.parse_args()
    # Your code goes here!
    print(args.param)

Minimal example with one argument (-p, --param):

#!/usr/bin/env python3
import argparse

if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("-p", "--param")
    args = parser.parse_args()
    # Your code goes here!
    print(args.param)

 

Posted by Uli Köhler in Python

Minimal example for iterable classes in Python

Here’s a minimal example for a custom class that is iterable similar to a generator:

class MyIterable(object):
    def __init__(self):
        pass # ... Your code goes here!

    def __iter__(self):
        # Usually there is no need to modify this!
        return self

    def __next__(self):
        # To indicate no further values, use: raise StopIteration
        return "test" # Return the next value here. Return n

Usage example:

it = MyIterable()
print(next(it)) # Prints "test"
print(next(it)) # Prints "test"

In case you want to read more about how to create your own iterators, I recommend this tutorial.

Posted by Uli Köhler in Python

How to convert Celsius/Fahrenheit/Kelvin temperatures in Python using UliEngineering

In this example, we’ll use the UliEngineering library to convert between the three most commonly used units of temperature: Celsius (°C), Fahrenheit (°F) and Kelvin (K)

In order to install UliEngineering (a Python 3 library) run:

sudo pip3 install -U UliEngineering

We can now use the following functions from the UliEngineering.Physics.Temperature package to convert between the units of temperature:

  • celsius_to_kelvin(temperature)
  • kelvin_to_celsius(temperature)
  • fahrenheit_to_kelvin(temperature)
  • fahrenheit_to_celsius(temperature)

Tip: You can pass both numbers (like 120) or strings (like 120 °C or 0.01 °F) to most UliEngineering functions. SI prefixes like k and m are automatically decoded.

Example:

from UliEngineering.EngineerIO import auto_print
from UliEngineering.Physics.Temperature import *

# Convert to celsius and store in variable
temp = fahrenheit_to_celsius("120 °F") # temp = 48.88888888888897

# Automatically format value & print. Prints "48.9 °C"
auto_print(fahrenheit_to_celsius, "120 °F")

Additionally, UliEngineering provides normalize_temperature_celsius(temp, default_unit="°C") which takes a string and automatically recognizes the unit (if there is no unit given, the default_unit is assumed).

Examples:

  • normalize_temperature_celsius("120 °C") == 120.0
  • normalize_temperature_celsius("120") == 120.0
  • normalize_temperature_celsius(120) == 120.0
  • normalize_temperature_celsius("120 °F") == 48.88888888888897
  • normalize_temperature_celsius("120 K") == -153.14999999999998
  • auto_print(normalize_temperature_celsius, "120 K") prints -153.15 °C

Note that while °K is recognized by UliEngineering‘s functions, K is correctly used without a degree symbol.

Also there is normalize_temperature(temp, default_unit="°C") which is equivalent to normalize_temperature_celsius() except it returns the temperature in Kelvin instead of degrees Celsius.

Posted by Uli Köhler in Python

Computing crystal load capacitance using Python & UliEngineering

When you implement a crystal oscillator circuit, one task that is both essential and often overlooked is how to compute the load capacitors of the crystal.

Tip: In many cases it is easier to use an integrated crystal oscillator for which you don’t have to care about load capacitors. Crystals are usually recommendable for low-power applications and high-volume manufacturing where oscillators are too expensive.

In this example, we’ll compute the load capacitors for a Abracon ABLS-16.000MHZ-B4-T (specified with a load capacitance of 18 pF according to the datasheet).

It is paramount to understand that 18 pF load capacitance specification does not mean that you can use 18 pF capacitors or 2x 9 pF capacitors. You need to actually calculate the correct values.

You need the following information:

  • Load capacitance of the crystal, specified in the crystal’s datasheet.
  • An estimate of the stray capacitance, i.e. the capacitance of the PCB traces from the crystal to the microcontroller. If you don’t know this, I recommend to use 2 pF
  • The pin capacitance of the microcontroller (or whatever device you want to connect your device to) as specified in the microcontroller’s datasheet. Often the capacitance of oscillator pins is not the same as for other pins and is hence specified separately. 3 pF is a good first estimate for most modern ICs. The value must be specified per pin!

In order to install UliEngineering (a Python 3 library) run:

sudo pip3 install -U UliEngineering

Now we can compute the correct load capacitors using:

from UliEngineering.EngineerIO import auto_print
from UliEngineering.Electronics.Crystal import *

# Print load capacitors: prints "29.0 pF"
auto_print(load_capacitors, cload="18 pF", cpin="3 pF", cstray="2 pF")

In this case, you need to have two load capacitors of 29.0 pF each.

You should use the closest available value of load capacitor, but always check the resulting frequency if you have high clock accuracy requirements.

Tip: You can pass both numbers (like 18e-12) or strings (like 18 pF or 0.018 nF) to most UliEngineering functions. SI prefixes like p and n are automatically decoded

If you want to use the value programmatically, call load_capacitors() directly:

from UliEngineering.EngineerIO import auto_print
from UliEngineering.Electronics.Crystal import *

# Coompute the load capacitor value (for both of the two load caps): 
load_caps = load_capacitors(cload="18 pF", cpin="3 pF", cstray="2 pF")
# load_caps = 2.9e-11

 

Posted by Uli Köhler in Electronics, Python

Computing resistor power dissipation in Python using UliEngineering

In this example we’ll calculate the power dissipation of a 1 kΩ resistor with a constant current of 30 mA flowing through it, using our science and engineering Python library UliEngineering.

In order to install UliEngineering (a Python3-only library), run

sudo pip3 install -U UliEngineering

Now we can compute the resistor power dissipation using power_dissipated_in_resistor_by_current()

from UliEngineering.EngineerIO import auto_print
from UliEngineering.Electronics.Resistors import *
# Just compute the value:
power = power_dissipated_in_resistor_by_current("1 kΩ", "30 mA") # power = 0.9

# Print value: prints: prints "900 mW"
auto_print(power_dissipated_in_resistor_by_current, "1 kΩ", "30 mA")

Since the result is 900 mW, you can deduce that you need to use a resistor with a power rating of at least one Watt.

Note that you can pass both numbers (like 0.03) or strings (like 30 mA or 0.03 A) to most UliEngineering functions. SI prefixes like k and M are automatically decoded

If you know the voltage across the resistor, you can use power_dissipated_in_resistor_by_voltage(). Let’s assume there is 1V dropped across the resistor:

from UliEngineering.EngineerIO import auto_print
from UliEngineering.Electronics.Resistors import *
# Just compute the value:
power = power_dissipated_in_resistor_by_voltage("1 kΩ", "30 mA") # power = 0.001

# Print value: prints: prints "1.00 mW"
auto_print(power_dissipated_in_resistor_by_voltage, "1 kΩ", "30 mA")

So in this case the power dissipation is extremely low – only 1.00 mW – and won’t matter for most practical applications.

In many cases, you can also pass NumPy arrays to UliEngineering functions:

from UliEngineering.EngineerIO import format_value
from UliEngineering.Electronics.Resistors import *
import numpy as np

# Compute the value:
resistors = np.asarray([100, 500, 1000]) # 100 Ω, 500 Ω, 1 kΩ
power = power_dissipated_in_resistor_by_voltage(resistors, "30 mA") # power = 0.001

# power = [9.0e-06 1.8e-06 9.0e-07]

 

Posted by Uli Köhler in Electronics, Python

How to fix ‘error: option –rednose not recognized’

Problem:

You want to test a Python package e.g. using python setup.py test.  However, you see the following error message:

usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
   or: setup.py --help [cmd1 cmd2 ...]
   or: setup.py --help-commands
   or: setup.py cmd --help

error: option --rednose not recognized

Solution:

Rednose is a Python package that you need to install:

sudo pip3 install rednose

or, if you don’t have pip3, try

sudo pip install rednose
Posted by Uli Köhler in Python

ElasticSearch: How to iterate / scroll through all documents in index

In ElasticSearch, you can use the Scroll API to scroll through all documents in an entire index.

In Python you can scroll like this:

def es_iterate_all_documents(es, index, pagesize=250, scroll_timeout="1m", **kwargs):
    """
    Helper to iterate ALL values from a single index
    Yields all the documents.
    """
    is_first = True
    while True:
        # Scroll next
        if is_first: # Initialize scroll
            result = es.search(index=index, scroll="1m", **kwargs, body={
                "size": pagesize
            })
            is_first = False
        else:
            result = es.scroll(body={
                "scroll_id": scroll_id,
                "scroll": scroll_timeout
            })
        scroll_id = result["_scroll_id"]
        hits = result["hits"]["hits"]
        # Stop after no more docs
        if not hits:
            break
        # Yield each entry
        yield from (hit['_source'] for hit in hits)

This function will yield each document encountered in the index.

Example usage for index my_index:

es = Elasticsearch([{"host": "localhost"}])

for entry in es_iterate_all_documents(es, 'my_index'):
    print(entry) # Prints the document as stored in the DB

 

Posted by Uli Köhler in Databases, ElasticSearch, Python

How to build package or program using maven

In order to build a package using maven, run

mvn package

If you also want to install the maven package in the local maven repository (usually in ~/.m2), use

mvn install

In case you don’t have Maven installed, you can install it on Ubuntu using

sudo apt install maven

 

Posted by Uli Köhler in Build systems, Java

How to fix ‘cannot find symbol’ in Java

Problem:

You want to compile some Java source code (e.g. using Maven), but you get an error message like

[ERROR] /home/user/myproject/src/main/java/com/mydomain/myproject/Main.java:[123,40] cannot find symbol
[ERROR] symbol: class MyClass

Solution:

Java does not know where to find MyClass.

First, check if you have imported MyClass correctly.

If MyClass can be found in your library, you most likely are missing an import statement at the top of the file where the error occurs.

If, on the other hand, MyClass is imported from an external library, check if:

  • You have the correct version of the library. Possibly, you are using an old version of the library where MyClass is not present. This is often the case when using SNAPSHOT versions in maven since different developers might have different SNAPSHOTs so one might have issue building while another might not.
  • You are using the correct import statement (refer to the docs or the source code of the library to check if you are using the correct package.

If you don’t know in which library you can find a certain symbol in, see our post [To be done].

What exactly are ‘symbols’?

The concept and term of a symbol is used in many different programming languages.
Basically it means ‘a name that refers to something declared somewhere else in more detail’.

Therefore, if you encounter error messages like ‘cannot find symbol’, the compiler is trying to tell you: “I don’t know what that name refers to”.

Example:

When you declare a class in Java, e.g.

class MyClass {
    /* Your code goes here ! */
}

you can later refer to that class by its name, MyClass, e.g. in

MyClass class = new MyClass();

In that line of code, MyClass is used symbolically to refer to the full class declaration (class MyClass { /* ... */}) which we listed before.

Hence, MyClass is used as a symbol to refer to your previous (full) declaration of MyClass.

If the name MyClass, however, has no associated full declaration, the compiler will tell you ‘cannot find symbol’.

Posted by Uli Köhler in Java

ElasticSearch: How to iterate all documents in index using Python (up to 10000 documents)

Important Note: This simple approach only works for up to ~10000 documents. Prefer using our scroll-based solution: See ElasticSearch: How to iterate / scroll through all documents in index

Use this helper function to iterate over all the documens in an index

def es_iterate_all_documents(es, index, pagesize=250, **kwargs):
    """
    Helper to iterate ALL values from
    Yields all the documents.
    """
    offset = 0
    while True:
        result = es.search(index=index, **kwargs, body={
            "size": pagesize,
            "from": offset
        })
        hits = result["hits"]["hits"]
        # Stop after no more docs
        if not hits:
            break
        # Yield each entry
        yield from (hit['_source'] for hit in hits)
        # Continue from there
        offset += pagesize

Usage example:

for entry in es_iterate_all_documents(es, 'my_index'):
    print(entry) # Prints the document as stored in the DB

How it works

You can iterate over all documents in an index in ElasticSearch by using queries like

{
    "size": 250,
    "from": 0
}

and increasing "from" by "size" after each iteration.

Posted by Uli Köhler in Databases, ElasticSearch, Python