Programming languages

How to get current datetime as numpy datetime (np.datetime64)

The simplest way of getting the current time as np.datetime64 object is to combine datetime.datetime.now() and the np.datetime64() constructor:

from datetime import datetime
import numpy as np

dt_now = np.datetime64(datetime.now())

print(dt_now)

Or use this function for convenience:

from datetime import datetime
import numpy as np

def datetime64_now():
    return np.datetime64(datetime.now())

dt_now = datetime64_now()
print(dt_now)

datetime64_now() is also available on my library UliEngineering.
Install using sudo pip3 install -U UliEngineering, then use like this:

from UliEngineering.Utils.NumPy import datetime64_now

dt_now = datetime64_now()
print(dt_now)
Posted by Uli Köhler in Python

How to list attributes of a Python object without __dict__

Usually you can list all attributes and functions of a Python object using __dir__:

from collections import namedtuple
nt = namedtuple("Foo", [])
print(nt.__dict__) # Prints attributes and functions of nt.

This can be very useful to find out e.g. which functions you can call on an object:

However for some objects like datetime.datetime this doesn’t work. Attempting to run

from datetime import datetime
dt = datetime.now()
print(dt.__dict__)

will result in

Traceback (most recent call last):
  File "test.py", line 3, in <module>
AttributeError: 'datetime.datetime' object has no attribute '__dict__'

So how can you find out which attributes your datetime.datetime object has and what function you can call on it?

Use dir():

from datetime import datetime
dt = datetime.now()
print(dir(dt))

This will print e.g.

['__add__', '__class__', '__delattr__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__ne__', '__new__', '__radd__', '__reduce__', '__reduce_ex__', '__repr__', '__rsub__', '__setattr__', '__sizeof__', '__str__', '__sub__', '__subclasshook__', 'astimezone', 'combine', 'ctime', 'date', 'day', 'dst', 'fold', 'fromordinal', 'fromtimestamp', 'hour', 'isocalendar', 'isoformat', 'isoweekday', 'max', 'microsecond', 'min', 'minute', 'month', 'now', 'replace', 'resolution', 'second', 'strftime', 'strptime', 'time', 'timestamp', 'timetuple', 'timetz', 'today', 'toordinal', 'tzinfo', 'tzname', 'utcfromtimestamp', 'utcnow', 'utcoffset', 'utctimetuple', 'weekday', 'year']

 

Posted by Uli Köhler in Python

How I fixed Nextcloud PHP Fatal error: Class contains 1 abstract method and must therefore be declared abstract

 

Recently my Nextcloud 16 instance (running via PHP 7.2 FPM on Nginx) return HTTP status 500 (internal server error) when trying to access it.

Analyzin the webserver log revealed this error message:

FastCGI sent in stderr: "PHP message: PHP Fatal error:  Class OC\Authentication\Token\PublicKeyToken contains 1 abstract method and must therefore be declared abstract or implement the remaining methods (OC\Authentication\Token\IToken::setExpires) in /var/sites/nextcloud.mydomain.com/lib/private/Authentication/Token/PublicKeyToken.php on line 47" while reading response header from upstream, client: 2003:ea:b747:ee00:7491:f492:480:57a9, server: nextcloud.mydomain.com, request: "PROPFIND /remote.php/dav/files/admin/ HTTP/1.1", upstream: "fastcgi://unix:/var/run/php/php7.2-fpm.sock:", host: "nextcloud.mydomain.com"

After some research, I found out that – at least in my case – the issue could be fixed by flushing the PHP opcache:

sudo service php7.2-fpm reload

After that, nextcloud started to work properly again.

Posted by Uli Köhler in Nextcloud, PHP

Minimal argparse example in Python

Minimal example with one positional parameter (param):

#!/usr/bin/env python3
import argparse

if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("param")
    args = parser.parse_args()
    # Your code goes here!
    print(args.param)

Minimal example with one argument (-p, --param):

#!/usr/bin/env python3
import argparse

if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("-p", "--param")
    args = parser.parse_args()
    # Your code goes here!
    print(args.param)

 

Posted by Uli Köhler in Python

Minimal example for iterable classes in Python

Here’s a minimal example for a custom class that is iterable similar to a generator:

class MyIterable(object):
    def __init__(self):
        pass # ... Your code goes here!

    def __iter__(self):
        # Usually there is no need to modify this!
        return self

    def __next__(self):
        # To indicate no further values, use: raise StopIteration
        return "test" # Return the next value here. Return n

Usage example:

it = MyIterable()
print(next(it)) # Prints "test"
print(next(it)) # Prints "test"

In case you want to read more about how to create your own iterators, I recommend this tutorial.

Posted by Uli Köhler in Python

How to convert Celsius/Fahrenheit/Kelvin temperatures in Python using UliEngineering

In this example, we’ll use the UliEngineering library to convert between the three most commonly used units of temperature: Celsius (°C), Fahrenheit (°F) and Kelvin (K)

In order to install UliEngineering (a Python 3 library) run:

sudo pip3 install -U UliEngineering

We can now use the following functions from the UliEngineering.Physics.Temperature package to convert between the units of temperature:

  • celsius_to_kelvin(temperature)
  • kelvin_to_celsius(temperature)
  • fahrenheit_to_kelvin(temperature)
  • fahrenheit_to_celsius(temperature)

Tip: You can pass both numbers (like 120) or strings (like 120 °C or 0.01 °F) to most UliEngineering functions. SI prefixes like k and m are automatically decoded.

Example:

from UliEngineering.EngineerIO import auto_print
from UliEngineering.Physics.Temperature import *

# Convert to celsius and store in variable
temp = fahrenheit_to_celsius("120 °F") # temp = 48.88888888888897

# Automatically format value & print. Prints "48.9 °C"
auto_print(fahrenheit_to_celsius, "120 °F")

Additionally, UliEngineering provides normalize_temperature_celsius(temp, default_unit="°C") which takes a string and automatically recognizes the unit (if there is no unit given, the default_unit is assumed).

Examples:

  • normalize_temperature_celsius("120 °C") == 120.0
  • normalize_temperature_celsius("120") == 120.0
  • normalize_temperature_celsius(120) == 120.0
  • normalize_temperature_celsius("120 °F") == 48.88888888888897
  • normalize_temperature_celsius("120 K") == -153.14999999999998
  • auto_print(normalize_temperature_celsius, "120 K") prints -153.15 °C

Note that while °K is recognized by UliEngineering‘s functions, K is correctly used without a degree symbol.

Also there is normalize_temperature(temp, default_unit="°C") which is equivalent to normalize_temperature_celsius() except it returns the temperature in Kelvin instead of degrees Celsius.

Posted by Uli Köhler in Python

Computing crystal load capacitance using Python & UliEngineering

When you implement a crystal oscillator circuit, one task that is both essential and often overlooked is how to compute the load capacitors of the crystal.

Tip: In many cases it is easier to use an integrated crystal oscillator for which you don’t have to care about load capacitors. Crystals are usually recommendable for low-power applications and high-volume manufacturing where oscillators are too expensive.

In this example, we’ll compute the load capacitors for a Abracon ABLS-16.000MHZ-B4-T (specified with a load capacitance of 18 pF according to the datasheet).

It is paramount to understand that 18 pF load capacitance specification does not mean that you can use 18 pF capacitors or 2x 9 pF capacitors. You need to actually calculate the correct values.

You need the following information:

  • Load capacitance of the crystal, specified in the crystal’s datasheet.
  • An estimate of the stray capacitance, i.e. the capacitance of the PCB traces from the crystal to the microcontroller. If you don’t know this, I recommend to use 2 pF
  • The pin capacitance of the microcontroller (or whatever device you want to connect your device to) as specified in the microcontroller’s datasheet. Often the capacitance of oscillator pins is not the same as for other pins and is hence specified separately. 3 pF is a good first estimate for most modern ICs. The value must be specified per pin!

In order to install UliEngineering (a Python 3 library) run:

sudo pip3 install -U UliEngineering

Now we can compute the correct load capacitors using:

from UliEngineering.EngineerIO import auto_print
from UliEngineering.Electronics.Crystal import *

# Print load capacitors: prints "29.0 pF"
auto_print(load_capacitors, cload="18 pF", cpin="3 pF", cstray="2 pF")

In this case, you need to have two load capacitors of 29.0 pF each.

You should use the closest available value of load capacitor, but always check the resulting frequency if you have high clock accuracy requirements.

Tip: You can pass both numbers (like 18e-12) or strings (like 18 pF or 0.018 nF) to most UliEngineering functions. SI prefixes like p and n are automatically decoded

If you want to use the value programmatically, call load_capacitors() directly:

from UliEngineering.EngineerIO import auto_print
from UliEngineering.Electronics.Crystal import *

# Coompute the load capacitor value (for both of the two load caps): 
load_caps = load_capacitors(cload="18 pF", cpin="3 pF", cstray="2 pF")
# load_caps = 2.9e-11

 

Posted by Uli Köhler in Electronics, Python

Computing resistor power dissipation in Python using UliEngineering

In this example we’ll calculate the power dissipation of a 1 kΩ resistor with a constant current of 30 mA flowing through it, using our science and engineering Python library UliEngineering.

In order to install UliEngineering (a Python3-only library), run

sudo pip3 install -U UliEngineering

Now we can compute the resistor power dissipation using power_dissipated_in_resistor_by_current()

from UliEngineering.EngineerIO import auto_print
from UliEngineering.Electronics.Resistors import *
# Just compute the value:
power = power_dissipated_in_resistor_by_current("1 kΩ", "30 mA") # power = 0.9

# Print value: prints: prints "900 mW"
auto_print(power_dissipated_in_resistor_by_current, "1 kΩ", "30 mA")

Since the result is 900 mW, you can deduce that you need to use a resistor with a power rating of at least one Watt.

Note that you can pass both numbers (like 0.03) or strings (like 30 mA or 0.03 A) to most UliEngineering functions. SI prefixes like k and M are automatically decoded

If you know the voltage across the resistor, you can use power_dissipated_in_resistor_by_voltage(). Let’s assume there is 1V dropped across the resistor:

from UliEngineering.EngineerIO import auto_print
from UliEngineering.Electronics.Resistors import *
# Just compute the value:
power = power_dissipated_in_resistor_by_voltage("1 kΩ", "30 mA") # power = 0.001

# Print value: prints: prints "1.00 mW"
auto_print(power_dissipated_in_resistor_by_voltage, "1 kΩ", "30 mA")

So in this case the power dissipation is extremely low – only 1.00 mW – and won’t matter for most practical applications.

In many cases, you can also pass NumPy arrays to UliEngineering functions:

from UliEngineering.EngineerIO import format_value
from UliEngineering.Electronics.Resistors import *
import numpy as np

# Compute the value:
resistors = np.asarray([100, 500, 1000]) # 100 Ω, 500 Ω, 1 kΩ
power = power_dissipated_in_resistor_by_voltage(resistors, "30 mA") # power = 0.001

# power = [9.0e-06 1.8e-06 9.0e-07]

 

Posted by Uli Köhler in Electronics, Python

How to fix ‘error: option –rednose not recognized’

Problem:

You want to test a Python package e.g. using python setup.py test.  However, you see the following error message:

usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
   or: setup.py --help [cmd1 cmd2 ...]
   or: setup.py --help-commands
   or: setup.py cmd --help

error: option --rednose not recognized

Solution:

Rednose is a Python package that you need to install:

sudo pip3 install rednose

or, if you don’t have pip3, try

sudo pip install rednose
Posted by Uli Köhler in Python

ElasticSearch: How to iterate / scroll through all documents in index

In ElasticSearch, you can use the Scroll API to scroll through all documents in an entire index.

In Python you can scroll like this:

def es_iterate_all_documents(es, index, pagesize=250, scroll_timeout="1m", **kwargs):
    """
    Helper to iterate ALL values from a single index
    Yields all the documents.
    """
    is_first = True
    while True:
        # Scroll next
        if is_first: # Initialize scroll
            result = es.search(index=index, scroll="1m", **kwargs, body={
                "size": pagesize
            })
            is_first = False
        else:
            result = es.scroll(body={
                "scroll_id": scroll_id,
                "scroll": scroll_timeout
            })
        scroll_id = result["_scroll_id"]
        hits = result["hits"]["hits"]
        # Stop after no more docs
        if not hits:
            break
        # Yield each entry
        yield from (hit['_source'] for hit in hits)

This function will yield each document encountered in the index.

Example usage for index my_index:

es = Elasticsearch([{"host": "localhost"}])

for entry in es_iterate_all_documents(es, 'my_index'):
    print(entry) # Prints the document as stored in the DB

 

Posted by Uli Köhler in Databases, ElasticSearch, Python

How to build package or program using maven

In order to build a package using maven, run

mvn package

If you also want to install the maven package in the local maven repository (usually in ~/.m2), use

mvn install

In case you don’t have Maven installed, you can install it on Ubuntu using

sudo apt install maven

 

Posted by Uli Köhler in Build systems, Java

How to fix ‘cannot find symbol’ in Java

Problem:

You want to compile some Java source code (e.g. using Maven), but you get an error message like

[ERROR] /home/user/myproject/src/main/java/com/mydomain/myproject/Main.java:[123,40] cannot find symbol
[ERROR] symbol: class MyClass

Solution:

Java does not know where to find MyClass.

First, check if you have imported MyClass correctly.

If MyClass can be found in your library, you most likely are missing an import statement at the top of the file where the error occurs.

If, on the other hand, MyClass is imported from an external library, check if:

  • You have the correct version of the library. Possibly, you are using an old version of the library where MyClass is not present. This is often the case when using SNAPSHOT versions in maven since different developers might have different SNAPSHOTs so one might have issue building while another might not.
  • You are using the correct import statement (refer to the docs or the source code of the library to check if you are using the correct package.

If you don’t know in which library you can find a certain symbol in, see our post [To be done].

What exactly are ‘symbols’?

The concept and term of a symbol is used in many different programming languages.
Basically it means ‘a name that refers to something declared somewhere else in more detail’.

Therefore, if you encounter error messages like ‘cannot find symbol’, the compiler is trying to tell you: “I don’t know what that name refers to”.

Example:

When you declare a class in Java, e.g.

class MyClass {
    /* Your code goes here ! */
}

you can later refer to that class by its name, MyClass, e.g. in

MyClass class = new MyClass();

In that line of code, MyClass is used symbolically to refer to the full class declaration (class MyClass { /* ... */}) which we listed before.

Hence, MyClass is used as a symbol to refer to your previous (full) declaration of MyClass.

If the name MyClass, however, has no associated full declaration, the compiler will tell you ‘cannot find symbol’.

Posted by Uli Köhler in Java

ElasticSearch: How to iterate all documents in index using Python (up to 10000 documents)

Important Note: This simple approach only works for up to ~10000 documents. Prefer using our scroll-based solution: See ElasticSearch: How to iterate / scroll through all documents in index

Use this helper function to iterate over all the documens in an index

def es_iterate_all_documents(es, index, pagesize=250, **kwargs):
    """
    Helper to iterate ALL values from
    Yields all the documents.
    """
    offset = 0
    while True:
        result = es.search(index=index, **kwargs, body={
            "size": pagesize,
            "from": offset
        })
        hits = result["hits"]["hits"]
        # Stop after no more docs
        if not hits:
            break
        # Yield each entry
        yield from (hit['_source'] for hit in hits)
        # Continue from there
        offset += pagesize

Usage example:

for entry in es_iterate_all_documents(es, 'my_index'):
    print(entry) # Prints the document as stored in the DB

How it works

You can iterate over all documents in an index in ElasticSearch by using queries like

{
    "size": 250,
    "from": 0
}

and increasing "from" by "size" after each iteration.

Posted by Uli Köhler in Databases, ElasticSearch, Python

How to fix ‘ng upgrade’ error ‘The specified command (“upgrade”) is invalid’

Problem:

You want to run ng upgrade to update your angular project libraries, but you see this error message:

The specified command ("upgrade") is invalid. For a list of available options,
run "ng help".

Did you mean "update"?

Solution:

You need to run

ng update

instead of ng upgrade (which is not a valid command).

This is what Did you mean "update"? in the error message is intended to point you towards.

Posted by Uli Köhler in Angular, Javascript

How to fix ElasticSearch ‘Types cannot be provided in put mapping requests, unless the include_type_name parameter is set to true’

Problem:

You want to create a mapping in ElasticSearch but you see an error message like

elasticsearch.exceptions.RequestError: RequestError(400, 'illegal_argument_exception', 'Types cannot be provided in put mapping requests, unless the include_type_name parameter is set to true.')

Solution:

As already suggested in the error message, set the include_type_name parameter to True.

With the Python API this is as simple as adding include_type_name=True to the put_mapping(...) call:

es.indices.put_mapping(index='my_index', body=my_mapping, doc_type='_doc', include_type_name=True)

In case you now see an error like

TypeError: put_mapping() got an unexpected keyword argument 'include_type_name'

you need to upgrade your elasticsearch python library, e.g. using

sudo pip3 install --upgrade elasticsearch

 

Posted by Uli Köhler in Databases, ElasticSearch, Python

How to get path to the wp_content directory in WordPress plugin

Problem:

You are writing a wordpress plugin in which you need the path to the wp-content directory on the filesystem.

Solution:

Use the WP_CONTENT_DIR constant.

$path_to_wp_content = WP_CONTENT_DIR; // e.g. "/var/sites/techoverflow.net/wp-content"

Note that WP_CONTENT_DIR has no trailing slash.

Use WP_CONTENT_DIR . "/" like this

$path_to_wp_content = WP_CONTENT_DIR . "/"; // e.g. "/var/sites/techoverflow.net/wp-content/"

to get the path to wp_content including a trailing slash.

Posted by Uli Köhler in PHP, Wordpress

How to fix WP_Query not listing any posts

Problem:

In your custom WordPress plugin or theme you have code like

$query = new WP_Query( );
while ( $query->have_posts() ) {
        $query->the_post();
        // ... your code to process the post ...
}

But $query->have_posts() is always false and the loop is never executed.

Solution:

WP_Query works only if you use an appropriate query. Using no query at all is not equivalent to list all posts!

This is likely what you want to do:

$query = new WP_Query(array('post_type' => 'post'));
Posted by Uli Köhler in PHP, Wordpress

How to recursively delete directory using C++17 filesystem library

To remove a file or directory (for example my-directory) use remove_all from the C++17 filesystem library:

remove_all("my-directory");

This will remove my-directory and all its sub-directories and files recursively.

Full example:

#include <experimental/filesystem>
using namespace std::experimental::filesystem;

int main() {
    remove_all("my-directory");
}

In case you are using GCC, you need to compile the file like this:

g++ -o delete-cpp17 delete-cpp17.cpp -lstdc++fs

You need to link the stdc++fs library so the functions from the C++17 filesystem library are available to your program.

If you just want to remove a file and don’t want to risk deleting an entire directory tree, use remove instead of remove_all or see our previous post How to delete file using C++17 filesystem library

Posted by Uli Köhler in C/C++

How to delete file using C++17 filesystem library

To remove a file (for example test.txt) use remove from the C++17 filesystem library:

remove("test.txt");

Full example:

#include <experimental/filesystem>
using namespace std::experimental::filesystem;

int main() {
    remove("test.txt");
}

In case you are using GCC, you need to compile the file like this:

g++ -o delete-cpp17 delete-cpp17.cpp -lstdc++fs

You need to link the stdc++fs library so the functions from the C++17 filesystem library are available to your program.

Note that remove does not recursively remove directories! Use remove_all or see How to recursively delete directory using C++17 filesystem library

Posted by Uli Köhler in C/C++

Is it a file or directory? Using C++17 filesystem library

You can use is_regular_file to check any path (either a C++17 path object or just a string). Similarly you can use is_directory to check if the given path belongs to a directory.

Note that these just return false if the file or directory does not exist!

// Check if something is a file
bool isTestTxtAFile = is_regular_file("test.txt"); // true
bool isMyDirectoryAFile = is_regular_file("my-directory"); // false
bool isDoesNotExistAFile = is_regular_file("does-not-exist"); // false
// Check if something is a directory
bool isTestTxtADirectory = is_directory("test.txt"); // false
bool isMyDirectoryADirectory = is_directory("my-directory"); // true
bool isDoesNotExistADirectory = is_directory("does-not-exist"); // false
Posted by Uli Köhler in C/C++