Python

How to iterate all days of year using Python

To iterate all days in a year using Python you can use this function snippet:

from collections import namedtuple
from calendar import monthrange

Date = namedtuple("Date", ["year", "month", "day"])

def all_dates_in_year(year=2019):
    for month in range(1, 13): # Month is always 1..12
        for day in range(1, monthrange(year, month)[1] + 1):
            yield Date(year, month, day)

Usage example:

for date in all_dates_in_year(2019):
    print(date)

This will print (excerpt):

Date(year=2019, month=1, day=1)
Date(year=2019, month=1, day=2)
Date(year=2019, month=1, day=3)
[...]
Date(year=2019, month=2, day=27)
Date(year=2019, month=2, day=28)
Date(year=2019, month=3, day=1)
[...]
Date(year=2019, month=12, day=29)
Date(year=2019, month=12, day=30)
Date(year=2019, month=12, day=31)

You can also install my Python3 library UliEngineering using sudo pip3 install -U UliEngineering and then use all_dates_in_year() by importing it like this:

from UliEngineering.Utils.Date import *

 

Also see How to get number of days in month in Python

 

Posted by Uli Köhler in Python

How to get number of days in month in Python

If you want to find out how many days a month using Python use this snippet:

from calendar import monthrange

num_days = monthrange(2019, 2)[1] # num_days = 28

print(num_days) # Prints 28

The calendar module automatically takes into account leap years.

You can also use this function snippet:

from calendar import monthrange

def number_of_days_in_month(year=2019, month=2):
    return monthrange(year, month)[1]

# Usage example:
print(number_of_days_in_month(2019, 2)) # Prints 28

You can also install my Python3 library UliEngineering using sudo pip3 install -U UliEngineering and then use number_of_days_in_month() by importing it like this:

from UliEngineering.Utils.Date import *

 

Posted by Uli Köhler in Python

How to get unit/resolution of NumPy np.datetime64 object

NumPy’s datetime64 objects are represented as 64 bit integers (that’s what the 64 in the name means).

In order to find out what the resolution (e.g. us, ns etc) first install my Python 3 UliEngineering library using

sudo pip3 install -U UliEngineering

and then use datetime64_unit() from UliEngineering.Utils.NumPy like this:

from UliEngineering.Utils.NumPy import *

my_datetime = datetime64_now()
print(datetime64_resolution(my_datetime)) # Prints "us"
Posted by Uli Köhler in Python

How to get unit/resolution of NumPy np.timedelta64 object

In order to get the unit (e.g. us, ms, ns, …) of a np.timedelta64 object like

my_timedelta = np.timedelta64(625, 'us') # Unit is 'us'

first install my Python 3 UliEngineering library using

sudo pip3 install -U UliEngineering

and then use timedelta64_unit() from UliEngineering.Utils.NumPy like this:

import numpy as np
from UliEngineering.Utils.NumPy import *

my_timedelta = np.timedelta64(625, 'us')
print(timedelta64_resolution(my_timedelta)) # Prints "us"
Posted by Uli Köhler in Python

How to convert numpy timedelta (np.timedelta64) object to integer

If you have a NumPy np.timedelta64 object like

import numpy as np
my_timedelta = np.timedelta64(625, 'us')

a common task is to convert it to an integer.

This is often as easy as

my_timedelta.astype(int) # = 625, type: np.int64

This will give you the number (which is always an integer!) stored in the np.timedelta64 object, however it will ignore the unit  (e.g. us i.e. microseconds in our example).

To get the unit of the timedelta, first install UliEngineering using sudo pip3 install -U UliEngineering which you can then use like this:

import numpy as np
from UliEngineering.Utils.NumPy import *

my_timedelta = np.timedelta64(625, 'us')
print(timedelta64_resolution(my_timedelta)) # Prints "us"
Posted by Uli Köhler in Python

How to create a new numpy timedelta (np.timedelta64) object

There are two basic ways to create a new np.timedelta64 object:

Construct directly:

import numpy as np
my_timedelta = np.timedelta64(625, 'us')

Note that the constructor will only work with ints, i.e. np.timedelta64(625.5, 'us') won’t work. See

Construct as a difference between two np.datetime64 objects:

See How to get current datetime as numpy datetime (np.datetime64) for more details on how to get the current datetime as np.datetime64.

For any two np.datetime64 instances, we can compute the difference using

from datetime import datetime
import numpy as np

# There will be a minimal time difference between those two
dt1 = np.datetime64(datetime.now())
dt2 = np.datetime64(datetime.now())

my_timedelta = dt2 - dt1
print(my_timedelta) # For me, this prints "16 microseconds"
Posted by Uli Köhler in Python

How to get current datetime as numpy datetime (np.datetime64)

The simplest way of getting the current time as np.datetime64 object is to combine datetime.datetime.now() and the np.datetime64() constructor:

from datetime import datetime
import numpy as np

dt_now = np.datetime64(datetime.now())

print(dt_now)

Or use this function for convenience:

from datetime import datetime
import numpy as np

def datetime64_now():
    return np.datetime64(datetime.now())

dt_now = datetime64_now()
print(dt_now)

datetime64_now() is also available on my library UliEngineering.
Install using sudo pip3 install -U UliEngineering, then use like this:

from UliEngineering.Utils.NumPy import datetime64_now

dt_now = datetime64_now()
print(dt_now)
Posted by Uli Köhler in Python

How to list attributes of a Python object without __dict__

Usually you can list all attributes and functions of a Python object using __dir__:

from collections import namedtuple
nt = namedtuple("Foo", [])
print(nt.__dict__) # Prints attributes and functions of nt.

This can be very useful to find out e.g. which functions you can call on an object:

However for some objects like datetime.datetime this doesn’t work. Attempting to run

from datetime import datetime
dt = datetime.now()
print(dt.__dict__)

will result in

Traceback (most recent call last):
  File "test.py", line 3, in <module>
AttributeError: 'datetime.datetime' object has no attribute '__dict__'

So how can you find out which attributes your datetime.datetime object has and what function you can call on it?

Use dir():

from datetime import datetime
dt = datetime.now()
print(dir(dt))

This will print e.g.

['__add__', '__class__', '__delattr__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__ne__', '__new__', '__radd__', '__reduce__', '__reduce_ex__', '__repr__', '__rsub__', '__setattr__', '__sizeof__', '__str__', '__sub__', '__subclasshook__', 'astimezone', 'combine', 'ctime', 'date', 'day', 'dst', 'fold', 'fromordinal', 'fromtimestamp', 'hour', 'isocalendar', 'isoformat', 'isoweekday', 'max', 'microsecond', 'min', 'minute', 'month', 'now', 'replace', 'resolution', 'second', 'strftime', 'strptime', 'time', 'timestamp', 'timetuple', 'timetz', 'today', 'toordinal', 'tzinfo', 'tzname', 'utcfromtimestamp', 'utcnow', 'utcoffset', 'utctimetuple', 'weekday', 'year']

 

Posted by Uli Köhler in Python

Minimal argparse example in Python

Minimal example with one positional parameter (param):

#!/usr/bin/env python3
import argparse

if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("param")
    args = parser.parse_args()
    # Your code goes here!
    print(args.param)

Minimal example with one argument (-p, --param):

#!/usr/bin/env python3
import argparse

if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("-p", "--param")
    args = parser.parse_args()
    # Your code goes here!
    print(args.param)

 

Posted by Uli Köhler in Python

Minimal example for iterable classes in Python

Here’s a minimal example for a custom class that is iterable similar to a generator:

class MyIterable(object):
    def __init__(self):
        pass # ... Your code goes here!

    def __iter__(self):
        # Usually there is no need to modify this!
        return self

    def __next__(self):
        # To indicate no further values, use: raise StopIteration
        return "test" # Return the next value here. Return n

Usage example:

it = MyIterable()
print(next(it)) # Prints "test"
print(next(it)) # Prints "test"

In case you want to read more about how to create your own iterators, I recommend this tutorial.

Posted by Uli Köhler in Python

How to convert Celsius/Fahrenheit/Kelvin temperatures in Python using UliEngineering

In this example, we’ll use the UliEngineering library to convert between the three most commonly used units of temperature: Celsius (°C), Fahrenheit (°F) and Kelvin (K)

In order to install UliEngineering (a Python 3 library) run:

sudo pip3 install -U UliEngineering

We can now use the following functions from the UliEngineering.Physics.Temperature package to convert between the units of temperature:

  • celsius_to_kelvin(temperature)
  • kelvin_to_celsius(temperature)
  • fahrenheit_to_kelvin(temperature)
  • fahrenheit_to_celsius(temperature)

Tip: You can pass both numbers (like 120) or strings (like 120 °C or 0.01 °F) to most UliEngineering functions. SI prefixes like k and m are automatically decoded.

Example:

from UliEngineering.EngineerIO import auto_print
from UliEngineering.Physics.Temperature import *

# Convert to celsius and store in variable
temp = fahrenheit_to_celsius("120 °F") # temp = 48.88888888888897

# Automatically format value & print. Prints "48.9 °C"
auto_print(fahrenheit_to_celsius, "120 °F")

Additionally, UliEngineering provides normalize_temperature_celsius(temp, default_unit="°C") which takes a string and automatically recognizes the unit (if there is no unit given, the default_unit is assumed).

Examples:

  • normalize_temperature_celsius("120 °C") == 120.0
  • normalize_temperature_celsius("120") == 120.0
  • normalize_temperature_celsius(120) == 120.0
  • normalize_temperature_celsius("120 °F") == 48.88888888888897
  • normalize_temperature_celsius("120 K") == -153.14999999999998
  • auto_print(normalize_temperature_celsius, "120 K") prints -153.15 °C

Note that while °K is recognized by UliEngineering‘s functions, K is correctly used without a degree symbol.

Also there is normalize_temperature(temp, default_unit="°C") which is equivalent to normalize_temperature_celsius() except it returns the temperature in Kelvin instead of degrees Celsius.

Posted by Uli Köhler in Python

Computing crystal load capacitance using Python & UliEngineering

When you implement a crystal oscillator circuit, one task that is both essential and often overlooked is how to compute the load capacitors of the crystal.

Tip: In many cases it is easier to use an integrated crystal oscillator for which you don’t have to care about load capacitors. Crystals are usually recommendable for low-power applications and high-volume manufacturing where oscillators are too expensive.

In this example, we’ll compute the load capacitors for a Abracon ABLS-16.000MHZ-B4-T (specified with a load capacitance of 18 pF according to the datasheet).

It is paramount to understand that 18 pF load capacitance specification does not mean that you can use 18 pF capacitors or 2x 9 pF capacitors. You need to actually calculate the correct values.

You need the following information:

  • Load capacitance of the crystal, specified in the crystal’s datasheet.
  • An estimate of the stray capacitance, i.e. the capacitance of the PCB traces from the crystal to the microcontroller. If you don’t know this, I recommend to use 2 pF
  • The pin capacitance of the microcontroller (or whatever device you want to connect your device to) as specified in the microcontroller’s datasheet. Often the capacitance of oscillator pins is not the same as for other pins and is hence specified separately. 3 pF is a good first estimate for most modern ICs. The value must be specified per pin!

In order to install UliEngineering (a Python 3 library) run:

sudo pip3 install -U UliEngineering

Now we can compute the correct load capacitors using:

from UliEngineering.EngineerIO import auto_print
from UliEngineering.Electronics.Crystal import *

# Print load capacitors: prints "29.0 pF"
auto_print(load_capacitors, cload="18 pF", cpin="3 pF", cstray="2 pF")

In this case, you need to have two load capacitors of 29.0 pF each.

You should use the closest available value of load capacitor, but always check the resulting frequency if you have high clock accuracy requirements.

Tip: You can pass both numbers (like 18e-12) or strings (like 18 pF or 0.018 nF) to most UliEngineering functions. SI prefixes like p and n are automatically decoded

If you want to use the value programmatically, call load_capacitors() directly:

from UliEngineering.EngineerIO import auto_print
from UliEngineering.Electronics.Crystal import *

# Coompute the load capacitor value (for both of the two load caps): 
load_caps = load_capacitors(cload="18 pF", cpin="3 pF", cstray="2 pF")
# load_caps = 2.9e-11

 

Posted by Uli Köhler in Electronics, Python

Computing resistor power dissipation in Python using UliEngineering

In this example we’ll calculate the power dissipation of a 1 kΩ resistor with a constant current of 30 mA flowing through it, using our science and engineering Python library UliEngineering.

In order to install UliEngineering (a Python3-only library), run

sudo pip3 install -U UliEngineering

Now we can compute the resistor power dissipation using power_dissipated_in_resistor_by_current()

from UliEngineering.EngineerIO import auto_print
from UliEngineering.Electronics.Resistors import *
# Just compute the value:
power = power_dissipated_in_resistor_by_current("1 kΩ", "30 mA") # power = 0.9

# Print value: prints: prints "900 mW"
auto_print(power_dissipated_in_resistor_by_current, "1 kΩ", "30 mA")

Since the result is 900 mW, you can deduce that you need to use a resistor with a power rating of at least one Watt.

Note that you can pass both numbers (like 0.03) or strings (like 30 mA or 0.03 A) to most UliEngineering functions. SI prefixes like k and M are automatically decoded

If you know the voltage across the resistor, you can use power_dissipated_in_resistor_by_voltage(). Let’s assume there is 1V dropped across the resistor:

from UliEngineering.EngineerIO import auto_print
from UliEngineering.Electronics.Resistors import *
# Just compute the value:
power = power_dissipated_in_resistor_by_voltage("1 kΩ", "30 mA") # power = 0.001

# Print value: prints: prints "1.00 mW"
auto_print(power_dissipated_in_resistor_by_voltage, "1 kΩ", "30 mA")

So in this case the power dissipation is extremely low – only 1.00 mW – and won’t matter for most practical applications.

In many cases, you can also pass NumPy arrays to UliEngineering functions:

from UliEngineering.EngineerIO import format_value
from UliEngineering.Electronics.Resistors import *
import numpy as np

# Compute the value:
resistors = np.asarray([100, 500, 1000]) # 100 Ω, 500 Ω, 1 kΩ
power = power_dissipated_in_resistor_by_voltage(resistors, "30 mA") # power = 0.001

# power = [9.0e-06 1.8e-06 9.0e-07]

 

Posted by Uli Köhler in Electronics, Python

How to fix ‘error: option –rednose not recognized’

Problem:

You want to test a Python package e.g. using python setup.py test.  However, you see the following error message:

usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
   or: setup.py --help [cmd1 cmd2 ...]
   or: setup.py --help-commands
   or: setup.py cmd --help

error: option --rednose not recognized

Solution:

Rednose is a Python package that you need to install:

sudo pip3 install rednose

or, if you don’t have pip3, try

sudo pip install rednose
Posted by Uli Köhler in Python

ElasticSearch: How to iterate / scroll through all documents in index

In ElasticSearch, you can use the Scroll API to scroll through all documents in an entire index.

In Python you can scroll like this:

def es_iterate_all_documents(es, index, pagesize=250, scroll_timeout="1m", **kwargs):
    """
    Helper to iterate ALL values from a single index
    Yields all the documents.
    """
    is_first = True
    while True:
        # Scroll next
        if is_first: # Initialize scroll
            result = es.search(index=index, scroll="1m", **kwargs, body={
                "size": pagesize
            })
            is_first = False
        else:
            result = es.scroll(body={
                "scroll_id": scroll_id,
                "scroll": scroll_timeout
            })
        scroll_id = result["_scroll_id"]
        hits = result["hits"]["hits"]
        # Stop after no more docs
        if not hits:
            break
        # Yield each entry
        yield from (hit['_source'] for hit in hits)

This function will yield each document encountered in the index.

Example usage for index my_index:

es = Elasticsearch([{"host": "localhost"}])

for entry in es_iterate_all_documents(es, 'my_index'):
    print(entry) # Prints the document as stored in the DB

 

Posted by Uli Köhler in Databases, ElasticSearch, Python

ElasticSearch: How to iterate all documents in index using Python (up to 10000 documents)

Important Note: This simple approach only works for up to ~10000 documents. Prefer using our scroll-based solution: See ElasticSearch: How to iterate / scroll through all documents in index

Use this helper function to iterate over all the documens in an index

def es_iterate_all_documents(es, index, pagesize=250, **kwargs):
    """
    Helper to iterate ALL values from
    Yields all the documents.
    """
    offset = 0
    while True:
        result = es.search(index=index, **kwargs, body={
            "size": pagesize,
            "from": offset
        })
        hits = result["hits"]["hits"]
        # Stop after no more docs
        if not hits:
            break
        # Yield each entry
        yield from (hit['_source'] for hit in hits)
        # Continue from there
        offset += pagesize

Usage example:

for entry in es_iterate_all_documents(es, 'my_index'):
    print(entry) # Prints the document as stored in the DB

How it works

You can iterate over all documents in an index in ElasticSearch by using queries like

{
    "size": 250,
    "from": 0
}

and increasing "from" by "size" after each iteration.

Posted by Uli Köhler in Databases, ElasticSearch, Python

How to fix ElasticSearch ‘Types cannot be provided in put mapping requests, unless the include_type_name parameter is set to true’

Problem:

You want to create a mapping in ElasticSearch but you see an error message like

elasticsearch.exceptions.RequestError: RequestError(400, 'illegal_argument_exception', 'Types cannot be provided in put mapping requests, unless the include_type_name parameter is set to true.')

Solution:

As already suggested in the error message, set the include_type_name parameter to True.

With the Python API this is as simple as adding include_type_name=True to the put_mapping(...) call:

es.indices.put_mapping(index='my_index', body=my_mapping, doc_type='_doc', include_type_name=True)

In case you now see an error like

TypeError: put_mapping() got an unexpected keyword argument 'include_type_name'

you need to upgrade your elasticsearch python library, e.g. using

sudo pip3 install --upgrade elasticsearch

 

Posted by Uli Köhler in Databases, ElasticSearch, Python

How to fix Python bottle Unsupported response type: <class ‘dict’>

Problem:

You are running a Python HTTP server using bottle. When you access your HTTP server endpoint, you see a HTTP 500 error message like

Unsupported response type: <class 'dict'>

Solution:

This occurs because you are trying to return a Python list of dictionaries, for example in

from bottle import route, run, template, response

@route('/')
def index():
    # We expect bottle to return a JSON here
    # but that doesn't happen!
    return [{"a": "b"}]

run(host='localhost', port=8080)

In order to work around this behaviour, you need to set response.content_type and explicitly use json.dumps() to convert your JSON into a string:

from bottle import route, run, template, response
import json

@route('/')
def index():
    response.content_type = 'application/json'
    return json.dumps([{"a": "b"}])

run(host='localhost', port=8080)

 

Posted by Uli Köhler in Python

How to fix ElasticSearch ‘Root mapping definition has unsupported parameters’

Problem:

You want to create an ElasticSearch index with a custom mapping or update the mapping of an existing ElasticSearch index but you see an error message like

elasticsearch.exceptions.RequestError: RequestError(400, 'mapper_parsing_exception', 'Root mapping definition has unsupported parameters:  [mappings : {properties={num_total={type=integer}, approved={type=integer}, num_translated={type=integer}, pattern_length={type=integer}, num_unapproved={type=integer}, pattern={type=keyword}, num_approved={type=integer}, translated={type=integer}, untranslated={type=integer}, num_untranslated={type=integer}, group={type=keyword}}}]')

Solution:

This can point to multiple issues. Essentially, ElasticSearch is trying to tell you that the structure of your JSON is not correct.

Often this error is misinterpreted as individual field definitions being wrong, but this is rarely the issue (and only if an individual field definition is completely malformed).

If your message is structured like

... unsupported parameters:  [mappings : ...

then the most likely root cause is that you have mappings nested inside mappings in your JSON. This also applies if you update a mapping (put_mapping) – in this case the outer mapping is implicit!

Example: Your code looks like this:

es.indices.put_mapping(index='my_index, doc_type='_doc', body={
    "mappings": {
        "properties": {
            "pattern": {
                "type":  "keyword"
            }
        }
    }
})

ElasticSearch will internally create a JSON like this internally:

{
    "mappings": {
        "mappings": {
            "properties": {
                "pattern": {
                    "type":  "keyword"
                }
            }
        }
    }
}

See that there are two mappings inside each other? ElasticSearch does not view this as a correctly structured JSON, therefore you need to remove the "mapping": {...} from your code, resulting in

es.indices.put_mapping(index='my_index, doc_type='_doc', body={
    "properties": {
        "pattern": {
            "type":  "keyword"
        }
    }
})
Posted by Uli Köhler in Databases, ElasticSearch, Python