If you have a date like
import arrow dt = arrow.get("2021-05-27")
you can add N
months (example: 18
months) to said date using
dt.shift(months=+18)
The result will be an arrow
representation of 2022-11-27
If you have a date like
import arrow dt = arrow.get("2021-05-27")
you can add N
months (example: 18
months) to said date using
dt.shift(months=+18)
The result will be an arrow
representation of 2022-11-27
You have to points represented by some coordinate string in Python:
a = "N 48° 06.112' E 11° 44.113'" b = "N 48° 06.525' E 11° 44.251'"
and you want to compute both bearing and distance between them
This can be done using a combination of two of our previous posts:
from geographiclib.geodesic import Geodesic from geopy.geocoders import ArcGIS geolocator = ArcGIS() a = geolocator.geocode("N 48° 06.112' E 11° 44.113'") b = geolocator.geocode("N 48° 06.525' E 11° 44.251'") result = Geodesic.WGS84.Inverse(a.latitude, a.longitude, b.latitude, b.longitude) distance = result["s12"] # in [m] (meters) bearing = result["azi1"] # in [°] (degrees)
Result for our example:
distance = 784.3069649126435 # m bearing = 12.613924599757134 # °
Let’s assume we have the following points represented by their latitude & longitude in Python:
a = (48.11617185, 11.743858785932662) b = (48.116026149999996, 11.743938922310974)
and we want to compute both distance and bearing between those points on the WGS84 or any other Geoid of your choosing.
We can use geographiclib to do that:
from geographiclib.geodesic import Geodesic result = Geodesic.WGS84.Inverse(*a, *b) distance = result["s12"] # in [m] (meters) bearing = result["azi1"] # in [°] (degrees)
Geodesic.WGS84.Inverse(*a, *b)
is just a shorthand for Geodesic.WGS84.Inverse(a[0], a[1], b[0], b[1])
, so don’t be too confused by the syntax.
Using our example coordinates from above result
is
{'lat1': 48.11617185, 'lon1': 11.743858785932662, 'lat2': 48.116026149999996, 'lon2': 11.743938922310974, 'a12': 0.00015532346032069415, 's12': 17.26461706032189, 'azi1': 159.78110567187977, 'azi2': 159.7811653333465}
Therefore,
distance = 17.26461706032189 # m bearing = 159.78110567187977 # °
When working with user-entered coordinates, you often have strings like N 48° 06.976 E 11° 44.638
, 48°06'58.6"N 11°44'38.3"E
or N48.116267, E11.743967
. These lan/lon coordinates come in many, many different formats which makes it rather hard to parse in an automated setting.
One simple solution for Python is to use geopy which provides access to a bunch of Online services such as ArcGIS. These services make it easy to parse pretty much any form of coordinate. You can install geopy
using
pip install geopy
Note that Nominatim does not work for the pure coordinates use case – it parses the coordinates just fine but will return the closest building / address.
from geopy.geocoders import ArcGIS geolocator = ArcGIS() result = geolocator.geocode("N 48° 06.976' E 11° 44.638'")
In case the coordinates can’t be parsed, result
will be None
After that, you can work with result, for example, in the following ways:
print(result)
will just print the result:
>>> print(result) Location(Y:48.116267 X:11.743967, (48.11626666666667, 11.743966666666667, 0.0))
You can extract the latitude and longitude using result.latitude
and result.longitude
.
>>> print(result.latitude, result.longitude) (48.11626666666667, 11.743966666666667)
For other ways to work with these coordinates, refer to the geopy documentation.
This function sums up all values from the given 3D array:
def primitive_pixel_sum(frame): result = 0.0 for x in range(frame.shape[0]): for y in range(frame.shape[1]): for z in range(frame.shape[2]): result += frame[x,y,z] return result
Whereas the following function is exactly the same algorithm, but using numba.jit
:
import numba @numba.jit def numba_pixel_sum(frame): result = 0.0 for x in range(frame.shape[0]): for y in range(frame.shape[1]): for z in range(frame.shape[2]): result += frame[x,y,z] return result
We can benchmark them in Jupyter using
%%timeit primitive_pixel_sum(frame)
and
%%timeit numba_pixel_sum(frame)
respectively.
We tested this with a random camera image taken from OpenCV of shape (480, 640, 3)
primitive_pixel_sum()
:
1.78 s ± 253 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
numba_pixel_sum()
:
4.06 ms ± 413 µs per loop (mean ± std. dev. of 7 runs, 1 loop each)
It should be clear from these results that the numba version is 438
times faster compared to the primitive version.
Note that when compiling complex functions using numba.jit
it can take many milliseconds or even seconds to compile – possibly longer than a simple Python function would take.
Since it’s so simple to use Numba, my recommendation is to just try it out for every function you suspect will eat up a lot of CPU time. Over time you will be able to develop an intuition for which functions it’s worth to use Numba and which functions won’t work at all or if it will be slower overall than just using Python.
Remember than often you can also use NumPy functions to achieve the same result. In our example, you could achieve the same thing using
np.sum(frame)
which is even faster than Numba:
%%timeit np.sum(frame)
Result:
2.5 ms ± 7.17 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
The following code uses the v4l2-ctl
executable to get and set v4l2
parameters such as exposure_absolute
. It also provides means of writing a parameter and verifying if it has been set correctly.
def v4l2_set_parameters_once(params, device="/dev/video0"): """ Given a dict of parameters: { "exposure_auto": 1, "exposure_absolute": 10, } this function sets those parameters using the v4l2-ctl command line executable """ set_ctrl_str = ",".join([f"{k}={v}" for k,v in params.items()]) # expsosure_absolute=400,exposure_auto=1 subprocess.check_output(["v4l2-ctl", "-d", device, f"--set-ctrl={set_ctrl_str}"]) def v4l2_get_parameters(params, device="/dev/video0"): """ Query a bunch of v4l2 parameters. params is a list like [ "exposure_auto", "exposure_absolute" ] Returns a dict of values: { "exposure_auto": 1, "exposure_absolute": 10, } """ get_ctrl_str = ",".join([f"{k}" for k in params]) out = subprocess.check_output(["v4l2-ctl", "-d", device, f"--get-ctrl={get_ctrl_str}"]) out = out.decode("utf-8") result = {} for line in out.split("\n"): # line should be like "exposure_auto: 1" if ":" not in line: continue k, _, v = line.partition(":") result[k.strip()] = v.strip() return result def v4l2_set_params_until_effective(params, device="/dev/video0"): """ Set V4L2 params and check if they have been set correctly. If V4L2 does not confirm the parameters correctly, they will be set again until they have an effect params is a dict like { "exposure_auto": 1, "exposure_absolute": 10, } """ while True: v4l2_set_parameters_once(params, device=device) result = v4l2_get_parameters(params.keys(), device=device) # Check if queried parameters match set parameters had_any_mismatch = False for k, v in params.items(): if k not in result: raise ValueError(f"Could not query {k}") # Note: Values from v4l2 are always strings. So we need to compare as strings if str(result.get(k)) != str(v): print(f"Mismatch in {k} = {result.get(k)} but should be {v}") had_any_mismatch = True # Check if there has been any mismatch if not had_any_mismatch: return
v4l2_set_params_until_effective({ "exposure_auto": 1, "exposure_absolute": 1000, })
Using OpenCV on Linux, if you have a video device that interfaces a V4L2 device such as a USB webcam:
camera = cv2.VideoCapture(0)
in order to set the manual white balance temperature, you first need to disable automatic white balancing using CAP_PROP_AUTO_WB
. See our previous post How to enable/disable manual white balance in OpenCV (Python) for more details on how you can do this, here’s only the short version that works with most cameras.
After that, you can set the white balance temperature using CAP_PROP_WB_TEMPERATURE
:
camera.set(cv2.CAP_PROP_AUTO_WB, 0.0) # Disable automatic white balance camera.set(cv2.CAP_PROP_WB_TEMPERATURE, 4200) # Set manual white balance temperature to 4200K
For V4L2 cameras, as you can see in our previous post on mapping of OpenCV parameters to V4L2 parameters, CAP_PROP_WB_TEMPERATURE
is mapped to V4L2_CID_WHITE_BALANCE_TEMPERATURE
which is shown in v4l2-ctl -d /dev/video0 --all
as white_balance_temperature
. Therefore, you can easily verify if, for example, disabling the auto white balance worked for your V4L2 camera such as any USB camera by looking at the white_balance_temperature
section of v4l2-ctl -d /dev/video0 --all
:
white_balance_temperature 0x0098091a (int) : min=2800 max=6500 step=1 default=4600 value=4200
Using OpenCV on Linux, if you have a video device that interfaces a V4L2 device such as a USB webcam:
camera = cv2.VideoCapture(0)
you can typically enable automatic white balance (= disable manual white balance) for any camera by using
camera.set(cv2.CAP_PROP_AUTO_WB, 1.0) # Enable automatic white balance
or disable automatic white balance (= enable manual white balance) using
camera.set(cv2.CAP_PROP_AUTO_WB, 0.0) # Disable automatic white balance
When disabling automatic white balance, you should also set the manual white balance temperature – see our post How to set manual white balance temperature in OpenCV (Python) for more details.
For V4L2 cameras, as you can see in our previous post on mapping of OpenCV parameters to V4L2 parameters, CAP_PROP_AUTO_WB
is mapped to V4L2_CID_AUTO_WHITE_BALANCE
which is shown in v4l2-ctl -d /dev/video0 --all
as white_balance_temperature_auto
. Therefore, you can easily verify if, for example, disabling the auto white balance worked for your V4L2 camera such as any USB camera by looking at the white_balance_temperature_auto
section of v4l2-ctl -d /dev/video0 --all
:
white_balance_temperature_auto 0x0098090c (bool) : default=1 value=0
Using OpenCV on Linux, if you have a video device that interfaces a V4L2 device such as a USB webcam:
camera = cv2.VideoCapture(0)
you can typically set the automatic exposure mode by setting exposure_auto
to 1
(the following output is from v4l2-ctl -d /dev/video0 --all
):
exposure_auto 0x009a0901 (menu) : min=0 max=3 default=3 value=1 1: Manual Mode 3: Aperture Priority Mode
As you can see in our previous blogpost, exposure_auto
(which is named V4L2_CID_EXPOSURE_AUTO
in V4L2 in C/C++) is mapped to CAP_PROP_AUTO_EXPOSURE
.
Therefore, you can enable manual exposure using
camera.set(cv2.CAP_PROP_AUTO_EXPOSURE, 1) # Set exposure to manual mode
You should, however, verify these settings using v4l2-ctl --all
using your specific camera.
This list can be easily obtained using the following Python code:
for v in [k for k in cv2.__dict__.keys() if k.startswith("CAP_PROP")]: print(f"cv2.{v}")
In our previous post How to get video metadata as JSON using ffmpeg/ffprobe we showed how to generate json
-formatted output using ffprobe
which comes bundled with ffmpeg
.
Assuming ffprobe
is installed, you can easily use this to obtain the duration of a video clip (say, in input.mp4
) using Python:
import subprocess import json input_filename = "input.mp4" out = subprocess.check_output(["ffprobe", "-v", "quiet", "-show_format", "-print_format", "json", input_filename]) ffprobe_data = json.loads(out) duration_seconds = float(ffprobe_data["format"]["duration"]) # Example: duration_seconds = 11.6685
When writing such code, be aware of the risk of shell code injection if you don’t using subprocess
correctly!
Note: This specific post only covers Jupyter Lab – not Jupyter Notebook. I have no post for Jupyter Notebook so far, but you can use a similar method here, just with slightly different config names etc.
On Ubuntu, my Jupyter Lab always opens using firefox
whereas I generally want to use chrome.
In order to fix this, I first needed to generate a default config file (the echo -e "\n"
part is to automatically answer no
when prompted if any existing config file should be overwritten:
echo -e "\n" | jupyter lab --generate-config
Now the config file in ~/.jupyter/jupyter_lab_config.py
contains this line:
# c.ServerApp.browser = ''
which we can automatically un-comment and set to chrome using:
sed -i -e "s/# c.ServerApp.browser = ''/c.ServerApp.browser = 'google-chrome'/g" ~/.jupyter/jupyter_lab_config.py
The resulting line looks like this:
c.ServerApp.browser = 'google-chrome'
google-chrome
instead of firefox
This is a script which you can copy & paste directly into your command line:
echo -e "\n" | jupyter lab --generate-config sed -i -e "s/# c.ServerApp.browser = ''/c.ServerApp.browser = 'google-chrome'/g" ~/.jupyter/jupyter_lab_config.py
In bottle you can generate a custom HTTP response code by first importing HTTPResponse
:
from bottle import HTTPResponse
and then, in your @route
function, raise
a HTTPResponse
, for example
raise HTTPResponse("This is a HTTP 401 test URL", status=401)
#!/usr/bin/env python3 from bottle import HTTPResponse, route, run @route('/test401') def hello(): raise HTTPResponse("This is a HTTP 401 test URL", status=401) run(host='localhost', port=8080, debug=True)
Run this, script, then open http://localhost/test401
which will result in a HTTP 401
(unauthorized) response code with
This is a HTTP 401 test URL
as response body.
In order to obtain the value of a query parameter named myparam
in bottle
, first import
request
:
from bottle import request
and then use
request.params["myparam"]
#!/usr/bin/env python3 from bottle import route, run, request @route('/test') def hello(): query_param = request.params["myparam"] return f"myparam is {query_param}" run(host='localhost', port=8080, debug=True)
Run this script and open http://localhost:8080/test?myparam=xyz
in your browser – you will see
myparam is xyz
This example does not send data to the serial port but only copies data received from the serial port to stdout. Newlines received from the serial port are preserved.
#!/usr/bin/env python3 import serial ser = serial.Serial("/dev/ttyUSB0", baudrate=115200) try: while True: response = ser.read() if response: print(response.decode("iso-8859-1"), end="") finally: ser.close()
By using iso-8859-1
decoding, we ensure that even binary bytes are decoded in some way and do not cause an exception.
This script copies myfolder
to myfolder2
recursively and renames all .txt
files to .xml
#!/usr/bin/env python3 import os import os.path import re import shutil srcfolder = "myfolder" dstfolder = "myfolder2" for subdir, dirs, files in os.walk(srcfolder): for file in files: # Rename .txt to .xml if file.endswith(".txt"): dstfile_name = re.sub(r"\.txt$", ".xml", file) # Regex to replace .txt only at the end of the string else: dstfile_name = file # Compute destination path srcpath = os.path.join(subdir, file) dstpath_relative_to_outdir = os.path.relpath(os.path.join(subdir, dstfile_name), start=srcfolder) dstpath = os.path.join(dstfolder, dstpath_relative_to_outdir) # Create directory if not exists os.makedirs(os.path.dirname(dstpath), exist_ok=True) # Copy file (copy2 preserves metadata) shutil.copy2(srcpath, dstpath)
This script will zip the folder myfolder
recursively to myzip
. Note that empty directories will not be copied over to the ZIP.
#!/usr/bin/env python3 import zipfile import os # This folder folder = "myfolder" with zipfile.ZipFile(f"{folder}.zip", "w") as outzip: for subdir, dirs, files in os.walk(folder): for file in files: # Read file srcpath = os.path.join(subdir, file) dstpath_in_zip = os.path.relpath(srcpath, start=folder) with open(srcpath, 'rb') as infile: # Write to zip outzip.writestr(dstpath_in_zip, infile.read())
This minimal example creates a threading.Lock
and locks it using a with
statement. This is the simplest way of using a Lock
.
from threading import Lock lock = Lock() with lock: print("Holding the lock!")
When storing real-time data from a Python script, it is often helpful to have a timestamp consisting of both date & time in your filename, such as
mydata-2022-09-02_00-31-50-613015.csv
With this specific syntax we avoid special characters which are an issue on Windows operating systems, and we provide a lexically sortable filename
In Python you can do that using UliEngineering.Utils.generate_datetime_filename()
from the UliEngineering library.
First, install the UliEngineering library using
pip install UliEngineering
Now you can generate your filename using
from UliEngineering.Utils.Date import * filename = generate_datetime_filename() # example: filename == 'data-2022-09-02_03-02-00-045587.csv'
or you can just open the file using with open()
:
with open(generate_datetime_filename(), "w") as outfile: # Example of what you can do with outfile outfile.write("test")
You can use this simplified version which does not support fractional seconds and will generate filenames like
data-2022-09-02_00-31-50.csv
Source code (simple version – the UliEngineering version is more robust and supports more features):
from datetime import datetime def generate_datetime_filename(label="data", extension="csv", dt=None): if dt is None: dt = datetime.now() return f"{label}-{dt.year}-{dt.month:02d}-{dt.day:02d}_{dt.hour:02d}-{dt.minute:02d}-{dt.second:02d}.{extension}"
This is a really simple example of how to add an extra thread to your Python script, to run some task in the background:
from threading import Thread import time def extra_thread_function(): while True: print("extra_thread is running") time.sleep(1) extra_thread = threading.Thread(target=extra_thread_function) extra_thread.start() # TODO Your code for the main thread goes here! # OPTIONAL: Wait for extra_thread to finish extra_thread.join()