How to add pandas pd.Timestamp

You can’t directly add Pandas pd.Timestamp instances:

t1 = pd.Timestamp('now')
t2 = pd.Timestamp('now')

t1 + t2
# TypeError: unsupported operand type(s) for +: 'Timestamp' and 'Timestamp'

But you can convert them to a numpy timestamp using their asm8 attribute, convert that timestamp to an integer, add it and convert it back:

t1 = pd.Timestamp('now')
t2 = pd.Timestamp('now')

tsum = (t1.asm8.astype(np.int64) + t2.asm8.astype(np.int64))
tsum_timestamp = pd.Timestamp(tsum.astype('<M8[ns]'))

 

Posted by Uli Köhler in pandas, Python

How to get average or mean between two pandas pd.Timestamp objects?

In order to compute the mean value between two pd.Timestamp instances, subtract them to obtain a pd.Timedelta and then add said Timedelta object to the first (smaller) timestamp:

t1 = pd.Timestamp('now')
t2 = pd.Timestamp('now')

mean_timestamp = t1 + (t2 - t1)

 

Posted by Uli Köhler in pandas, Python

How to display BytesIO containing a PNG image in Jupyter/IPython

In IPython, you can use the Image() function to show an image:

 

from IPython.display import Image
Image(filename="img1.png")

But what if you don’t have the PNG data in a file but in a BytesIO?

Use .getvalue():

from IPython.display import Image

my_bio = ... # Insert your code here
Image(my_bio.getvalue())

 

 

Posted by Uli Köhler in Python

How to export certificates from Traefik certificate store

Traefik stores certificates as base64 encoded X.509 certificates and keys inside the certificate store.

This is a python script to export certificate from Traefik certificate store .json file:

import json
import base64

# Read Traefik ACME JSON
with open("acme.json") as acme_file:
    acme = json.load(acme_file)

# Select certificates from a specific resolver
resolver_name = "my-resolver"
certificates = acme[resolver_name]["Certificates"]

# Find the specific certificate we are looking for
certificate = [certificate for certificate in certificates if "myddomain.com" in certificate["domain"].get("sans", [])][0]

# Extract X.509 certificate data
certificate_data = base64.b64decode(certificate["certificate"])
key_data = base64.b64decode(certificate["key"])

# Export certificate and key to file
with open("certificate.pem", "wb") as certfile:
    certfile.write(certificate_data)

with open("key.pem", "wb") as keyfile:
    keyfile.write(key_data)

Note that depending on what is the primary name for your certificate, you might need to use

if "myddomain.com" == certificate["domain"]["main"]

instead of

if "myddomain.com" in certificate["domain"].get("sans", [])

 

Posted by Uli Köhler in Python, Traefik

How to find XCP-NG host-uuid

For many xe commands on XCP-NG you have to use a host-uuid=... parameter.

In order to find out your Host UUID, use

xe host-list

Example:

[03:19 virt01-xcpng ~]# xe host-list
uuid ( RO)                : 71739b18-2999-4794-a024-87d5d26215d1
          name-label ( RW): virt01-xcpng
    name-description ( RW): Default install

In this example, the Host UUID of the only host is 71739b18-2999-4794-a024-87d5d26215d1.

Posted by Uli Köhler in Virtualization

How to filter ways by tag from .osm.pbf file using Python & Osmium

In our previous post Minimal example how to read .osm.pbf file using Python & osmium we investigated how to read a .osm.pbf file and count all nodes, ways and relations.

Today, we’ll investigate how to filter for specific ways by tag using osmium working on .osm.pbf files. In this example, we’ll filter by highwaytrunk

#!/usr/bin/env python3
import osmium as osm
from collections import namedtuple
import itertools

Way = namedtuple("Way", ["id", "nodes", "tags"])

class FindHighwayTrunks(osm.SimpleHandler):
    """
    Find ways with "highway: trunk" tag
    """

    def __init__(self):
        osm.SimpleHandler.__init__(self)
        self.ways = []

    def way(self, way):
        # If this way is a highway: trunk, ...
        if way.tags.get("highway") == "trunk":
            # ... add it to self.ways
            # NOTE: We may not keep a reference to the way object,
            # so we have to create a new Way object
            nodes = [node.ref for node in way.nodes]
            self.ways.append(Way(way.id, nodes, dict(way.tags)))

# Find ways with the given tag
way_finder = FindHighwayTrunks()
way_finder.apply_file("kenya-latest.osm.pbf")

print(f"Found {len(way_finder.ways)} ways")

This example uses kenya-latest.osm.pbf downloaded from Geofabrik, but you can use any .osm.pbf file. Parsing said file takes about 10 seconds on my Desktop.

Note that we can’t just self.ways.append(way) because way is a temporary/internal Protobuf object and may not be used directly since. Hence, we create our own object that just wraps the extracted contents of way: Its id, its list of nodes (node IDs) and a dictionary containing the tags. Otherwise, you’ll see this error message:

Traceback (most recent call last):
  File "run.py", line 32, in <module>
    trunk_handler.apply_file("kenya-latest.osm.pbf")
RuntimeError: Way callback keeps reference to OSM object. This is not allowed.
Posted by Uli Köhler in Geography, Geoinformatics, OpenStreetMap, Python

How to install cartopy on Ubuntu

First, you need to install libproj (which is a dependency of cartopy) using

sudo apt -y install libproj-dev

After that, install cartopy using

sudo pip3 install cartopy

 

Posted by Uli Köhler in Cartopy, Geoinformatics, Python

How fix Cartopy pip install error ‘Proj 4.9.0 must be installed.’

Problem:

When trying to install cartopy using e.g. pip3 install cartopy you see this error message:

Collecting cartopy
  Downloading Cartopy-0.19.0.post1.tar.gz (12.1 MB)
     |████████████████████████████████| 12.1 MB 13.2 MB/s 
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  ERROR: Command errored out with exit status 1:
   command: /usr/bin/python3 /tmp/tmp2q7tvpo8 get_requires_for_build_wheel /tmp/tmpfwd5htse
       cwd: /tmp/pip-install-dg2i313t/cartopy
  Complete output (1 lines):
  Proj 4.9.0 must be installed.
  ----------------------------------------
ERROR: Command errored out with exit status 1: /usr/bin/python3 /tmp/tmp2q7tvpo8 get_requires_for_build_wheel /tmp/tmpfwd5htse Check the logs for full command output.

Solution:

Install proj using e.g.

sudo apt -y install libproj-dev

and then install cartopy again using e.g.

sudo pip3 install cartopy

 

Posted by Uli Köhler in Cartopy, Geoinformatics, Python

How I use lame to encode Music to mp3

lame -h -V 3 mymusic.wav

This uses variable bitrate mode with -h (high quality but slightly slower) and produces the following frame bitrate histogram:

LAME 3.100 64bits (http://lame.sf.net)
Using polyphase lowpass filter, transition band: 18000 Hz - 18581 Hz
Encoding mymusic.wav
      to mymusic.mp3
Encoding as 48 kHz j-stereo MPEG-1 Layer III VBR(q=3)
    Frame          |  CPU time/estim | REAL time/estim | play/CPU |    ETA 
 11459/11459 (100%)|    0:05/    0:05|    0:05/    0:05|   49.421x|    0:00 
 32 [  118] %*****
 40 [    0] 
 48 [    0] 
 56 [    0] 
 64 [    1] *
 80 [    0] 
 96 [   26] **
112 [  280] %************
128 [ 1412] %%%%%%%%%*******************************************************
160 [ 4372] %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%***************************************************************************************************************************************************
192 [ 2319] %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%*************************************************************
224 [  864] %%%%%%%%%%%%***************************
256 [ 1206] %%%%%%%%%%%%%%%%%**************************************
320 [  861] %%%%%%%%%%%%%%%%***********************
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
   kbps        LR    MS  %     long switch short %
  186.8       28.3  71.7        70.8  12.5  16.7
Writing LAME Tag...done
ReplayGain: -9.7dB

 

Posted by Uli Köhler in Data science

What does WireGuard AllowedIPs actually do?

Wireguard’s allowed_ips field does two different things. Let’s consider the following WireGuard config (generated by the WireguardConfig Site2Site example):

[Interface]
# Name = office1.mydomain.org
PrivateKey = ......
Address = 10.82.85.1/24
ListenPort = 19628

[Peer]
# Name = office2.mydomain.org
PublicKey = ...
AllowedIPs = 10.82.85.2/32, 192.168.200.0/24
PersistentKeepalive = 60

We can see that for the peer office2.mydomain.org the AllowedIPs field is set to 10.82.85.2/32, 192.168.200.0/24.

AllowedIPs does two things:

  • It adds a route to the given networks, i.e. packets addressed to 10.82.85.2/32 or to 192.168.200.0/24 will be routed through the WireGuard interface to that peer
  • It will allow packets with the source IPs 10.82.85.2/32 or 192.168.200.0/24 to be routed from the given peer on the WireGuard interface

Note especially the second point. Any packet from the given peer with a source IP address which is not listed in AllowedIPs will be discarded! While this does not replace a firewall, it serves a an integral part of Wireguard’s security model.

Posted by Uli Köhler in VPN, Wireguard

How to migrate from OpenVPN to Wireguard for Site-to-Site VPNs

The following diagram will assist you in transitioning your VPNs from OpenVPN to Wireguard.

Many users have difficulties in grasping the allowed_ips concept, even though it’s mostly similar to OpenVPN’s route and the architecture where no. In most usecases, you can consider any WireGuard instance with ListenPort = ... to be a server and any WireGuard instance with Endpoint = ... defined for a given peer as client.

Diagram source

The WireGuard config is available as WireguardConfig example Site2Site.json

Posted by Uli Köhler in OpenVPN, VPN, Wireguard

How to catch ANY exception in gdb

In gdb, run

catch throw

in order to break on any exception that is being thrown.

If you want to only break on a specific exception type instead, use

catch throw [exception type]

 

Posted by Uli Köhler in C/C++

How to compile boost unit test program using CMake

Add this to your CMakeLists.txt:

#
# Compile test suite
#
find_package(Boost COMPONENTS system filesystem unit_test_framework REQUIRED)

add_executable(test_myprogram
    tests/MyTest.cpp
)
target_include_directories(test_myprogram PUBLIC "${CMAKE_CURRENT_LIST_DIR}/include")
target_compile_features(test_myprogram PRIVATE cxx_std_17)

target_link_libraries(test_myprogram
                      ${Boost_FILESYSTEM_LIBRARY}
                      ${Boost_SYSTEM_LIBRARY}
                      ${Boost_UNIT_TEST_FRAMEWORK_LIBRARY}
                      )
add_test(test_myprogram test_myprogram)
# make "make test" run the test program
add_custom_target(test
     DEPENDS myprogram
     COMMAND ./test_myprogram
)

In order to build the program and the test, run

make

In order to run the tests, use

make test
Posted by Uli Köhler in Boost, C/C++, CMake

How to fix CMake error Could not find a package configuration file provided by “boost_unit_test_framework”

Problem:

When compiling your CMake-based C++

CMake Error at /usr/lib/x86_64-linux-gnu/cmake/Boost-1.71.0/BoostConfig.cmake:117 (find_package):
  Could not find a package configuration file provided by
  "boost_unit_test_framework" (requested version 1.71.0) with any of the
  following names:

    boost_unit_test_frameworkConfig.cmake
    boost_unit_test_framework-config.cmake

  Add the installation prefix of "boost_unit_test_framework" to
  CMAKE_PREFIX_PATH or set "boost_unit_test_framework_DIR" to a directory
  containing one of the above files.  If "boost_unit_test_framework" provides
  a separate development package or SDK, be sure it has been installed.
Call Stack (most recent call first):
  /usr/lib/x86_64-linux-gnu/cmake/Boost-1.71.0/BoostConfig.cmake:182 (boost_find_component)
  /usr/share/cmake-3.16/Modules/FindBoost.cmake:443 (find_package)
  OCCUtils/CMakeLists.txt:58 (find_package)


-- Configuring incomplete, errors occurred!

Solution:

The error message is trying to tell you that you don’t have the boost unit test library installed (or CMake can’t find it).

On Ubuntu, for example, install it using

sudo apt -y install libboost-test-dev

On Windows you can install it using the prebuilt boost binaries for Windows.

Posted by Uli Köhler in C/C++, CMake

How to fix CMake error Could not find a package configuration file provided by “boost_filesystem”

Problem:

When compiling your CMake-based C++

CMake Error at /usr/lib/x86_64-linux-gnu/cmake/Boost-1.71.0/BoostConfig.cmake:117 (find_package):
  Could not find a package configuration file provided by "boost_filesystem"
  (requested version 1.71.0) with any of the following names:

    boost_filesystemConfig.cmake
    boost_filesystem-config.cmake

  Add the installation prefix of "boost_filesystem" to CMAKE_PREFIX_PATH or
  set "boost_filesystem_DIR" to a directory containing one of the above
  files.  If "boost_filesystem" provides a separate development package or
  SDK, be sure it has been installed.
Call Stack (most recent call first):
  /usr/lib/x86_64-linux-gnu/cmake/Boost-1.71.0/BoostConfig.cmake:182 (boost_find_component)
  /usr/share/cmake-3.16/Modules/FindBoost.cmake:443 (find_package)
  CMakeLists.txt:58 (find_package)

Solution:

The error message is trying to tell you that you don’t have the boost filesystem library installed (or CMake can’t find it).

On Ubuntu, for example, install it using

sudo apt -y install libboost-filesystem-dev

On Windows you can install it using the prebuilt boost binaries for Windows.

Posted by Uli Köhler in C/C++, CMake

Minimal Boost C++ Unit test example

#define BOOST_TEST_MAIN
#include <boost/test/unit_test.hpp>

BOOST_AUTO_TEST_CASE( test1 )
{
    BOOST_CHECK( 2 == 1 );
}

 

Posted by Uli Köhler in C/C++

How to fix apt update EXPKEYSIG 1F3045A5DF7587C3

Problem:

When running sudo apt update you see the following error message:

Err:19 https://josm.openstreetmap.de/apt focal InRelease
  The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 130A439C78FC0F87
Reading package lists... Done
W: GPG error: https://josm.openstreetmap.de/apt focal InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 130A439C78FC0F87

Solution:

Add the JOSM key using

wget -q https://josm.openstreetmap.de/josm-apt.key -O- | sudo apt-key add -

After that, the error should be fixed and you can run

sudo apt update

again.

Posted by Uli Köhler in Linux