How to perform bitwise boolean operations on bytes() in Python3

Performing bitwise operations on bytes() instances in Python3.2+ is easy but not straightforward:

  1. Use int.from_bytes(...) to acquire an integer representing the byte array
  2. Perform bitwise operations with said integer
  3. Use result.to_bytes(...) to convert back the integer to a bytes() array

Note that for the result to make any sense, you need to ensure that both bytes() instances have the same length.

Python code:

def bitwise_and_bytes(a, b):
    result_int = int.from_bytes(a, byteorder="big") & int.from_bytes(b, byteorder="big")
    return result_int.to_bytes(max(len(a), len(b)), byteorder="big")

def bitwise_or_bytes(a, b):
    result_int = int.from_bytes(a, byteorder="big") | int.from_bytes(b, byteorder="big")
    return result_int.to_bytes(max(len(a), len(b)), byteorder="big")

def bitwise_xor_bytes(a, b):
    result_int = int.from_bytes(a, byteorder="big") ^ int.from_bytes(b, byteorder="big")
    return result_int.to_bytes(max(len(a), len(b)), byteorder="big")

Example usage:

a = bytes([0x00, 0x01, 0x02, 0x03])
b = bytes([0x03, 0x02, 0x01, 0xff])

print(bitwise_and_bytes(a, b)) # b'\x00\x00\x00\x03'
print(bitwise_or_bytes(a, b)) # b'\x03\x03\x03\xff'
print(bitwise_xor_bytes(a, b)) # b'\x03\x03\x03\xfc'

 

Posted by Uli Köhler in Python

Bitwise operation with IPv6 addresses and networks in Python

Python3 features the easy-to-use ipaddress library providing many calculations. However, bitwise boolean operators are not available for addresses.

This post shows you how to perform bitwise operations with IPv6Address() objects. We’ll use the following strategy:

  1. Use .packed to get a binary bytes() instance of the IP address
  2. Use int.from_bytes() to acquire an integer representing the binary address
  3. Perform bitwise operations with said integer
  4. Use result.to_bytes(16, ...) to convert back the integer to a bytes() array in the correct byte order
  5. Construct an IPv6Address() object from the resulting byte array.

Python code:

import ipaddress

def bitwise_and_ipv6(addr1, addr2):
    result_int = int.from_bytes(addr1.packed, byteorder="big") & int.from_bytes(addr2.packed, byteorder="big")
    return ipaddress.IPv6Address(result_int.to_bytes(16, byteorder="big"))

def bitwise_or_ipv6(addr1, addr2):
    result_int = int.from_bytes(addr1.packed, byteorder="big") | int.from_bytes(addr2.packed, byteorder="big")
    return ipaddress.IPv6Address(result_int.to_bytes(16, byteorder="big"))

def bitwise_xor_ipv6(addr1, addr2):
    result_int = int.from_bytes(addr1.packed, byteorder="big") ^ int.from_bytes(addr2.packed, byteorder="big")
    return ipaddress.IPv6Address(result_int.to_bytes(16, byteorder="big"))

Example usage:

a = ipaddress.IPv6Address('2001:16b8:2703:8835:9ec7:a6ff:febe:96b1')
b = ipaddress.IPv6Address('2001:16b8:2703:4241:9ec7:a6ff:febe:96b1')

print(bitwise_and_ipv6(a, b)) # IPv6Address('2001:16b8:2703:1:9ec7:a6ff:febe:96b1')
print(bitwise_or_ipv6(a, b)) # IPv6Address('2001:16b8:2703:ca75:9ec7:a6ff:febe:96b1')
print(bitwise_xor_ipv6(a, b)) # IPv6Address('0:0:0:ca74::')

Similarly, you can use the code in order to manipulate IPv6Network() instances:

a = ipaddress.IPv6Network('2001:16b8:2703:8835:9ec7:a6ff:febe::/112')
b = ipaddress.IPv6Network('2001:16b8:2703:4241:9ec7:a6ff:febe::/112')

print(bitwise_and_ipv6(a.network_address, b.network_address)) # IPv6Address('2001:16b8:2703:1:9ec7:a6ff:febe:0')
print(bitwise_or_ipv6(a.network_address, b.network_address)) # IPv6Address('2001:16b8:2703:ca75:9ec7:a6ff:febe:0')
print(bitwise_xor_ipv6(a.network_address, b.network_address)) # IPv6Address('0:0:0:ca74::')

Note that the return type will always be IPv6Address() and never IPv6Network() since the result of the bitwise operation doesn’t have any netmask associated with it.

Besides .network_address you can also use other properties of IPv6Address() instances like .broadcast_address or .hostmask or .netmask.

Posted by Uli Köhler in Networking, Python

How to get current page URL in pyppeteer

In pyppeteer you can use

url = await page.evaluate("() => window.location.href")

in order to get the current URL. Note that page.evaluate() runs whatever Javascript your give it – hence you can use your Javascript skills in order to create the desired effect.

Full example

import asyncio
from pyppeteer import launch

async def main():
    browser = await launch()
    page = await browser.newPage()
    await page.goto('https://www.techoverflow.net')

    # Get the URL and print it
    url = await page.evaluate("() => window.location.href")
    print(url) # prints https://www.techoverflow.net/

    # Cleanuip
    await browser.close()

asyncio.get_event_loop().run_until_complete(main())

 

Posted by Uli Köhler in Pyppeteer, Python

How simulate click using pyppeteer

In order to click a button or a link using the the pyppeteer library, you can use page.evaluate().

If you have an <button> element or a link (<a>) like

<button id="mybutton">

you can use

# Now click the search button    
await page.evaluate(f"""() => {{
    document.getElementById('mybutton').dispatchEvent(new MouseEvent('click', {{
        bubbles: true,
        cancelable: true,
        view: window
    }}));
}}""")

in order to generate a MouseEvent that simulates a click. Note that page.evaluate() will run any Javascript code you pass to it, so you can use your Javascript skills in order to create the desired effect

Also see https://gomakethings.com/how-to-simulate-a-click-event-with-javascript/ for more details on how to simulate mouse clicks in pure Javascript without relying on jQuery.

Note that page.evaluate() will just run any Javascript code you give it, so you can put your Javascript skills to use in order to manipulate the page.

Full example

This example will open https://techoverflow.net, enter a search term into the search field, click the search button and then create a screenshot

import asyncio
from pyppeteer import launch

async def main():
    browser = await launch()
    page = await browser.newPage()
    await page.goto('https://techoverflow.net')

    # Fill content into the search field
    content = "pypetteer"
    await page.evaluate(f"""() => {{
        document.getElementById('s').value = '{content}';
    }}""")

    # Now click the search button    
    await page.evaluate(f"""() => {{
        document.getElementById('searchsubmit').dispatchEvent(new MouseEvent('click', {{
            bubbles: true,
            cancelable: true,
            view: window
        }}));
    }}""")

    # Wait until search results page has been loaded
    await page.waitForSelector(".archive-title")

    # Now take screenshot and exit
    await page.screenshot({'path': 'screenshot.png'})
    await browser.close()

asyncio.get_event_loop().run_until_complete(main())

The result will look like this:

Posted by Uli Köhler in Pyppeteer, Python

How to fill <input> field using pyppeteer

In order to fill an input field using the pyppeteer library, you can use page.evaluate().

If you have an <input> element like

<input name="myinput" id="myinput" type="text">

you can use

content = "My content" # This will be filled into <input id="myinput"> !
await page.evaluate(f"""() => {{
    document.getElementById('myinput').value = '{content}';
}}""")

Note that page.evaluate() will just run any Javascript code you give it, so you can put your Javascript skills to use in order to manipulate the page.

Full example

This example will open https://techoverflow.net, enter a search term into the search field and then create a screenshot

#!/usr/bin/env python3
import asyncio
from pyppeteer import launch

async def main():
    browser = await launch()
    page = await browser.newPage()
    await page.goto('https://techoverflow.net')
    
    # This example fills content into the search field
    content = "My search term"
    await page.evaluate(f"""() => {{
        document.getElementById('s').value = '{content}';
    }}""")

    # Make screenshot
    await page.screenshot({'path': 'screenshot.png'})
    await browser.close()

asyncio.get_event_loop().run_until_complete(main())

The result will look like this:

Posted by Uli Köhler in Pyppeteer, Python

Traefik docker-compose configuration with secure dashboard and Let’s Encrypt

This configuration only provides only the minimum to get the Traefik Dashboard running with Let’s Encrypt-driven SSL encryption and user authentication. It also redirects all HTTP requests to HTTPS in order to avoid insecure access to the Dashboard and other services.

Let’s encrypt is used with the HTTP-01 challenge. This means that Traefik MUST be reachable by Port 80 from the Internet.

In order to install docker & docker-compose, see How to install docker and docker-compose on Ubuntu in 30 seconds.

First prepare the directory (/var/lib/traefik):

sudo mkdir /var/lib/traefik
sudo chown -R $USER: /var/lib/traefik
cd /var/lib/traefik
mkdir acme conf

Now create docker-compose.yml:

version: "3.3"

services:
  traefik:
    image: "traefik:v2.3"
    container_name: "traefik"
    command:
      - "--api=true"
    ports:
      - "80:80"
      - "443:443"
      - "8080:8080"
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
      - "./acme:/etc/traefik/acme"
      - "./traefik.toml:/etc/traefik/traefik.toml"
      - "./conf:/etc/traefik/conf"

Now create the main traefik.toml configuration file:

defaultEntryPoints = ["http", "https"]

[api]
dashboard = true

# You can create config files in /var/lib/traefik/traefik.conf and Traefik will automatically reload them
[providers]
[providers.file]
directory = "/etc/traefik/conf/"
watch = true

# Change this to INFO if you don't want as much debug output
[log]
level = "DEBUG"

[entryPoints.web]
address = ":80"
[entryPoints.web.http]
[entryPoints.web.http.redirections]
[entryPoints.web.http.redirections.entryPoint]
to = "websecure"
scheme = "https"
[entryPoints.websecure]
address = ":443"

[certificatesResolvers.letsencrypttls.acme]
# TODO Add your email here
email = "techoverflow@example.com"
storage = "/etc/traefik/acme/acme.json"
[certificatesResolvers.letsencrypttls.acme.httpChallenge]
entryPoint = "web"

Now we need to create the API config file in conf/api.toml:

[http.routers.traefik-api]
# TODO: Set your domain here !!!
rule = "Host(`traefik.example.com`)"
service = "api@internal"
middlewares = ["auth"]
[http.routers.traefik-api.tls]
certresolver = "letsencrypttls"
[http.middlewares.auth.basicAuth]
# TODO Add your admin user & password here, generate e.g. using https://wtools.io/generate-htpasswd-online
users = [
  "admin:$1$ySFBr~_y$GsKgEasDQkpCX8sO8vNia0",
]

Don’t forget to change your email address and the domain name in the config files (marked by TODO). Ensure you have setup all DNS records correctly so that your domains points to the server running Traefik!

Now it’s time to startup Traefik for the first time:

docker-compose up

Traefik will take a few seconds to automatically generate the Let’s Encrypt certificate for your domain. Once you see a message like

traefik    | time="2020-09-20T23:48:30Z" level=debug msg="Certificates obtained for domains [traefik.mydomain.com]" providerName=letsencrypttls.acme routerName=traefik-api@file rule="Host(`traefik.mydomain.com`)"

the certificate is available and loaded automatically.

Now you can go to https://traefik.mydomain.com/ , login with the username and password you have generated and check out the dashboard.

 

If desired, you can also setup a systemd service to automatically start Traefik on boot (generated using docker-compose systemd .service generator). In order to do this, first stop the running docker-compose instance using Ctrl-C if you still have the terminal open and docker-compose down.

Now add this as /etc/systemd/system/traefik.service:

[Unit]
Description=traefik
Requires=docker.service
After=docker.service

[Service]
Restart=always
User=root
Group=docker
WorkingDirectory=/var/lib/traefik
# Shutdown container (if running) when unit is stopped
ExecStartPre=/usr/local/bin/docker-compose -f docker-compose.yml down -v
# Start container when unit is started
ExecStart=/usr/local/bin/docker-compose -f docker-compose.yml up
# Stop container when unit is stopped
ExecStop=/usr/local/bin/docker-compose -f docker-compose.yml down -v

[Install]
WantedBy=multi-user.target

and run

sudo systemctl enable traefik.service
sudo systemctl start traefik.service

 

Posted by Uli Köhler in Traefik

How to fix Traefik Dashboard/API 404 page not found “api is not enabled”

Problem:

You are trying to configure the Traefik API/Dashboard in a secure way, but every time you try to access the API, you’re only getting a 404 error (unless you set api.insecure=true)

In the logs, you see an error message like this: (if log.level = "DEBUG"):

traefik    | time="2020-09-20T22:53:51Z" level=error msg="api is not enabled" routerName=my-api@file entryPointName=websecure

Solution:

You have to pass --api=true to Traefik, e.g. using docker-compose:

command:
  - "--api=true"

and also set

[api]
dashboard = true

in your traefik.toml.

After that, restart Traefik and you should be able to access your dashboard at /dashboard.

Credits to multiple GitHub users for the original solution.

Posted by Uli Köhler in Traefik

How to fix Traefik “command traefik error: field not found, node: dnsProvider”

Problem:

You are trying to configure your Traefik server but you see an error message like

traefik    | 2020/09/20 22:07:11 command traefik error: field not found, node: dnsProvider

Solution:

dnsProvider is a configuration option from Traefik 1.x. You need to use provider for Traefik 2.x. Example:

[certificatesResolvers.lecloudflare.acme.dnsChallenge]
provider = "cloudflare"

Full example:

[certificatesResolvers.lecloudflare.acme]
email = "email@domain.com"
storage = "/etc/traefik/acme/acme.json"

[certificatesResolvers.lecloudflare.acme.dnsChallenge]
provider = "cloudflare"



 

 

Posted by Uli Köhler in Traefik

How to run TamperMonkey function after window.location.href change

When automating workflows using TamperMonkey scripts, you often have a situation where you want to run some function, then set window.location.href and after that page has loaded, run another part of your code. However, since the browser physically reloads the page, the Tampermonkey script is also reloaded – hence, your original function will stop being executed.

Fixing this is easy once you grasp the concept:

  1. Before reloading the page, we set a specific key in sessionStorage, indicating what we want to do once the page has finished loading
  2. Then we can set window.location.href
  3. On each page load, we check if the key is set and, if so, we run the appropriate function and delete the key

Note that in the TamperMonkey context, this is limited to pages that are included in the @match configuration of the script (otherwise, the script won’t be executed on the new page that is being loaded):

// ==UserScript==
// @name         TamperMonkey continuation example
// @namespace    http://tampermonkey.net/
// @version      0.1
// @author       You
// @match        https://techoverflow.net
// @grant        none
// ==/UserScript==

(function() {
    'use strict';
    const continuationKey = "_my_script_continuation_key"; // This sessionstorage key

    // Check continuation
    const continuationActions = {
        "printMsg": onPrintMsg
    };
    const _continuation = sessionStorage.getItem(continuationKey);
    if(_continuation) {
        sessionStorage.removeItem(continuationKey);
        const action = continuationActions[_continuation]
        if(action) {action();}
    }

    function onPrintMsg() {
        console.log("This is run after reloading the page");
    }

    function onAltQ() {
        console.log("Will now reload the page...");
        sessionStorage.setItem(continuationKey, "printMsg");
        window.location.href = window.location.origin + "/";
    }

    function onKeydown(evt) {
        // Use https://keycode.info/ to get keys
        if (evt.altKey && evt.keyCode == 81) {
            onAltQ();
        }
    }
    document.addEventListener('keydown', onKeydown, true);
})();

 

Posted by Uli Köhler in Javascript

How to go to URL within same domain in Javascript

In order to go to an URL relative to the current domain name in Javascript, use window.location.origin like this:

window.location.href = window.location.origin + "/myurl"
Posted by Uli Köhler in Javascript

Minimal TamperMonkey template for keypress-triggered events

This template allows you to build your own TamperMonkey scripts that react on keypresses on a specific page.

This script shows an alert message when you press Alt+Q.

// ==UserScript==
// @name         TamperMonkey keypress example
// @namespace    http://tampermonkey.net/
// @version      0.1
// @description  try to take over the world!
// @author       You
// @match        https://techoverflow.net/*
// @grant        none
// ==/UserScript==

(function() {
    'use strict';

    function onAltQ() {
        alert("Alt Q pressed!");
    }

    function onKeydown(evt) {
        // Use https://keycode.info/ to get keys
        if (evt.altKey && evt.keyCode == 81) {
            onAltQ();
        }
    }
    document.addEventListener('keydown', onKeydown, true);
})();

 

Posted by Uli Köhler in Javascript

How to fix OpenVPN “TLS Error: cannot locate HMAC in incoming packet from …”

Problem:

Your OpenVPN clients can’t connect to your OpenVPN server and the server log shows an error message like

TLS Error: cannot locate HMAC in incoming packet from [AF_INET6]::ffff:187.100.14.13:41874 (via ::ffff:25.16.25.29%xn0)

Solution:

You have enabled a TLS key (tls-auth option) in your OpenVPN configuration, but your client does not know that it should use the additional layer of authentication.

The server is looking for the HMAC in the incoming packets but can’t find it.

Either disable the tls-auth option in your server config. The config line will look like

tls-auth /var/etc/openvpn/server2.tls-auth 0

or

Enable the correct tls-auth configuration in your client. Remember that you also need to share the correct key.

Posted by Uli Köhler in Networking, OpenVPN, VPN

How to get duration of WAV file in Python (minimal example)

Use

duration_seconds = mywav.getnframes() / mywav.getframerate()

to get the duration of a WAV file in seconds.

Full example:

import wave

with wave.open("myaudio.wav") as mywav:
    duration_seconds = mywav.getnframes() / mywav.getframerate()
    print(f"Length of the WAV file: {duration_seconds:.1f} s")

 

Posted by Uli Köhler in Audio, Python

How to delete all .DS_Store files recursively on the command line

Use this command to recursively delete all .DS_Store files present in the directory mydir and all its subdirectories:

find mydir -type f -name .DS_Store -print0 | xargs -0 rm

Warning: Deleting the files can’t be undone.

Posted by Uli Köhler in Linux

How to fix SSH not accepting public key in authorized_keys

Problem:

You have added your SSH public key to a remote server manually or using ssh-copy-id but still you can’t login using that public key

Solution:

Typically this is caused by bad permissions of your ~/.ssh directory and/or your authorized_keys file. Fix that using:

chmod -R 700 ~/.ssh

then try to login again

Posted by Uli Köhler in Linux

How to run systemd timer twice daily

In order to run a systemd timer twice daily at fixed times, use this syntax in the .timer file:

OnCalendar=*-*-* 00,12:00:00

This line means: Run the service on each and every day (*-*-*) at 00:00:00 and 12:00:00 (00,12:00:00)

Posted by Uli Köhler in Linux

Find and remove all empty directories using the Linux command line

In order to find and remove all empty subdirectories in the current directory (.) recursively, use this command:

find . -depth -type d -print0 | xargs -0 rmdir

This command will only remove empty directories! Any file or non-empty directory won’t be modified.

Explanation of what the parts mean:

  • find: Use the find command to find directories recursively
  • .: Start recursing from the current directory. In case you want to start from a different directory, use that directory name here.
  • -type d: Only find directories – ignore files
  • -depth: Before printing a directory name, print all its sub-directory names. This avoids having to run this command repeatedly because the parent directory can’t be removed since its empty sub-directories need to be removed first
  • -print0 When printing all the directories that have been found, print a NUL character between directories. This is required in order to handle spaces in the directory names correctly
  • | xargs: Pipe the directories to xargs, a program that runs
  • -0: Split the input by NUL characters instead of newlines. This corresponds with the -print0 option to find and is required to handle spaces in directory names correctly.
  • rmdir: For each directory found, run rmdir i.e. try to remove the directory if it’s empty.
Posted by Uli Köhler in Linux

How to fix “OTGS Installer, responsible for receiving automated updates for WPML and Toolset, requires the following PHP component(s) in order to function:cURL” on Ubuntu 20.04

 Problem:

Your WordPress installation running on Ubuntu 20.04 shows you this error message:

OTGS Installer, responsible for receiving automated updates for WPML and Toolset, requires the following PHP component(s) in order to function:cURL

Learn more: Minimum WPML requirements

Solution:

You need to install the curl package for your PHP version.

Typically you can use

sudo apt -y install php7.4-curl

Doing this should immediately fix the issue (just reload your WordPress dashboard). In case you’re using PHP 7.2 you need to sudo apt -y install php7.2-curl instead.

In case the issue is still not fixed after doing this, try restarting your webserver (e.g. sudo service apache restart or sudo service nginx restart), try restarting PHP-FPM by using sudo service php7.4-fpm restart or sudo service php7.2-fpm restart or try rebooting your server. In case that doesn’t help either, you need to manually check which PHP instance the wordpress site is using and install the curl package for that PHP instance.

 

Posted by Uli Köhler in PHP, Wordpress