Programming languages

How to use child_process.exec in Koa (async/await)

First install the child-process-promise library

npm i --save child-process-promise

Then you can use it like this:

const router = require('koa-router')();
const {exec} = require('child-process-promise');

router.get('/test', async ctx => {
  const [stdout, stderr] = await exec('python myscript.py')
  const ipv6 = stdout.toString();
  ctx.body = ipv6;
});

 

Posted by Uli Köhler in Javascript

How to format IPv6 address in groups of 32 bits in Python

def format_ipv6addr_group2(addr):
    """
    Format IPv6 addresses in 4 \n-separated groups of 32 bits
    Returns a string
    """
    addr_s = str(addr.exploded)
    return f"{addr_s[:10]}\n{addr_s[10:20]}\n{addr_s[20:30]}\n{addr_s[30:40]}"

Usage example:

addr = ipaddress.IPv6Address("fd66:6cbb:8c10:1234:4567:89ab:cdef:0123")
print(format_ipv6addr_group2(addr))

Output:

fd66:6cbb:
8c10:1234:
4567:89ab:
cdef:0123

 

Posted by Uli Köhler in Python

How to generate random IPv6 addresses in a given network using Python

This code generates random IPv6 addresses in a given network using Python’s ipaddress module:

import ipaddress
import random

def random_ipv6_addr(network):
    """
    Generate a random IPv6 address in the given network
    Example: random_ipv6_addr("fd66:6cbb:8c10::/48")
    Returns an IPv6Address object.
    """
    net = ipaddress.IPv6Network(network)
    # Which of the network.num_addresses we want to select?
    addr_no = random.randint(0, net.num_addresses)
    # Create the random address by converting to a 128-bit integer, adding addr_no and converting back
    network_int = int.from_bytes(net.network_address.packed, byteorder="big")
    addr_int = network_int + addr_no
    addr = ipaddress.IPv6Address(addr_int.to_bytes(16, byteorder="big"))
    return addr

# Usage example
print(random_ipv6_addr("fdce:4879:a1e9::/48"))
# Prints e.g. fdce:4879:a1e9:e351:1a01:be9:4d9a:157d

It works by first converting the IPv6 network address to binary and then adding a random host number. After that, it will be converted back to an IPv6Address object.

Posted by Uli Köhler in Networking, Python

How to modify file inside a ZIP file using Python

Python provides the zipfile module to read and write ZIP files. Our previous posts Python example: List files in ZIP archive and Downloading & reading a ZIP file in memory using Python show how to list and read files inside a ZIP file.

In this example, we will show how to copy the files from one ZIP file to another and modify one of the files in the progress. This is often the case if you want to use ZIP file formats like ODT or LBX as templates, replacing parts of the text content of a file.

import zipfile

with zipfile.ZipFile(srcfile) as inzip, zipfile.ZipFile(dstfile, "w") as outzip:
    # Iterate the input files
    for inzipinfo in inzip.infolist():
        # Read input file
        with inzip.open(inzipinfo) as infile:
            if inzipinfo.filename == "test.txt":
                content = infile.read()
                # Modify the content of the file by replacing a string
                content = content.replace("abc", "123")
                # Write conte
                outzip.writestr(inzipinfo.filename, content)
            else: # Other file, dont want to modify => just copy it
                

After opening both the input file and the output ZIP using

with zipfile.ZipFile(srcfile) as inzip, zipfile.ZipFile(dstfile, "w") as outzip:

we iterate through all the files in the input ZIP file:

for inzipinfo in inzip.infolist():

In case we’ve encountered the file we want to modify, which is identified by it’s filename test.txt:

if inzipinfo.filename == "test.txt":

we read and modify the content ….

with inzip.open(inzipinfo) as infile:
    content = infile.read().replace("abc", "123")

… and write the modified content to the output ZIP:

outzip.writestr("test.txt", content)

Otherwise, if the current file is not the file we want to modify,  we just copy the file to the output ZIP using

outzip.writestr(inzipinfo.filename, infile.read())

Note that the algorithm will always .read() the file from the input ZIP, hence its entire content will be temporarily stored in memory. Therefore, it doesn’t work well for files which are large when uncompressed.

Posted by Uli Köhler in Python

Where to find info about mbed mbed_app.json overridable parameters?

The first resource can have a look at is the platform configuration option page. Additionally, check the manual on how to use the mbed CLI to show configuration options.

Additionally, you can look at targets.json on GitHub:

For example, "Target" => "config" => "default-adc-vref" would need to be entered like this into mbed_app.json:

{
    "target_overrides": {
      "*": {
          "target.default-adc-vref": 3300
      }
    }
}

 

Posted by Uli Köhler in C/C++, mbed, PlatformIO

How to fix mbed AnalogIn.read_voltage() returning nan or 0.000 (PlatformIO)

Problem:

You are trying to read an ADC voltage in mbed / PlatformIO like this:

AnalogIn myADC(PA_5); 
// Read and print voltage, then return
float v = myADC.read_voltage();
printf("%f\n", v);

but this only prints nan or 0.000.

Solution:

mbed doesn’t know the reference voltage for your platform. The easiest method is to provide the referene voltage in the constructor of AnalogIn:

AnalogIn myADC(PA_5, 3.3);

This specifies a reference voltage of 3.3V. While this applies to most applications in their default configuration, note that the reference voltage might be different depending on the configuration of your microcontroller.

In my experience, it’s almost always better to experimentally verify the reference voltage instead of trying to theorize about it if it’s not immediately obvious.

Full example:

#include <mbed.h>

BufferedSerial pc(USBTX, USBRX, 115200); // tx, rx

AnalogIn   myADC(PA_5, 3.3);

FileHandle *mbed::mbed_override_console(int fd) {
    return &pc;
}

int main() {
  while(1) {
    float v = myADC.read_voltage();
    printf("%f\n", v);
    ThisThread::sleep_for(100ms);
  }
}
{
    "target_overrides": {
      "*": {
        "target.printf_lib": "std"
      }
    }
}
Posted by Uli Köhler in C/C++, mbed, PlatformIO

How to fix mbed printf() ignoring decimals in PlatformIO

Problem:

You are using code like

printf("%.2f\n", myFloat);

in your mbed/PlatformIO application, but instead of printing myFloat with 2 decimal places, it always prints it with 6 decimal places (like 0.000000).

Solution:

mbed uses the minimal-printf library by default which is configured to save space on the Microcontroller. Hence, float max decimals support is disabled by default. In order to get all printf features at the expense of more flash usage and much slower executing, us add mbed_app.json in the root directory of the PlatformIO project with "target.printf_lib": "std":

{
    "target_overrides": {
      "*": {
        "target.printf_lib": "std"
      }
    }
}

See the platform configuration option page for more details and similar options.

Posted by Uli Köhler in C/C++, mbed, PlatformIO

How to fix mbed printf() printing literal %f in PlatformIO

Problem:

You are using code like

printf("%f\n", myFloat);

in your mbed/PlatformIO application, but instead of printing myFloat it prints literal %f.

Solution:

mbed uses the minimal-printf library by default which is configured to save space on the Microcontroller. Hence, float support (i.e. %f support) is disabled by default. You need to enable it by adding mbed_app.json in the root directory of the PlatformIO project with "platform.minimal-printf-enable-floating-point": true:

{
    "target_overrides": {
      "*": {
        "platform.minimal-printf-enable-floating-point": true
      }
    }
}

See the platform configuration option page for more details and similar options.

Posted by Uli Köhler in C/C++, mbed, PlatformIO

How to fix Python asyncio RuntimeError: There is no current event loop in thread …

Problem:

You are trying to run your Python application, but you see an error message like

Traceback (most recent call last):
  [...]
  File "/mnt/KATranslationCheck/CrowdinLogin.py", line 38, in get_crowdin_tokens
    return asyncio.get_event_loop().run_until_complete(async_get_crowdin_tokens(username, password))
  File "/usr/lib/python3.6/asyncio/events.py", line 694, in get_event_loop
    return get_event_loop_policy().get_event_loop()
  File "/usr/lib/python3.6/asyncio/events.py", line 602, in get_event_loop
    % threading.current_thread().name)
RuntimeError: There is no current event loop in thread 'worker 3'.

Solution:

You are trying to run asyncio.get_event_loop() in some thread other than the main thread – however, asyncio only generates an event loop for the main thread.

Use this function instead of asyncio.get_event_loop():

import asyncio

def get_or_create_eventloop():
    try:
        return asyncio.get_event_loop()
    except RuntimeError as ex:
        if "There is no current event loop in thread" in str(ex):
            loop = asyncio.new_event_loop()
            asyncio.set_event_loop(loop)
            return asyncio.get_event_loop()

It will first try asyncio.get_event_loop(). In case that doesn’t work, it will generate a new event loop for the current thread using

loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)

and then returns this event loop.

Note that while this works well for generating the event loop, but depending on the way you use the event loop, you might encounter further error messages like

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/pyppeteer/launcher.py", line 305, in launch
    return await Launcher(options, **kwargs).launch()
  File "/usr/local/lib/python3.6/dist-packages/pyppeteer/launcher.py", line 157, in launch
    signal.signal(signal.SIGINT, _close_process)
  File "/usr/lib/python3.6/signal.py", line 47, in signal
    handler = _signal.signal(_enum_to_int(signalnum), _enum_to_int(handler))
ValueError: signal only works in main thread

that are usually not as easy to fix and require some restructuring of your program.

Posted by Uli Köhler in Python

C# minimal program that shows a message popup dialog

This program just shows a message box and exits:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Windows.Forms;

namespace MinimalMessageBoxProgram
{
    class MinimalMessageBoxProgram
    {
        public static void Main(string[] args)
        {
            MessageBox.Show("This is the minimal text shown in the message dialog.");
        }
    }
}

 

Posted by Uli Köhler in C#

How to fix pyppeteer pyppeteer.errors.BrowserError: Browser closed unexpectedly:

Problem:

You want to run your Pyppeteer application on Linux, but you see an error message like

Traceback (most recent call last):
  File "PyppeteerExample.py", line 15, in <module>
    asyncio.get_event_loop().run_until_complete(main())
  File "/usr/lib/python3.6/asyncio/base_events.py", line 484, in run_until_complete
    return future.result()
  File "PyppeteerExample.py", line 6, in main
    browser = await launch()
  File "/usr/local/lib/python3.6/dist-packages/pyppeteer/launcher.py", line 305, in launch
    return await Launcher(options, **kwargs).launch()
  File "/usr/local/lib/python3.6/dist-packages/pyppeteer/launcher.py", line 166, in launch
    self.browserWSEndpoint = get_ws_endpoint(self.url)
  File "/usr/local/lib/python3.6/dist-packages/pyppeteer/launcher.py", line 225, in get_ws_endpoint
    raise BrowserError('Browser closed unexpectedly:\n')
pyppeteer.errors.BrowserError: Browser closed unexpectedly:

Solution:

In most cases, the underlying error for this error message is Puppetteer’s libX11-xcb.so.1: cannot open shared object file: No such file or directory. In order to fix that, you need to install dependency libraries for Chromium which is used internally by Puppeteer / Pyppeteer:

sudo apt install -y gconf-service libasound2 libatk1.0-0 libc6 libcairo2 libcups2 libdbus-1-3 libexpat1 libfontconfig1 libgcc1 libgconf-2-4 libgdk-pixbuf2.0-0 libglib2.0-0 libgtk-3-0 libnspr4 libpango-1.0-0 libpangocairo-1.0-0 libstdc++6 libx11-6 libx11-xcb1 libxcb1 libxcomposite1 libxcursor1 libxdamage1 libxext6 libxfixes3 libxi6 libxrandr2 libxrender1 libxss1 libxtst6 ca-certificates fonts-liberation libappindicator1 libnss3 lsb-release xdg-utils wget

 

Posted by Uli Köhler in Pyppeteer, Python

How to get hostmask/netmask for given prefix length in Python

In order to get the host mask for e.g. a /112 IPv6 prefix, use:

import ipaddress
# Get netmask for a /112 prefix
ipaddress.IPv6Network("::/112").netmask

# Get host mask for a /112 prefix
ipaddress.IPv6Network("::/112").hostmask

 

Posted by Uli Köhler in Networking, Python

How to fix Python3 TypeError: unsupported operand type(s) for &: ‘bytes’ and ‘bytes’

Problem:

You want to perform bitwise boolean operations on bytes() arrays in Python, but you see an error message like

TypeError: unsupported operand type(s) for &: 'bytes' and 'bytes'

or

TypeError: unsupported operand type(s) for |: 'bytes' and 'bytes'

or

TypeError: unsupported operand type(s) for ^: 'bytes' and 'bytes'

Solution:

Python can’t perform bitwise operations directly on byte arrays. However, you can use the code from How to perform bitwise boolean operations on bytes() in Python3:

def bitwise_and_bytes(a, b):
    result_int = int.from_bytes(a, byteorder="big") & int.from_bytes(b, byteorder="big")
    return result_int.to_bytes(max(len(a), len(b)), byteorder="big")

def bitwise_or_bytes(a, b):
    result_int = int.from_bytes(a, byteorder="big") | int.from_bytes(b, byteorder="big")
    return result_int.to_bytes(max(len(a), len(b)), byteorder="big")

def bitwise_xor_bytes(a, b):
    result_int = int.from_bytes(a, byteorder="big") ^ int.from_bytes(b, byteorder="big")
    return result_int.to_bytes(max(len(a), len(b)), byteorder="big")

# Example usage:

a = bytes([0x00, 0x01, 0x02, 0x03])
b = bytes([0x03, 0x02, 0x01, 0xff])

print(bitwise_and_bytes(a, b)) # b'\x00\x00\x00\x03'
print(bitwise_or_bytes(a, b)) # b'\x03\x03\x03\xff'
print(bitwise_xor_bytes(a, b)) # b'\x03\x03\x03\xfc'
Posted by Uli Köhler in Python

How to perform bitwise boolean operations on bytes() in Python3

Performing bitwise operations on bytes() instances in Python3.2+ is easy but not straightforward:

  1. Use int.from_bytes(...) to acquire an integer representing the byte array
  2. Perform bitwise operations with said integer
  3. Use result.to_bytes(...) to convert back the integer to a bytes() array

Note that for the result to make any sense, you need to ensure that both bytes() instances have the same length.

Python code:

def bitwise_and_bytes(a, b):
    result_int = int.from_bytes(a, byteorder="big") & int.from_bytes(b, byteorder="big")
    return result_int.to_bytes(max(len(a), len(b)), byteorder="big")

def bitwise_or_bytes(a, b):
    result_int = int.from_bytes(a, byteorder="big") | int.from_bytes(b, byteorder="big")
    return result_int.to_bytes(max(len(a), len(b)), byteorder="big")

def bitwise_xor_bytes(a, b):
    result_int = int.from_bytes(a, byteorder="big") ^ int.from_bytes(b, byteorder="big")
    return result_int.to_bytes(max(len(a), len(b)), byteorder="big")

Example usage:

a = bytes([0x00, 0x01, 0x02, 0x03])
b = bytes([0x03, 0x02, 0x01, 0xff])

print(bitwise_and_bytes(a, b)) # b'\x00\x00\x00\x03'
print(bitwise_or_bytes(a, b)) # b'\x03\x03\x03\xff'
print(bitwise_xor_bytes(a, b)) # b'\x03\x03\x03\xfc'

 

Posted by Uli Köhler in Python

Bitwise operation with IPv6 addresses and networks in Python

Python3 features the easy-to-use ipaddress library providing many calculations. However, bitwise boolean operators are not available for addresses.

This post shows you how to perform bitwise operations with IPv6Address() objects. We’ll use the following strategy:

  1. Use .packed to get a binary bytes() instance of the IP address
  2. Use int.from_bytes() to acquire an integer representing the binary address
  3. Perform bitwise operations with said integer
  4. Use result.to_bytes(16, ...) to convert back the integer to a bytes() array in the correct byte order
  5. Construct an IPv6Address() object from the resulting byte array.

Python code:

import ipaddress

def bitwise_and_ipv6(addr1, addr2):
    result_int = int.from_bytes(addr1.packed, byteorder="big") & int.from_bytes(addr2.packed, byteorder="big")
    return ipaddress.IPv6Address(result_int.to_bytes(16, byteorder="big"))

def bitwise_or_ipv6(addr1, addr2):
    result_int = int.from_bytes(addr1.packed, byteorder="big") | int.from_bytes(addr2.packed, byteorder="big")
    return ipaddress.IPv6Address(result_int.to_bytes(16, byteorder="big"))

def bitwise_xor_ipv6(addr1, addr2):
    result_int = int.from_bytes(addr1.packed, byteorder="big") ^ int.from_bytes(addr2.packed, byteorder="big")
    return ipaddress.IPv6Address(result_int.to_bytes(16, byteorder="big"))

Example usage:

a = ipaddress.IPv6Address('2001:16b8:2703:8835:9ec7:a6ff:febe:96b1')
b = ipaddress.IPv6Address('2001:16b8:2703:4241:9ec7:a6ff:febe:96b1')

print(bitwise_and_ipv6(a, b)) # IPv6Address('2001:16b8:2703:1:9ec7:a6ff:febe:96b1')
print(bitwise_or_ipv6(a, b)) # IPv6Address('2001:16b8:2703:ca75:9ec7:a6ff:febe:96b1')
print(bitwise_xor_ipv6(a, b)) # IPv6Address('0:0:0:ca74::')

Similarly, you can use the code in order to manipulate IPv6Network() instances:

a = ipaddress.IPv6Network('2001:16b8:2703:8835:9ec7:a6ff:febe::/112')
b = ipaddress.IPv6Network('2001:16b8:2703:4241:9ec7:a6ff:febe::/112')

print(bitwise_and_ipv6(a.network_address, b.network_address)) # IPv6Address('2001:16b8:2703:1:9ec7:a6ff:febe:0')
print(bitwise_or_ipv6(a.network_address, b.network_address)) # IPv6Address('2001:16b8:2703:ca75:9ec7:a6ff:febe:0')
print(bitwise_xor_ipv6(a.network_address, b.network_address)) # IPv6Address('0:0:0:ca74::')

Note that the return type will always be IPv6Address() and never IPv6Network() since the result of the bitwise operation doesn’t have any netmask associated with it.

Besides .network_address you can also use other properties of IPv6Address() instances like .broadcast_address or .hostmask or .netmask.

Posted by Uli Köhler in Networking, Python

How to get current page URL in pyppeteer

In pyppeteer you can use

url = await page.evaluate("() => window.location.href")

in order to get the current URL. Note that page.evaluate() runs whatever Javascript your give it – hence you can use your Javascript skills in order to create the desired effect.

Full example

import asyncio
from pyppeteer import launch

async def main():
    browser = await launch()
    page = await browser.newPage()
    await page.goto('https://www.techoverflow.net')

    # Get the URL and print it
    url = await page.evaluate("() => window.location.href")
    print(url) # prints https://www.techoverflow.net/

    # Cleanuip
    await browser.close()

asyncio.get_event_loop().run_until_complete(main())

 

Posted by Uli Köhler in Pyppeteer, Python

How simulate click using pyppeteer

In order to click a button or a link using the the pyppeteer library, you can use page.evaluate().

If you have an <button> element or a link (<a>) like

<button id="mybutton">

you can use

# Now click the search button    
await page.evaluate(f"""() => {{
    document.getElementById('mybutton').dispatchEvent(new MouseEvent('click', {{
        bubbles: true,
        cancelable: true,
        view: window
    }}));
}}""")

in order to generate a MouseEvent that simulates a click. Note that page.evaluate() will run any Javascript code you pass to it, so you can use your Javascript skills in order to create the desired effect

Also see https://gomakethings.com/how-to-simulate-a-click-event-with-javascript/ for more details on how to simulate mouse clicks in pure Javascript without relying on jQuery.

Note that page.evaluate() will just run any Javascript code you give it, so you can put your Javascript skills to use in order to manipulate the page.

Full example

This example will open https://techoverflow.net, enter a search term into the search field, click the search button and then create a screenshot

import asyncio
from pyppeteer import launch

async def main():
    browser = await launch()
    page = await browser.newPage()
    await page.goto('https://techoverflow.net')

    # Fill content into the search field
    content = "pypetteer"
    await page.evaluate(f"""() => {{
        document.getElementById('s').value = '{content}';
    }}""")

    # Now click the search button    
    await page.evaluate(f"""() => {{
        document.getElementById('searchsubmit').dispatchEvent(new MouseEvent('click', {{
            bubbles: true,
            cancelable: true,
            view: window
        }}));
    }}""")

    # Wait until search results page has been loaded
    await page.waitForSelector(".archive-title")

    # Now take screenshot and exit
    await page.screenshot({'path': 'screenshot.png'})
    await browser.close()

asyncio.get_event_loop().run_until_complete(main())

The result will look like this:

Posted by Uli Köhler in Pyppeteer, Python

How to fill <input> field using pyppeteer

In order to fill an input field using the pyppeteer library, you can use page.evaluate().

If you have an <input> element like

<input name="myinput" id="myinput" type="text">

you can use

content = "My content" # This will be filled into <input id="myinput"> !
await page.evaluate(f"""() => {{
    document.getElementById('myinput').value = '{content}';
}}""")

Note that page.evaluate() will just run any Javascript code you give it, so you can put your Javascript skills to use in order to manipulate the page.

Full example

This example will open https://techoverflow.net, enter a search term into the search field and then create a screenshot

#!/usr/bin/env python3
import asyncio
from pyppeteer import launch

async def main():
    browser = await launch()
    page = await browser.newPage()
    await page.goto('https://techoverflow.net')
    
    # This example fills content into the search field
    content = "My search term"
    await page.evaluate(f"""() => {{
        document.getElementById('s').value = '{content}';
    }}""")

    # Make screenshot
    await page.screenshot({'path': 'screenshot.png'})
    await browser.close()

asyncio.get_event_loop().run_until_complete(main())

The result will look like this:

Posted by Uli Köhler in Pyppeteer, Python

How to run TamperMonkey function after window.location.href change

When automating workflows using TamperMonkey scripts, you often have a situation where you want to run some function, then set window.location.href and after that page has loaded, run another part of your code. However, since the browser physically reloads the page, the Tampermonkey script is also reloaded – hence, your original function will stop being executed.

Fixing this is easy once you grasp the concept:

  1. Before reloading the page, we set a specific key in sessionStorage, indicating what we want to do once the page has finished loading
  2. Then we can set window.location.href
  3. On each page load, we check if the key is set and, if so, we run the appropriate function and delete the key

Note that in the TamperMonkey context, this is limited to pages that are included in the @match configuration of the script (otherwise, the script won’t be executed on the new page that is being loaded):

// ==UserScript==
// @name         TamperMonkey continuation example
// @namespace    http://tampermonkey.net/
// @version      0.1
// @author       You
// @match        https://techoverflow.net
// @grant        none
// ==/UserScript==

(function() {
    'use strict';
    const continuationKey = "_my_script_continuation_key"; // This sessionstorage key

    // Check continuation
    const continuationActions = {
        "printMsg": onPrintMsg
    };
    const _continuation = sessionStorage.getItem(continuationKey);
    if(_continuation) {
        sessionStorage.removeItem(continuationKey);
        const action = continuationActions[_continuation]
        if(action) {action();}
    }

    function onPrintMsg() {
        console.log("This is run after reloading the page");
    }

    function onAltQ() {
        console.log("Will now reload the page...");
        sessionStorage.setItem(continuationKey, "printMsg");
        window.location.href = window.location.origin + "/";
    }

    function onKeydown(evt) {
        // Use https://keycode.info/ to get keys
        if (evt.altKey && evt.keyCode == 81) {
            onAltQ();
        }
    }
    document.addEventListener('keydown', onKeydown, true);
})();

 

Posted by Uli Köhler in Javascript