How to cleanup large gitlab prometheus/data in Omnibus/Docker setting

In many of my dockerized Gitlab instances, the prometheus/data folder was eating up multiple Gigabytes of hard drive space even though I was not using Prometheus at all.

In order to fix this, I first disabled Prometheus in the docker-compose.yml config using

prometheus_monitoring['enable'] = false

Also see How I reduced gitlab memory consumption in my docker-based setup for a detailed explanantion.

After that, you need to restart gitlab in order for the settings change to take effect.

Now you can just delete the Prometheus data folder. Make a backup of the entire gitlab data folder before this step.

Run this command from within your gitlab data folder:

rm -rf prometheus/data

 

Posted by Uli Köhler in Allgemein, Linux

Where to find info about mbed mbed_app.json overridable parameters?

The first resource can have a look at is the platform configuration option page. Additionally, check the manual on how to use the mbed CLI to show configuration options.

Additionally, you can look at targets.json on GitHub:

For example, "Target" => "config" => "default-adc-vref" would need to be entered like this into mbed_app.json:

{
    "target_overrides": {
      "*": {
          "target.default-adc-vref": 3300
      }
    }
}

 

Posted by Uli Köhler in C/C++, mbed, PlatformIO

How to fix mbed AnalogIn.read_voltage() returning nan or 0.000 (PlatformIO)

Problem:

You are trying to read an ADC voltage in mbed / PlatformIO like this:

AnalogIn myADC(PA_5); 
// Read and print voltage, then return
float v = myADC.read_voltage();
printf("%f\n", v);

but this only prints nan or 0.000.

Solution:

mbed doesn’t know the reference voltage for your platform. The easiest method is to provide the referene voltage in the constructor of AnalogIn:

AnalogIn myADC(PA_5, 3.3);

This specifies a reference voltage of 3.3V. While this applies to most applications in their default configuration, note that the reference voltage might be different depending on the configuration of your microcontroller.

In my experience, it’s almost always better to experimentally verify the reference voltage instead of trying to theorize about it if it’s not immediately obvious.

Full example:

#include <mbed.h>

BufferedSerial pc(USBTX, USBRX, 115200); // tx, rx

AnalogIn   myADC(PA_5, 3.3);

FileHandle *mbed::mbed_override_console(int fd) {
    return &pc;
}

int main() {
  while(1) {
    float v = myADC.read_voltage();
    printf("%f\n", v);
    ThisThread::sleep_for(100ms);
  }
}
{
    "target_overrides": {
      "*": {
        "target.printf_lib": "std"
      }
    }
}
Posted by Uli Köhler in C/C++, mbed, PlatformIO

How to fix mbed printf() ignoring decimals in PlatformIO

Problem:

You are using code like

printf("%.2f\n", myFloat);

in your mbed/PlatformIO application, but instead of printing myFloat with 2 decimal places, it always prints it with 6 decimal places (like 0.000000).

Solution:

mbed uses the minimal-printf library by default which is configured to save space on the Microcontroller. Hence, float max decimals support is disabled by default. In order to get all printf features at the expense of more flash usage and much slower executing, us add mbed_app.json in the root directory of the PlatformIO project with "target.printf_lib": "std":

{
    "target_overrides": {
      "*": {
        "target.printf_lib": "std"
      }
    }
}

See the platform configuration option page for more details and similar options.

Posted by Uli Köhler in C/C++, mbed, PlatformIO

How to fix mbed printf() printing literal %f in PlatformIO

Problem:

You are using code like

printf("%f\n", myFloat);

in your mbed/PlatformIO application, but instead of printing myFloat it prints literal %f.

Solution:

mbed uses the minimal-printf library by default which is configured to save space on the Microcontroller. Hence, float support (i.e. %f support) is disabled by default. You need to enable it by adding mbed_app.json in the root directory of the PlatformIO project with "platform.minimal-printf-enable-floating-point": true:

{
    "target_overrides": {
      "*": {
        "platform.minimal-printf-enable-floating-point": true
      }
    }
}

See the platform configuration option page for more details and similar options.

Posted by Uli Köhler in C/C++, mbed, PlatformIO

How to fix mbed error: ‘wait’ was not declared in this scope (PlatformIO)

Problem:

While compiling your mbed / PlatformIO application, you see an error message like

src/actuators.cpp:253:5: error: 'wait' was not declared in this scope
  253 |     wait(1.0);
      |     ^~~~

 

Solution:

wait is an old API and has been deprecated in favour of the C++ standard ThisThread::sleep_for. Use

ThisThread::sleep_for(1s);

 

Posted by Uli Köhler in Embedded, mbed, PlatformIO

How to fix Python asyncio RuntimeError: There is no current event loop in thread …

Problem:

You are trying to run your Python application, but you see an error message like

Traceback (most recent call last):
  [...]
  File "/mnt/KATranslationCheck/CrowdinLogin.py", line 38, in get_crowdin_tokens
    return asyncio.get_event_loop().run_until_complete(async_get_crowdin_tokens(username, password))
  File "/usr/lib/python3.6/asyncio/events.py", line 694, in get_event_loop
    return get_event_loop_policy().get_event_loop()
  File "/usr/lib/python3.6/asyncio/events.py", line 602, in get_event_loop
    % threading.current_thread().name)
RuntimeError: There is no current event loop in thread 'worker 3'.

Solution:

You are trying to run asyncio.get_event_loop() in some thread other than the main thread – however, asyncio only generates an event loop for the main thread.

Use this function instead of asyncio.get_event_loop():

import asyncio

def get_or_create_eventloop():
    try:
        return asyncio.get_event_loop()
    except RuntimeError as ex:
        if "There is no current event loop in thread" in str(ex):
            loop = asyncio.new_event_loop()
            asyncio.set_event_loop(loop)
            return asyncio.get_event_loop()

It will first try asyncio.get_event_loop(). In case that doesn’t work, it will generate a new event loop for the current thread using

loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)

and then returns this event loop.

Note that while this works well for generating the event loop, but depending on the way you use the event loop, you might encounter further error messages like

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/pyppeteer/launcher.py", line 305, in launch
    return await Launcher(options, **kwargs).launch()
  File "/usr/local/lib/python3.6/dist-packages/pyppeteer/launcher.py", line 157, in launch
    signal.signal(signal.SIGINT, _close_process)
  File "/usr/lib/python3.6/signal.py", line 47, in signal
    handler = _signal.signal(_enum_to_int(signalnum), _enum_to_int(handler))
ValueError: signal only works in main thread

that are usually not as easy to fix and require some restructuring of your program.

Posted by Uli Köhler in Python

C# minimal program that shows a message popup dialog

This program just shows a message box and exits:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Windows.Forms;

namespace MinimalMessageBoxProgram
{
    class MinimalMessageBoxProgram
    {
        public static void Main(string[] args)
        {
            MessageBox.Show("This is the minimal text shown in the message dialog.");
        }
    }
}

 

Posted by Uli Köhler in C#

How to fix pyppeteer pyppeteer.errors.BrowserError: Browser closed unexpectedly:

Problem:

You want to run your Pyppeteer application on Linux, but you see an error message like

Traceback (most recent call last):
  File "PyppeteerExample.py", line 15, in <module>
    asyncio.get_event_loop().run_until_complete(main())
  File "/usr/lib/python3.6/asyncio/base_events.py", line 484, in run_until_complete
    return future.result()
  File "PyppeteerExample.py", line 6, in main
    browser = await launch()
  File "/usr/local/lib/python3.6/dist-packages/pyppeteer/launcher.py", line 305, in launch
    return await Launcher(options, **kwargs).launch()
  File "/usr/local/lib/python3.6/dist-packages/pyppeteer/launcher.py", line 166, in launch
    self.browserWSEndpoint = get_ws_endpoint(self.url)
  File "/usr/local/lib/python3.6/dist-packages/pyppeteer/launcher.py", line 225, in get_ws_endpoint
    raise BrowserError('Browser closed unexpectedly:\n')
pyppeteer.errors.BrowserError: Browser closed unexpectedly:

Solution:

In most cases, the underlying error for this error message is Puppetteer’s libX11-xcb.so.1: cannot open shared object file: No such file or directory. In order to fix that, you need to install dependency libraries for Chromium which is used internally by Puppeteer / Pyppeteer:

sudo apt install -y gconf-service libasound2 libatk1.0-0 libc6 libcairo2 libcups2 libdbus-1-3 libexpat1 libfontconfig1 libgcc1 libgconf-2-4 libgdk-pixbuf2.0-0 libglib2.0-0 libgtk-3-0 libnspr4 libpango-1.0-0 libpangocairo-1.0-0 libstdc++6 libx11-6 libx11-xcb1 libxcb1 libxcomposite1 libxcursor1 libxdamage1 libxext6 libxfixes3 libxi6 libxrandr2 libxrender1 libxss1 libxtst6 ca-certificates fonts-liberation libappindicator1 libnss3 lsb-release xdg-utils wget

 

Posted by Uli Köhler in Pyppeteer, Python

Pyppeteer minimal example

This script is a minimal example on how to use Pyppeteer to fetch a web page and extract the page title:

#!/usr/bin/env python3
import asyncio
from pyppeteer import launch

async def main():
    browser = await launch()
    page = await browser.newPage()
    await page.goto('https://www.techoverflow.net')
    # Get the URL and print it
    title = await page.evaluate("() => document.querySelector('.logo-default').textContent")
    print(f"Page title: {title}") # prints Page title: TechOverflow
    # Cleanup
    await browser.close()

asyncio.get_event_loop().run_until_complete(main())

How to run:

sudo pip3 install pyppeteer
python3 PyppeteerExample.py

 

Posted by Uli Köhler in Pyppeteer

How to fix apt update error NO_PUBKEY 78BD65473CB3BD13

If you encounter this error message during APT update:

The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 78BD65473CB3BD13

you need to import the key using

sudo apt-key adv --keyserver hkp://pool.sks-keyservers.net:80 --recv-keys 78BD65473CB3BD13

 

Posted by Uli Köhler in Linux

How to get hostmask/netmask for given prefix length in Python

In order to get the host mask for e.g. a /112 IPv6 prefix, use:

import ipaddress
# Get netmask for a /112 prefix
ipaddress.IPv6Network("::/112").netmask

# Get host mask for a /112 prefix
ipaddress.IPv6Network("::/112").hostmask

 

Posted by Uli Köhler in Networking, Python

How to fix Python3 TypeError: unsupported operand type(s) for &: ‘bytes’ and ‘bytes’

Problem:

You want to perform bitwise boolean operations on bytes() arrays in Python, but you see an error message like

TypeError: unsupported operand type(s) for &: 'bytes' and 'bytes'

or

TypeError: unsupported operand type(s) for |: 'bytes' and 'bytes'

or

TypeError: unsupported operand type(s) for ^: 'bytes' and 'bytes'

Solution:

Python can’t perform bitwise operations directly on byte arrays. However, you can use the code from How to perform bitwise boolean operations on bytes() in Python3:

def bitwise_and_bytes(a, b):
    result_int = int.from_bytes(a, byteorder="big") & int.from_bytes(b, byteorder="big")
    return result_int.to_bytes(max(len(a), len(b)), byteorder="big")

def bitwise_or_bytes(a, b):
    result_int = int.from_bytes(a, byteorder="big") | int.from_bytes(b, byteorder="big")
    return result_int.to_bytes(max(len(a), len(b)), byteorder="big")

def bitwise_xor_bytes(a, b):
    result_int = int.from_bytes(a, byteorder="big") ^ int.from_bytes(b, byteorder="big")
    return result_int.to_bytes(max(len(a), len(b)), byteorder="big")

# Example usage:

a = bytes([0x00, 0x01, 0x02, 0x03])
b = bytes([0x03, 0x02, 0x01, 0xff])

print(bitwise_and_bytes(a, b)) # b'\x00\x00\x00\x03'
print(bitwise_or_bytes(a, b)) # b'\x03\x03\x03\xff'
print(bitwise_xor_bytes(a, b)) # b'\x03\x03\x03\xfc'
Posted by Uli Köhler in Python

How to perform bitwise boolean operations on bytes() in Python3

Performing bitwise operations on bytes() instances in Python3.2+ is easy but not straightforward:

  1. Use int.from_bytes(...) to acquire an integer representing the byte array
  2. Perform bitwise operations with said integer
  3. Use result.to_bytes(...) to convert back the integer to a bytes() array

Note that for the result to make any sense, you need to ensure that both bytes() instances have the same length.

Python code:

def bitwise_and_bytes(a, b):
    result_int = int.from_bytes(a, byteorder="big") & int.from_bytes(b, byteorder="big")
    return result_int.to_bytes(max(len(a), len(b)), byteorder="big")

def bitwise_or_bytes(a, b):
    result_int = int.from_bytes(a, byteorder="big") | int.from_bytes(b, byteorder="big")
    return result_int.to_bytes(max(len(a), len(b)), byteorder="big")

def bitwise_xor_bytes(a, b):
    result_int = int.from_bytes(a, byteorder="big") ^ int.from_bytes(b, byteorder="big")
    return result_int.to_bytes(max(len(a), len(b)), byteorder="big")

Example usage:

a = bytes([0x00, 0x01, 0x02, 0x03])
b = bytes([0x03, 0x02, 0x01, 0xff])

print(bitwise_and_bytes(a, b)) # b'\x00\x00\x00\x03'
print(bitwise_or_bytes(a, b)) # b'\x03\x03\x03\xff'
print(bitwise_xor_bytes(a, b)) # b'\x03\x03\x03\xfc'

 

Posted by Uli Köhler in Python

Bitwise operation with IPv6 addresses and networks in Python

Python3 features the easy-to-use ipaddress library providing many calculations. However, bitwise boolean operators are not available for addresses.

This post shows you how to perform bitwise operations with IPv6Address() objects. We’ll use the following strategy:

  1. Use .packed to get a binary bytes() instance of the IP address
  2. Use int.from_bytes() to acquire an integer representing the binary address
  3. Perform bitwise operations with said integer
  4. Use result.to_bytes(16, ...) to convert back the integer to a bytes() array in the correct byte order
  5. Construct an IPv6Address() object from the resulting byte array.

Python code:

import ipaddress

def bitwise_and_ipv6(addr1, addr2):
    result_int = int.from_bytes(addr1.packed, byteorder="big") & int.from_bytes(addr2.packed, byteorder="big")
    return ipaddress.IPv6Address(result_int.to_bytes(16, byteorder="big"))

def bitwise_or_ipv6(addr1, addr2):
    result_int = int.from_bytes(addr1.packed, byteorder="big") | int.from_bytes(addr2.packed, byteorder="big")
    return ipaddress.IPv6Address(result_int.to_bytes(16, byteorder="big"))

def bitwise_xor_ipv6(addr1, addr2):
    result_int = int.from_bytes(addr1.packed, byteorder="big") ^ int.from_bytes(addr2.packed, byteorder="big")
    return ipaddress.IPv6Address(result_int.to_bytes(16, byteorder="big"))

Example usage:

a = ipaddress.IPv6Address('2001:16b8:2703:8835:9ec7:a6ff:febe:96b1')
b = ipaddress.IPv6Address('2001:16b8:2703:4241:9ec7:a6ff:febe:96b1')

print(bitwise_and_ipv6(a, b)) # IPv6Address('2001:16b8:2703:1:9ec7:a6ff:febe:96b1')
print(bitwise_or_ipv6(a, b)) # IPv6Address('2001:16b8:2703:ca75:9ec7:a6ff:febe:96b1')
print(bitwise_xor_ipv6(a, b)) # IPv6Address('0:0:0:ca74::')

Similarly, you can use the code in order to manipulate IPv6Network() instances:

a = ipaddress.IPv6Network('2001:16b8:2703:8835:9ec7:a6ff:febe::/112')
b = ipaddress.IPv6Network('2001:16b8:2703:4241:9ec7:a6ff:febe::/112')

print(bitwise_and_ipv6(a.network_address, b.network_address)) # IPv6Address('2001:16b8:2703:1:9ec7:a6ff:febe:0')
print(bitwise_or_ipv6(a.network_address, b.network_address)) # IPv6Address('2001:16b8:2703:ca75:9ec7:a6ff:febe:0')
print(bitwise_xor_ipv6(a.network_address, b.network_address)) # IPv6Address('0:0:0:ca74::')

Note that the return type will always be IPv6Address() and never IPv6Network() since the result of the bitwise operation doesn’t have any netmask associated with it.

Besides .network_address you can also use other properties of IPv6Address() instances like .broadcast_address or .hostmask or .netmask.

Posted by Uli Köhler in Networking, Python

How to get current page URL in pyppeteer

In pyppeteer you can use

url = await page.evaluate("() => window.location.href")

in order to get the current URL. Note that page.evaluate() runs whatever Javascript your give it – hence you can use your Javascript skills in order to create the desired effect.

Full example

import asyncio
from pyppeteer import launch

async def main():
    browser = await launch()
    page = await browser.newPage()
    await page.goto('https://www.techoverflow.net')

    # Get the URL and print it
    url = await page.evaluate("() => window.location.href")
    print(url) # prints https://www.techoverflow.net/

    # Cleanuip
    await browser.close()

asyncio.get_event_loop().run_until_complete(main())

 

Posted by Uli Köhler in Pyppeteer, Python

How simulate click using pyppeteer

In order to click a button or a link using the the pyppeteer library, you can use page.evaluate().

If you have an <button> element or a link (<a>) like

<button id="mybutton">

you can use

# Now click the search button    
await page.evaluate(f"""() => {{
    document.getElementById('mybutton').dispatchEvent(new MouseEvent('click', {{
        bubbles: true,
        cancelable: true,
        view: window
    }}));
}}""")

in order to generate a MouseEvent that simulates a click. Note that page.evaluate() will run any Javascript code you pass to it, so you can use your Javascript skills in order to create the desired effect

Also see https://gomakethings.com/how-to-simulate-a-click-event-with-javascript/ for more details on how to simulate mouse clicks in pure Javascript without relying on jQuery.

Note that page.evaluate() will just run any Javascript code you give it, so you can put your Javascript skills to use in order to manipulate the page.

Full example

This example will open https://techoverflow.net, enter a search term into the search field, click the search button and then create a screenshot

import asyncio
from pyppeteer import launch

async def main():
    browser = await launch()
    page = await browser.newPage()
    await page.goto('https://techoverflow.net')

    # Fill content into the search field
    content = "pypetteer"
    await page.evaluate(f"""() => {{
        document.getElementById('s').value = '{content}';
    }}""")

    # Now click the search button    
    await page.evaluate(f"""() => {{
        document.getElementById('searchsubmit').dispatchEvent(new MouseEvent('click', {{
            bubbles: true,
            cancelable: true,
            view: window
        }}));
    }}""")

    # Wait until search results page has been loaded
    await page.waitForSelector(".archive-title")

    # Now take screenshot and exit
    await page.screenshot({'path': 'screenshot.png'})
    await browser.close()

asyncio.get_event_loop().run_until_complete(main())

The result will look like this:

Posted by Uli Köhler in Pyppeteer, Python

How to fill <input> field using pyppeteer

In order to fill an input field using the pyppeteer library, you can use page.evaluate().

If you have an <input> element like

<input name="myinput" id="myinput" type="text">

you can use

content = "My content" # This will be filled into <input id="myinput"> !
await page.evaluate(f"""() => {{
    document.getElementById('myinput').value = '{content}';
}}""")

Note that page.evaluate() will just run any Javascript code you give it, so you can put your Javascript skills to use in order to manipulate the page.

Full example

This example will open https://techoverflow.net, enter a search term into the search field and then create a screenshot

#!/usr/bin/env python3
import asyncio
from pyppeteer import launch

async def main():
    browser = await launch()
    page = await browser.newPage()
    await page.goto('https://techoverflow.net')
    
    # This example fills content into the search field
    content = "My search term"
    await page.evaluate(f"""() => {{
        document.getElementById('s').value = '{content}';
    }}""")

    # Make screenshot
    await page.screenshot({'path': 'screenshot.png'})
    await browser.close()

asyncio.get_event_loop().run_until_complete(main())

The result will look like this:

Posted by Uli Köhler in Pyppeteer, Python