How to remove all unused docker images

Use

docker image prune -f

to remove all unused/old (“dangling”) docker images. -f will skip the confirmation prompt.

This command will remove all images that a) don’t have a tag themselves and b) no image with a tag depends on the image.

Example output:

Deleted Images:
deleted: sha256:b6e0e69c6ba1e811063921d18b52627a9fb905e0bdfd80e226e86958831df636
[...]
deleted: sha256:6a40d04ddf00ad0f1806df7b2d4a2d44a6d8031cab5c369a4bf3d1694d5c48b4

Total reclaimed space: 985.3MB

 

Posted by Uli Köhler in Container, Docker

Lumen to Candela online calculator & Python code

lm

°



Formula

\Omega_{sr} = 2\cdot\pi\cdot(1-\cos(\frac{\theta}{2}))
I_{v} = \frac{\Phi_v}{\Omega_{sr}}

where:

  • \theta is the apex angle in radians
  • \Omega_{sr} is the solid angle in Steradians
  • \Phi_v is the luminous flux in lux (lx).
  • I_{v} is the luminous intensity in candela (cd).

Python code

You can use the UliEngineering library like this:

from UliEngineering.Physics.Light import lumen_to_candela_by_apex_angle
from UliEngineering.EngineerIO import auto_format, auto_print

# These are equivalent:
intensity = lumen_to_candela_by_apex_angle("25 lm", "120°") # intensity = 7.9577 (cd)
intensity = lumen_to_candela_by_apex_angle(25.0, 120.0) # intensity = 7.9577 (cd)

# ... or get out a human-readable value:
intensity_str = auto_format(lumen_to_candela_by_apex_angle, "25 lm", "120°") # "7.96 cd"
# ... or print directly
auto_print(lumen_to_candela_by_apex_angle, "25 lm", "120°") # prints "7.96 cd"

In case you can’t use UliEngineering, use this Python function:

import math

def lumen_to_candela_by_apex_angle(flux, angle):
    """
    Compute the luminous intensity from the luminous flux,
    assuming that the flux of <flux> is distributed equally around
    a cone with apex angle <angle>.

    Keyword parameters
    ------------------
    flux : value, engineer string or NumPy array
        The luminous flux in Lux.
    angle : value, engineer string or NumPy array
        The apex angle of the emission cone, in degrees
        For many LEDs, this is 

    >>> lumen_to_candela_by_apex_angle(25., 120.)
    7.957747154594769
    """
    solid_angle = 2*math.pi*(1.-math.cos((angle*math.pi/180.)/2.0))
    return flux / solid_angle

# Usage example
print(lumen_to_candela_by_apex_angle(25., 120.)) # Prints 7.957747154594769 (cd)

 

Posted by Uli Köhler in Calculators, Physics

How to set Access-Control-Allow-Origin * in nginx

To allow access from all origins, use

add_header 'Access-Control-Allow-Origin' '*' always;

You might want to add

add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS';

as well.

Posted by Uli Köhler in nginx

How to emulate the Enter key in Puppeteer

To emulate a keypress to the Enter key in Puppeteer, use

await page.keyboard.press("Enter");

The E needs to be uppercase for this to work!

Posted by Uli Köhler in Javascript, Puppeteer

How to emulate keyboard input in Puppeteer

To emulate the user typing something on the keyboard, use

await page.keyboard.type("the text");

This will type the text extremely fast with virtually no delay between the characters.

In order to simulate the finite typing speed of real users, use

await page.keyboard.type("the text", {delay: 100});

instead. The delay betwen characters in this example is 100 Milliseconds, i.e. the emulated user types 10 characters per second.

Posted by Uli Köhler in Javascript, NodeJS, Puppeteer

How to emulate TAB key press in Puppeteer

In order to emulate a tab key press in Puppeteer, use

await page.keyboard.press("Tab");

Full example:

// Minimal puppeteer example
const puppeteer = require('puppeteer');
(async () => {
  const browser = await puppeteer.launch({defaultViewport: {width: 1920, height: 1080}});
  const page = await browser.newPage();
  await page.goto('https://techoverflow.net', {waitUntil: 'domcontentloaded'});
  // Press tab 10 times (effectively scrolls down on techoverflow.net)
  for (let i = 0; i < 10; i++) {
      await page.keyboard.press("Tab");
  }
  // Screenshot to verify result
  await page.screenshot({path: 'screenshot.png'});
  // Cleanup
  await browser.close();
})();

 

Posted by Uli Köhler in Javascript, Puppeteer

How to fix Puppeteer ‘Error: Unknown key: “tab”‘

Problem:

You want to emulate a tab key press using

await page.keyboard.press("tab");

but you get an error message like

(node:30594) UnhandledPromiseRejectionWarning: Error: Unknown key: "tab"
    at assert (/home/uli/dev/myproject/node_modules/puppeteer/lib/helper.js:270:11)
    at Keyboard._keyDescriptionForString (/home/uli/dev/myproject/node_modules/puppeteer/lib/Input.js:96:5)
    at Keyboard.down (/home/uli/dev/myproject/node_modules/puppeteer/lib/Input.js:44:30)
    at Keyboard.<anonymous> (/home/uli/dev/myproject/node_modules/puppeteer/lib/helper.js:112:23)
    at Keyboard.press (/home/uli/dev/myproject/node_modules/puppeteer/lib/Input.js:178:16)
    at Keyboard.<anonymous> (/home/uli/dev/myproject/node_modules/puppeteer/lib/helper.js:112:23)
    at /home/uli/dev/myproject/test.js:8:23
    at processTicksAndRejections (internal/process/task_queues.js:85:5)
(node:30594) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 2)
(node:30594) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

Solution:

You need to use Tab, not tab ! The T needs to be uppercase!

Use

await page.keyboard.press("Tab");

Full example:

// Minimal puppeteer example
const puppeteer = require('puppeteer');
(async () => {
  const browser = await puppeteer.launch({defaultViewport: {width: 1920, height: 1080}});
  const page = await browser.newPage();
  await page.goto('https://techoverflow.net', {waitUntil: 'domcontentloaded'});
  // Press tab 10 times (effectively scrolls down on techoverflow.net)
  for (let i = 0; i < 10; i++) {
      await page.keyboard.press("Tab");
  }
  // Screenshot to verify result
  await page.screenshot({path: 'screenshot.png'});
  // Cleanup
  await browser.close();
})();

 

Posted by Uli Köhler in Javascript

How to set <input> value in Puppeteer

Use this snippet to set the value of an HTML <input> element in Puppeteer:

const newInputValue = "test 123";
await page.evaluate(val => document.querySelector('.search-form-input').value = val, newInputValue);amp

Remember to replace '.search-form-input' by whatever CSS selector is suitable to select your <input>. Examples include 'input[name="username"]' or '.username > input'.

Full example:

// Minimal puppeteer example
const puppeteer = require('puppeteer');
(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();
  await page.goto('https://techoverflow.net', {waitUntil: 'domcontentloaded'});
  // Set input value
  const newInputValue = "test 123";
  await page.evaluate(val => document.querySelector('.search-form-input').value = val, newInputValue);
  // Screenshot to verify result
  await page.screenshot({path: 'screenshot.png'});
  // Cleanup
  await browser.close();
})();

Note that this method will work for any simple <input>, however it might not work for some heavily Javascripted inputs which you can find on some modern websites.

Posted by Uli Köhler in Javascript, Puppeteer

How to save screenshot in Puppeteer as PNG?

You can take a screenshot in Puppeteer using

await page.screenshot({path: 'screenshot.png'});

The path is relative to the current working directory.

Want to have a screenshot in a size different to 800×600? See How to set screenshot size in Puppeteer?

Full example:

// Minimal puppeteer example
const puppeteer = require('puppeteer');
(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();
  await page.goto('https://techoverflow.net', {waitUntil: 'domcontentloaded'});
  // Screenshot to verify result
  await page.screenshot({path: 'screenshot.png'});
  // Cleanup
  await browser.close();
})();

 

Posted by Uli Köhler in Javascript, NodeJS, Puppeteer

How to identify large directories for ‘No space left on device’ on Linux

TL;DR

cd / and run

sudo du -sh * --exclude proc --exclude sys --exclude dev

and then repeat for the largest directory shown (by cding to that directory and running the command above)

Long answer

If you get No space left on device errors on Linux, this means that one of your mounted disks has (virtually) no space left to write on.

First, check which device is the one that has no space left:

$ sudo df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            2.4G     0  2.4G   0% /dev
tmpfs           494M   51M  444M  11% /run
/dev/xvda1       46G   17G   11M 100% /
tmpfs           2.5G     0  2.5G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           2.5G     0  2.5G   0% /sys/fs/cgroup
tmpfs           494M     0  494M   0% /run/user/1003

Check the Use % column – you can see that the device mounted on / (your root filesystem, i.e. the one where your system is mounted) is full. In 95% of all cases it’s the root filesystem that’s full.

This post only covers the case where your root filesystem is full. In most other cases, it’s either /dev/shm (in which case rebooting your system typically works) or an external drive (in which case you have to figure out for yourself which directories can be deleted).

Probabilistic method:

This quick check tries to identify the most likely candidates first (based on admin experience). Run this in your shell:

sudo du -sh /var/lib/docker /var/lib/mysql /var/lib/postgresql /home/

This might take some time to complete.

Check if one of those directories is so large that it eats up a significant fraction of your drive space. If not, I recommend going forward with the Simple method:

Simple method:

Copy & paste this into your shell:

function findLargestSubdir { cd "$1" && sudo du -sb * --exclude proc --exclude sys --exclude dev | sort -n ; }

then run

findLargestSubdir /

This will tell you which directory is the largest. It might take a long time to compute the size of all directoriesThis command will only print anything once it’s finished! The numbers are in bytes.

The last element in the list is the largest one!

Now, run the same command inside the largest directory to find the largest sub-directory. For example, if /var happens to be the largest directory, run

findLargestSubdir /var

which will show you the largest directory in /var. Continue checking the largest subdir(s) using findLargestSubdir until you found out what ate up all the space on your disk.

Advanced method: Much quicker but more complex

If it takes more than 2 minutes to compute the size of all directories, I recommend following this interactive procedure as root (sudo):

  1. ls -1 to show all files and directories in /. Example output:
    # ls -1 /
    bin
    boot
    dev
    etc
    home
    initrd.img
    initrd.img.old
    lib
    lib64
    lost+found
    media
    mnt
    opt
    proc
    root
    run
    sbin
    snap
    srv
    sys
    tmp
    usr
    var
    vmlinuz
    vmlinuz.old
  2. Now run du -sh *. This will try to compute the size of each of those files and directories. Typically this command will stall for a long time when trying to compute the size of one directory (if that directory has a huge number of files in it). For example, when you see the output
    # du -sh *
    17M     bin
    157M    boot
    0       dev
    7.7M    etc

    and then nothing happens for more than 30 seconds, look up the next entry after the last entry in the list ( etc in this example) in the output of ls -1 above. In this example, this would be /home. Since du -sh took so long computing the size of /home it’s very likely (though not guaranteed) that /home is the directory that takes up so much space. Also check the number

  3. In whatever directory you found to be a candidate for being the largest directory, run ls -1 <directory> again and check

Note that this method sometimes has a tendency to identify directories that recursively contain many files as opposed to directories whose total size is large. Therefore, you might need to go back in case you can’t identify any directories that eat up a large fraction of your hard drive space.

Posted by Uli Köhler in Linux

How to make your own ‘get my current IP address’ server using only nginx

This nginx config will return the user’s IP address to the user. Note that it won’t work behind a reverse proxy, it actually needs to listen on port 80 and/or 443 of the IP address of the domain it is served as.

location = /api/get-my-ip {
    add_header 'Access-Control-Allow-Origin' '*' always;
    add_header 'Access-Control-Allow-Methods' 'GET, OPTIONS';
    add_header 'Content-Type' 'text/plain charset=UTF-8';
    return 200 '$remote_addr';
}

 

This is a JSON-returning variant:

location = /api/get-my-ip-json {
    add_header 'Access-Control-Allow-Origin' '*' always;
    add_header 'Access-Control-Allow-Methods' 'GET, OPTIONS';
    add_header 'Content-Type' 'text/plain charset=UTF-8';
    return 200 '{"ip": "$remote_addr"}';
}

This configuration is available online, for free, at techoverflow.net: See How to get your current IPv4 address using wget and How to get your current IPv6 address using wget

Posted by Uli Köhler in Networking, nginx

How to sleep in NodeJS using async/await

The easiest way to sleep using async/await in NodeJS is the sleep-promise package:

npm install --save sleep-promise

Using that package you can simply use await sleep(milliseconds) syntax like this:

const sleep = require('sleep-promise');

// In any async function:
await sleep(2000); // Wait 2000 ms

Note that this sleep() variant is fully asynchronous, IO and other asynchronous operations will still be able to continue in the background – sleep() will not block the NodeJS process.

Full example:

const sleep = require('sleep-promise');

(async () => {
    console.log("This prints immediately");
    await sleep(2000);
    console.log("This prints 2 seconds later");
})();

 

Posted by Uli Köhler in Javascript, NodeJS

How to fix nginx rewrite redirect causing certbot verification errors

When using a redirection to a new domain using rewrite in my nginx config, I had issues with Let’s Encrypt not being able to verify the domain ownership when using certbot --nginx.

My redirect statement was

location / {
    rewrite ^/$ https://newdomain.com permanent;
    rewrite ^/(.*)$ https://newdomain.com/$1 permanent;
}

I was able to fix the issue by instead using this redirect statement:

location / {
     return 301 https://newdomain.com$request_uri;
}

After using this, I was able to use certbot --nginx without any issues.

Posted by Uli Köhler in nginx

How to redirect to a new domain using nginx

In order to redirect to a different domain using nginx, use this snippet inside your server {...} block:

location / {
     return 301 https://newdomain.com$request_uri;
}

Full example:

server {
    server_name old.domain;
    listen 80;

    location / {
         return 301 https://techoverflow.net$request_uri;
    }
}
Posted by Uli Köhler in nginx

How to get your current IPv6 address using wget

Also see How to get your current IPv4 address using wget

Use

wget -qO- https://ipv6.techoverflow.net/api/get-my-ip

which uses the TechOverflow get-my-ip API.

Example:

$ wget -qO- https://ipv6.techoverflow.net/api/get-my-ip
2003:ea:70a:d600:ee98:42ca:b0bb:ac75

If your computer does not have IPv6 connectivity enabled, you will not see any output.

Prefer a JSON response?

Use https://ipv6.techoverflow.net/api/get-my-ip-json instead:

wget -qO- https://ipv6.techoverflow.net/api/get-my-ip-json

Example:

$ wget -qO- https://ipv6.techoverflow.net/api/get-my-ip-json
{"ip": "2003:ea:70a:d600:ee98:42ca:b0bb:ac75"}

Want this API on your own server? Check out How to make your own ‘get my current IP address’ server using only nginx

Posted by Uli Köhler in APIs, Networking

How to get your current IPv4 address using wget

Also see How to get your current IPv6 address using wget

Use

wget -qO- https://ipv4.techoverflow.net/api/get-my-ip

which uses the TechOverflow get-my-ip API.

Example:

$ wget -qO- https://ipv4.techoverflow.net/api/get-my-ip
80.138.203.18

Prefer a JSON response?

Use https://ipv4.techoverflow.net/api/get-my-ip-json instead:

wget -qO- https://ipv4.techoverflow.net/api/get-my-ip-json

Example:

$ wget -qO- https://ipv4.techoverflow.net/api/get-my-ip-json
{"ip": "80.138.201.17"}

Want this API on your own server? Check out How to make your own ‘get my current IP address’ server using only nginx

Posted by Uli Köhler in APIs, Networking

NodeJS equivalent to Python’s time.sleep() using async/await

Install the sleep-promise package:

npm install --save sleep-promise

then you can use

const sleep = require('sleep-promise');

// In any async function:
await sleep(2000); // Wait 2000 ms

Full example:

const sleep = require('sleep-promise');

(async () => {
    console.log("This prints immediately");
    await sleep(2000);
    console.log("This prints 2 seconds later");
})();

 

Posted by Uli Köhler in Javascript, NodeJS

Minimal puppeteer response interception example

Using Python (pyppeteer)? Check out Pyppetteer minimal network response interception example

This example shows you how to intercept network responses in puppeteer.

Note: This intercepts the response, not the request! This means you can abort the request before it is actually sent to the server, but you can’t read the content of the response! See Minimal puppeteer request interception example for an example on how to intercept requests.

const puppeteer = require('puppeteer');
(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();
  // Enable response interception
  page.on('response', async (response) => {
      console.info("URL", response.request().url());
      console.info("Method", response.request().method())
      console.info("Response headers", response.headers())
      console.info("Request headers", response.request().headers())
      // Use this to get the content as text
      const responseText = await response.text();
      // ... or as buffer (for binary data)
      const responseBuffer = await response.buffer();
      // ... or as JSON, if it's a JSON (else, this will throw!)
      const responseObj = await response.json();
  })
  await page.goto('https://techoverflow.net', {waitUntil: 'domcontentloaded'});
  // Make a screenshot
  await page.screenshot({path: 'screenshot.png'});
  await browser.close();
})();

 

Posted by Uli Köhler in Javascript, Puppeteer

Minimal puppeteer request interception example

Using Python (pyppeteer)? Check out Pyppetteer minimal network request interception example

This example shows you how to intercept network requests in puppeteer:

Note: This intercepts the request, not the response! This means you can abort the request made, but you can’t read the content of the response! See Minimal puppeteer response interception example for an example on how to intercept responses.

const puppeteer = require('puppeteer');
(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();
  // Enable request interception
  await page.setRequestInterception(true);
  page.on('request', async (request) => {
      console.info("URL", request.url());
      console.info("Method", request.method())
      console.info("Headers", request.headers())
      return request.continue(); // Allow request to continue
      // return request.abort(); // use this instead to abort the request!
  })
  await page.goto('https://techoverflow.net', {waitUntil: 'domcontentloaded'});
  // Make a screenshot
  await page.screenshot({path: 'screenshot.png'});
  await browser.close();
})();

 

Posted by Uli Köhler in Javascript, Puppeteer

Javascript equivalent to Python’s re.findall()

The equivalent to this Python code which is using re.findall()

import re

hashtag_regex = r"(\B#\w\w+)"
hits = re.findall(hashtag_regex, "This is a string #with #hashtags")
print(hits) # prints ['#with', '#hashtags']

in Javascript is

const string = "This is a string #with #hashtags";
const re = /(\B#\w\w+)/g;
const hits = [];
// Iterate hits
let match = null;
do {
    match = re.exec(string);
    if(match) {
        hits.push(match[0]);
    }
} while (match);

console.log(hits); // Prints [ '#with', '#hashtags' ]

You need to assign your regular expression to a variable like re! If you use

match = /(\B#\w\w+)/g.exec(string); // WRONG ! Don't do this !

you will create an infinite loop which always generates the first hit in the string, if any!

Posted by Uli Köhler in Javascript