How simulate click using pyppeteer

In order to click a button or a link using the the pyppeteer library, you can use page.evaluate().

If you have an <button> element or a link (<a>) like

<button id="mybutton">

you can use

# Now click the search button    
await page.evaluate(f"""() => {{
    document.getElementById('mybutton').dispatchEvent(new MouseEvent('click', {{
        bubbles: true,
        cancelable: true,
        view: window
    }}));
}}""")

in order to generate a MouseEvent that simulates a click. Note that page.evaluate() will run any Javascript code you pass to it, so you can use your Javascript skills in order to create the desired effect

Also see https://gomakethings.com/how-to-simulate-a-click-event-with-javascript/ for more details on how to simulate mouse clicks in pure Javascript without relying on jQuery.

Note that page.evaluate() will just run any Javascript code you give it, so you can put your Javascript skills to use in order to manipulate the page.

Full example

This example will open https://techoverflow.net, enter a search term into the search field, click the search button and then create a screenshot

import asyncio
from pyppeteer import launch

async def main():
    browser = await launch()
    page = await browser.newPage()
    await page.goto('https://techoverflow.net')

    # Fill content into the search field
    content = "pypetteer"
    await page.evaluate(f"""() => {{
        document.getElementById('s').value = '{content}';
    }}""")

    # Now click the search button    
    await page.evaluate(f"""() => {{
        document.getElementById('searchsubmit').dispatchEvent(new MouseEvent('click', {{
            bubbles: true,
            cancelable: true,
            view: window
        }}));
    }}""")

    # Wait until search results page has been loaded
    await page.waitForSelector(".archive-title")

    # Now take screenshot and exit
    await page.screenshot({'path': 'screenshot.png'})
    await browser.close()

asyncio.get_event_loop().run_until_complete(main())

The result will look like this:

Posted by Uli Köhler in Pyppeteer, Python

How to fill <input> field using pyppeteer

In order to fill an input field using the pyppeteer library, you can use page.evaluate().

If you have an <input> element like

<input name="myinput" id="myinput" type="text">

you can use

content = "My content" # This will be filled into <input id="myinput"> !
await page.evaluate(f"""() => {{
    document.getElementById('myinput').value = '{content}';
}}""")

Note that page.evaluate() will just run any Javascript code you give it, so you can put your Javascript skills to use in order to manipulate the page.

Full example

This example will open https://techoverflow.net, enter a search term into the search field and then create a screenshot

#!/usr/bin/env python3
import asyncio
from pyppeteer import launch

async def main():
    browser = await launch()
    page = await browser.newPage()
    await page.goto('https://techoverflow.net')
    
    # This example fills content into the search field
    content = "My search term"
    await page.evaluate(f"""() => {{
        document.getElementById('s').value = '{content}';
    }}""")

    # Make screenshot
    await page.screenshot({'path': 'screenshot.png'})
    await browser.close()

asyncio.get_event_loop().run_until_complete(main())

The result will look like this:

Posted by Uli Köhler in Pyppeteer, Python

pypetteer is a spelling mistake. It’s called pyppeteer !

If you found this page when looking for the pyppeteer library, you must have spelled it wrong.

The correct spelling is pyppeteer: Two ps and only one t!

Posted by Uli Köhler in Pyppeteer, Python

Traefik docker-compose configuration with secure dashboard and Let’s Encrypt

This configuration only provides only the minimum to get the Traefik Dashboard running with Let’s Encrypt-driven SSL encryption and user authentication. It also redirects all HTTP requests to HTTPS in order to avoid insecure access to the Dashboard and other services.

Let’s encrypt is used with the HTTP-01 challenge. This means that Traefik MUST be reachable by Port 80 from the Internet.

In order to install docker & docker-compose, see How to install docker and docker-compose on Ubuntu in 30 seconds.

First prepare the directory (/var/lib/traefik):

sudo mkdir /var/lib/traefik
sudo chown -R $USER: /var/lib/traefik
cd /var/lib/traefik
mkdir acme conf

Now create docker-compose.yml:

version: "3.3"

services:
  traefik:
    image: "traefik:v2.3"
    container_name: "traefik"
    command:
      - "--api=true"
    ports:
      - "80:80"
      - "443:443"
      - "8080:8080"
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
      - "./acme:/etc/traefik/acme"
      - "./traefik.toml:/etc/traefik/traefik.toml"
      - "./conf:/etc/traefik/conf"

Now create the main traefik.toml configuration file:

defaultEntryPoints = ["http", "https"]

[api]
dashboard = true

# You can create config files in /var/lib/traefik/traefik.conf and Traefik will automatically reload them
[providers]
[providers.file]
directory = "/etc/traefik/conf/"
watch = true

# Change this to INFO if you don't want as much debug output
[log]
level = "DEBUG"

[entryPoints.web]
address = ":80"
[entryPoints.web.http]
[entryPoints.web.http.redirections]
[entryPoints.web.http.redirections.entryPoint]
to = "websecure"
scheme = "https"
[entryPoints.websecure]
address = ":443"

[certificatesResolvers.letsencrypttls.acme]
# TODO Add your email here
email = "[email protected]"
storage = "/etc/traefik/acme/acme.json"
[certificatesResolvers.letsencrypttls.acme.httpChallenge]
entryPoint = "web"

Now we need to create the API config file in conf/api.toml:

[http.routers.traefik-api]
# TODO: Set your domain here !!!
rule = "Host(`traefik.example.com`)"
service = "api@internal"
middlewares = ["auth"]
[http.routers.traefik-api.tls]
certresolver = "letsencrypttls"
[http.middlewares.auth.basicAuth]
# TODO Add your admin user & password here, generate e.g. using https://wtools.io/generate-htpasswd-online
users = [
  "admin:$1$ySFBr~_y$GsKgEasDQkpCX8sO8vNia0",
]

Don’t forget to change your email address and the domain name in the config files (marked by TODO). Ensure you have setup all DNS records correctly so that your domains points to the server running Traefik!

Now it’s time to startup Traefik for the first time:

docker-compose up

Traefik will take a few seconds to automatically generate the Let’s Encrypt certificate for your domain. Once you see a message like

traefik    | time="2020-09-20T23:48:30Z" level=debug msg="Certificates obtained for domains [traefik.mydomain.com]" providerName=letsencrypttls.acme routerName=traefik-api@file rule="Host(`traefik.mydomain.com`)"

the certificate is available and loaded automatically.

Now you can go to https://traefik.mydomain.com/ , login with the username and password you have generated and check out the dashboard.

 

If desired, you can also setup a systemd service to automatically start Traefik on boot (generated using docker-compose systemd .service generator). In order to do this, first stop the running docker-compose instance using Ctrl-C if you still have the terminal open and docker-compose down.

Now add this as /etc/systemd/system/traefik.service:

[Unit]
Description=traefik
Requires=docker.service
After=docker.service

[Service]
Restart=always
User=root
Group=docker
WorkingDirectory=/var/lib/traefik
# Shutdown container (if running) when unit is stopped
ExecStartPre=/usr/local/bin/docker-compose -f docker-compose.yml down -v
# Start container when unit is started
ExecStart=/usr/local/bin/docker-compose -f docker-compose.yml up
# Stop container when unit is stopped
ExecStop=/usr/local/bin/docker-compose -f docker-compose.yml down -v

[Install]
WantedBy=multi-user.target

and run

sudo systemctl enable traefik.service
sudo systemctl start traefik.service

 

Posted by Uli Köhler in Traefik

How to fix Traefik Dashboard/API 404 page not found “api is not enabled”

Problem:

You are trying to configure the Traefik API/Dashboard in a secure way, but every time you try to access the API, you’re only getting a 404 error (unless you set api.insecure=true)

In the logs, you see an error message like this: (if log.level = "DEBUG"):

traefik    | time="2020-09-20T22:53:51Z" level=error msg="api is not enabled" routerName=my-api@file entryPointName=websecure

Solution:

You have to pass --api=true to Traefik, e.g. using docker-compose:

command:
  - "--api=true"

and also set

[api]
dashboard = true

in your traefik.toml.

After that, restart Traefik and you should be able to access your dashboard at /dashboard.

Credits to multiple GitHub users for the original solution.

Posted by Uli Köhler in Traefik

How to fix Traefik “command traefik error: field not found, node: dnsProvider”

Problem:

You are trying to configure your Traefik server but you see an error message like

traefik    | 2020/09/20 22:07:11 command traefik error: field not found, node: dnsProvider

Solution:

dnsProvider is a configuration option from Traefik 1.x. You need to use provider for Traefik 2.x. Example:

[certificatesResolvers.lecloudflare.acme.dnsChallenge]
provider = "cloudflare"

Full example:

[certificatesResolvers.lecloudflare.acme]
email = "[email protected]"
storage = "/etc/traefik/acme/acme.json"

[certificatesResolvers.lecloudflare.acme.dnsChallenge]
provider = "cloudflare"



 

 

Posted by Uli Köhler in Traefik

How to run TamperMonkey function after window.location.href change

When automating workflows using TamperMonkey scripts, you often have a situation where you want to run some function, then set window.location.href and after that page has loaded, run another part of your code. However, since the browser physically reloads the page, the Tampermonkey script is also reloaded – hence, your original function will stop being executed.

Fixing this is easy once you grasp the concept:

  1. Before reloading the page, we set a specific key in sessionStorage, indicating what we want to do once the page has finished loading
  2. Then we can set window.location.href
  3. On each page load, we check if the key is set and, if so, we run the appropriate function and delete the key

Note that in the TamperMonkey context, this is limited to pages that are included in the @match configuration of the script (otherwise, the script won’t be executed on the new page that is being loaded):

// ==UserScript==
// @name         TamperMonkey continuation example
// @namespace    http://tampermonkey.net/
// @version      0.1
// @author       You
// @match        https://techoverflow.net
// @grant        none
// ==/UserScript==

(function() {
    'use strict';
    const continuationKey = "_my_script_continuation_key"; // This sessionstorage key

    // Check continuation
    const continuationActions = {
        "printMsg": onPrintMsg
    };
    const _continuation = sessionStorage.getItem(continuationKey);
    if(_continuation) {
        sessionStorage.removeItem(continuationKey);
        const action = continuationActions[_continuation]
        if(action) {action();}
    }

    function onPrintMsg() {
        console.log("This is run after reloading the page");
    }

    function onAltQ() {
        console.log("Will now reload the page...");
        sessionStorage.setItem(continuationKey, "printMsg");
        window.location.href = window.location.origin + "/";
    }

    function onKeydown(evt) {
        // Use https://keycode.info/ to get keys
        if (evt.altKey && evt.keyCode == 81) {
            onAltQ();
        }
    }
    document.addEventListener('keydown', onKeydown, true);
})();

 

Posted by Uli Köhler in Javascript

How to go to URL within same domain in Javascript

In order to go to an URL relative to the current domain name in Javascript, use window.location.origin like this:

window.location.href = window.location.origin + "/myurl"
Posted by Uli Köhler in Javascript

Minimal TamperMonkey template for keypress-triggered events

This template allows you to build your own TamperMonkey scripts that react on keypresses on a specific page.

This script shows an alert message when you press Alt+Q.

// ==UserScript==
// @name         TamperMonkey keypress example
// @namespace    http://tampermonkey.net/
// @version      0.1
// @description  try to take over the world!
// @author       You
// @match        https://techoverflow.net/*
// @grant        none
// ==/UserScript==

(function() {
    'use strict';

    function onAltQ() {
        alert("Alt Q pressed!");
    }

    function onKeydown(evt) {
        // Use https://keycode.info/ to get keys
        if (evt.altKey && evt.keyCode == 81) {
            onAltQ();
        }
    }
    document.addEventListener('keydown', onKeydown, true);
})();

 

Posted by Uli Köhler in Javascript

How to fix OpenVPN “TLS Error: cannot locate HMAC in incoming packet from …”

Problem:

Your OpenVPN clients can’t connect to your OpenVPN server and the server log shows an error message like

TLS Error: cannot locate HMAC in incoming packet from [AF_INET6]::ffff:187.100.14.13:41874 (via ::ffff:25.16.25.29%xn0)

Solution:

You have enabled a TLS key (tls-auth option) in your OpenVPN configuration, but your client does not know that it should use the additional layer of authentication.

The server is looking for the HMAC in the incoming packets but can’t find it.

Either disable the tls-auth option in your server config. The config line will look like

tls-auth /var/etc/openvpn/server2.tls-auth 0

or

Enable the correct tls-auth configuration in your client. Remember that you also need to share the correct key.

Posted by Uli Köhler in Networking, OpenVPN, VPN

How to get duration of WAV file in Python (minimal example)

Use

duration_seconds = mywav.getnframes() / mywav.getframerate()

to get the duration of a WAV file in seconds.

Full example:

import wave

with wave.open("myaudio.wav") as mywav:
    duration_seconds = mywav.getnframes() / mywav.getframerate()
    print(f"Length of the WAV file: {duration_seconds:.1f} s")

 

Posted by Uli Köhler in Audio, Python

Find total filesize of all WAV files in a directory using command line tools

Use this command to find the total file size of all .wav files (including .WAV files)

find . -name "*.wav" -or -name "*.WAV" -exec du -ch {} + | tail -n 1

Example output:

6,7G    total

 

Posted by Uli Köhler in Linux

How to delete all .DS_Store files recursively on the command line

Use this command to recursively delete all .DS_Store files present in the directory mydir and all its subdirectories:

find mydir -type f -name .DS_Store -print0 | xargs -0 rm

Warning: Deleting the files can’t be undone.

Posted by Uli Köhler in Linux

How to fix SSH not accepting public key in authorized_keys

Problem:

You have added your SSH public key to a remote server manually or using ssh-copy-id but still you can’t login using that public key

Solution:

Typically this is caused by bad permissions of your ~/.ssh directory and/or your authorized_keys file. Fix that using:

chmod -R 700 ~/.ssh

then try to login again

Posted by Uli Köhler in Linux

How to run systemd timer twice daily

In order to run a systemd timer twice daily at fixed times, use this syntax in the .timer file:

OnCalendar=*-*-* 00,12:00:00

This line means: Run the service on each and every day (*-*-*) at 00:00:00 and 12:00:00 (00,12:00:00)

Posted by Uli Köhler in Linux

Find and remove all empty directories using the Linux command line

In order to find and remove all empty subdirectories in the current directory (.) recursively, use this command:

find . -depth -type d -print0 | xargs -0 rmdir

This command will only remove empty directories! Any file or non-empty directory won’t be modified.

Explanation of what the parts mean:

  • find: Use the find command to find directories recursively
  • .: Start recursing from the current directory. In case you want to start from a different directory, use that directory name here.
  • -type d: Only find directories – ignore files
  • -depth: Before printing a directory name, print all its sub-directory names. This avoids having to run this command repeatedly because the parent directory can’t be removed since its empty sub-directories need to be removed first
  • -print0 When printing all the directories that have been found, print a NUL character between directories. This is required in order to handle spaces in the directory names correctly
  • | xargs: Pipe the directories to xargs, a program that runs
  • -0: Split the input by NUL characters instead of newlines. This corresponds with the -print0 option to find and is required to handle spaces in directory names correctly.
  • rmdir: For each directory found, run rmdir i.e. try to remove the directory if it’s empty.
Posted by Uli Köhler in Linux

How to fix “OTGS Installer, responsible for receiving automated updates for WPML and Toolset, requires the following PHP component(s) in order to function:cURL” on Ubuntu 20.04

 Problem:

Your WordPress installation running on Ubuntu 20.04 shows you this error message:

OTGS Installer, responsible for receiving automated updates for WPML and Toolset, requires the following PHP component(s) in order to function:cURL

Learn more: Minimum WPML requirements

Solution:

You need to install the curl package for your PHP version.

Typically you can use

sudo apt -y install php7.4-curl

Doing this should immediately fix the issue (just reload your WordPress dashboard). In case you’re using PHP 7.2 you need to sudo apt -y install php7.2-curl instead.

In case the issue is still not fixed after doing this, try restarting your webserver (e.g. sudo service apache restart or sudo service nginx restart), try restarting PHP-FPM by using sudo service php7.4-fpm restart or sudo service php7.2-fpm restart or try rebooting your server. In case that doesn’t help either, you need to manually check which PHP instance the wordpress site is using and install the curl package for that PHP instance.

 

Posted by Uli Köhler in PHP, Wordpress

How to emulate Page up / Page down key in Puppeteer

To emulate a keypress to the Page up key in Puppeteer, use

await page.keyboard.press("PageUp");

To emulate a keypress to the Page down key in Puppeteer, use

await page.keyboard.press("PageDown");

 

Posted by Uli Köhler in Javascript, NodeJS, Puppeteer

How to run Nextcloud “php occ” in a docker-compose configuration

Use this command to run php occ inside a running docker Nextcloud container:

docker-compose exec -u www-data nextcloud php occ [command]

where:

  • nextcloud is the name of your container in docker-compose
  • -u www-data tells docker-compose to run the command as the www-data user that owns all the Nextcloud files.
  • Replace [command] by the php occ command you want to run, for example versions:cleanup. If you want to see a list of all available commands, just omit [command]

Note: You need to run this command in the directory where your nextcloud docker-compose.yml is located.

Example:

docker-compose exec -u www-data nextcloud php occ versions:cleanup

 
or:

docker-compose exec -u www-data nextcloud php occ trash:cleanup --all-users

 

Posted by Uli Köhler in PHP

mbed STM32 timer interrupt example

You can use the mbed Ticker API to add a timer interrupt to your mbed application. This example will use a Ticker-based timer interrupt to toggle the LED once per second, for example on the STM32F429I-DISCO board:

#include <mbed.h>

DigitalOut led1(LED1);
Ticker ticker;

/**
 * This function will be run once per second
 */
void timerTick() {
  // Toggle LED
  led1 = !led1;
}

int main() {
  ticker.attach(timerTick, 1.0 /* seconds */);

  // What you do in the main loop is not important
  while(1) {
  }
}

 

Posted by Uli Köhler in C/C++, mbed
This website uses cookies to improve your experience. We'll assume you're ok with this, but you can opt-out if you wish. Cookie settingsACCEPTPrivacy &amp; Cookies Policy