Programming languages

How to list gitlab-rake tasks

You can list the tasks available in your GitLab’s gitlab-rake using

gitlab-rake -T

In case you’re running a docker-compose based configuration, you can use

docker-compose exec gitlab gitlab-rake -T

where gitlab is the name of the docker-compose service in docker-compose.yml.

This will show you a list like

rake about                                                      # List versions of all Rails frameworks and the environment
rake acts_as_taggable_on_engine:install:migrations              # Copy migrations from acts_as_taggable_on_engine to app...
rake acts_as_taggable_on_engine:tag_names:collate_bin           # Forcing collate of tag names to utf8_bin
rake acts_as_taggable_on_engine:tag_names:collate_ci            # Forcing collate of tag names to utf8_general_ci
rake app:template                                               # Applies the template supplied by LOCATION=(/path/to/te...
rake app:update                                                 # Update configs and some other initially generated file...
rake assets:clean[keep]                                         # Remove old compiled assets
rake assets:clobber                                             # Remove compiled assets
rake assets:environment                                         # Load asset compile environment
rake assets:precompile                                          # Compile all the assets named in config.assets.precompile
rake brakeman                                                   # Security check via brakeman
rake cache:clear:redis                                          # GitLab | Cache | Clear redis cache
rake cache_digests:dependencies                                 # Lookup first-level dependencies for TEMPLATE (like mes...
rake cache_digests:nested_dependencies                          # Lookup nested dependencies for TEMPLATE (like messages...
rake ci:cleanup:builds                                          # GitLab | CI | Clean running builds
rake clean                                                      # Remove any temporary products
rake clobber                                                    # Remove any generated files
rake config_lint                                                # Checks syntax for shell scripts and nginx config files...
rake danger_local                                               # Run local Danger rules
rake db:create                                                  # Creates the database from DATABASE_URL or config/datab...
rake db:drop                                                    # Drops the database from DATABASE_URL or config/databas...
rake db:environment:set                                         # Set the environment value for the database
rake db:fixtures:load                                           # Loads fixtures into the current environment's database
rake db:migrate                                                 # Migrate the database (options: VERSION=x, VERBOSE=fals...
rake db:migrate:status                                          # Display status of migrations
rake db:obsolete_ignored_columns                                # Show a list of obsolete `ignored_columns`
rake db:prepare                                                 # Runs setup if database does not exist, or runs migrati...
rake db:rollback                                                # Rolls the schema back to the previous version (specify...
rake db:schema:cache:clear                                      # Clears a db/schema_cache.yml file
rake db:schema:cache:dump                                       # Creates a db/schema_cache.yml file
rake db:schema:dump                                             # Creates a db/schema.rb file that is portable against a...
rake db:schema:load                                             # Loads a schema.rb file into the database
rake db:seed                                                    # Loads the seed data from db/seeds.rb
rake db:seed:replant                                            # Truncates tables of each database for current environm...
rake db:seed_fu                                                 # Loads seed data for the current environment
rake db:setup                                                   # Creates the database, loads the schema, and initialize...
rake db:structure:dump                                          # Dumps the database structure to db/structure.sql
rake db:structure:load                                          # Recreates the databases from the structure.sql file
rake db:version                                                 # Retrieves the current schema version number
rake dev:load                                                   # GitLab | Dev | Eager load application
rake dev:setup                                                  # GitLab | Dev | Setup developer environment (db, fixtures)
rake downtime_check                                             # Checks if migrations in a branch require downtime
rake file_hooks:validate                                        # Validate existing file hooks
rake gemojione:aliases                                          # Generates Emoji SHA256 digests
rake gemojione:install_assets                                   # Install Emoji Image Assets
rake gettext:add_language[language]                             # add a new language
rake gettext:find                                               # Update pot/po files
rake gettext:lint                                               # Lint all po files in `locale/
rake gettext:pack                                               # Create mo-files
rake gettext:po_to_json                                         # Convert PO files to JS files
rake gettext:regenerate                                         # Regenerate gitlab.pot file
rake gettext:store_model_attributes                             # write the model attributes to <locale_path>/model_attr...
rake gitlab:app:check                                           # GitLab | App | Check the configuration of the GitLab R...
rake gitlab:artifacts:check                                     # GitLab | Artifacts | Check integrity of uploaded job a...
rake gitlab:artifacts:migrate                                   # GitLab | Artifacts | Migrate files for artifacts to co...
rake gitlab:assets:check_page_bundle_mixins_css_for_sideeffects # GitLab | Assets | Check that scss mixins do not introd...
rake gitlab:assets:clean                                        # GitLab | Assets | Clean up old compiled frontend assets
rake gitlab:assets:compile                                      # GitLab | Assets | Compile all frontend assets
rake gitlab:assets:compile_webpack_if_needed                    # GitLab | Assets | Compile all Webpack assets
rake gitlab:assets:fix_urls                                     # GitLab | Assets | Fix all absolute url references in CSS
rake gitlab:assets:purge                                        # GitLab | Assets | Remove all compiled frontend assets
rake gitlab:assets:purge_modules                                # GitLab | Assets | Uninstall frontend dependencies
rake gitlab:assets:vendor                                       # GitLab | Assets | Compile vendor assets
rake gitlab:backup:create                                       # GitLab | Backup | Create a backup of the GitLab system
rake gitlab:backup:restore                                      # GitLab | Backup | Restore a previously created backup
rake gitlab:check                                               # GitLab | Check the configuration of GitLab and its env...
rake gitlab:cleanup:block_removed_ldap_users                    # GitLab | Cleanup | Block users that have been removed ...
rake gitlab:cleanup:moved                                       # GitLab | Cleanup | Delete moved repositories
rake gitlab:cleanup:orphan_job_artifact_files                   # GitLab | Cleanup | Clean orphan job artifact files
rake gitlab:cleanup:orphan_lfs_file_references                  # GitLab | Cleanup | Clean orphan LFS file references
rake gitlab:cleanup:orphan_lfs_files                            # GitLab | Cleanup | Clean orphan LFS files
rake gitlab:cleanup:project_uploads                             # GitLab | Cleanup | Clean orphaned project uploads
rake gitlab:cleanup:remote_upload_files                         # GitLab | Cleanup | Clean orphan remote upload files th...
rake gitlab:cleanup:sessions:active_sessions_lookup_keys        # GitLab | Cleanup | Sessions | Clean ActiveSession look...
rake gitlab:container_registry:configure                        # GitLab | Container Registry | Configure
rake gitlab:db:clean_structure_sql                              # This adjusts and cleans db/structure.sql - it runs aft...
rake gitlab:db:composite_primary_keys_add                       # GitLab | DB | Adds primary keys to tables that only ha...
rake gitlab:db:composite_primary_keys_drop                      # GitLab | DB | Removes previously added composite prima...
rake gitlab:db:configure                                        # GitLab | DB | Configures the database by running migra...
rake gitlab:db:create_dynamic_partitions                        # Create missing dynamic database partitions
rake gitlab:db:downtime_check[ref]                              # GitLab | DB | Checks if migrations require downtime or...
rake gitlab:db:drop_tables                                      # GitLab | DB | Drop all tables
rake gitlab:db:dump_custom_structure                            # This dumps GitLab specific database details - it runs ...
rake gitlab:db:load_custom_structure                            # This loads GitLab specific database details - runs aft...
rake gitlab:db:mark_migration_complete[version]                 # GitLab | DB | Manually insert schema migration version
rake gitlab:db:reindex[index_name]                              # reindex a regular (non-unique) index without downtime ...
rake gitlab:db:setup_ee                                         # GitLab | DB | Sets up EE specific database functionality
rake gitlab:db:unattended                                       # GitLab | DB | Run database migrations and print `unatt...
rake gitlab:doctor:secrets                                      # GitLab | Check if the database encrypted values can be...
rake gitlab:env:info                                            # GitLab | Env | Show information about GitLab and its e...
rake gitlab:exclusive_lease:clear[scope]                        # GitLab | Exclusive Lease | Clear existing exclusive le...
rake gitlab:external_diffs:force_object_storage                 # Override external diffs in file storage to be in objec...
rake gitlab:features:enable_rugged                              # GitLab | Features | Enable direct Git access via Rugge...
rake gitlab:generate_sample_prometheus_data[environment_id]     # GitLab | Generate Sample Prometheus Data
rake gitlab:git:fsck                                            # GitLab | Git | Check all repos integrity
rake gitlab:gitaly:check                                        # GitLab | Gitaly | Check the health of Gitaly
rake gitlab:gitaly:install[dir,storage_path,repo]               # GitLab | Gitaly | Install or upgrade gitaly
rake gitlab:gitlab_shell:check                                  # GitLab | GitLab Shell | Check the configuration of Git...
rake gitlab:import:all_users_to_all_groups                      # GitLab | Import | Add all users to all groups (admin u...
rake gitlab:import:all_users_to_all_projects                    # GitLab | Import | Add all users to all projects (admin...
rake gitlab:import:repos[import_path]                           # GitLab | Import | Import bare repositories from reposi...
rake gitlab:import:user_to_groups[email]                        # GitLab | Import | Add a specific user to all groups (a...
rake gitlab:import:user_to_projects[email]                      # GitLab | Import | Add a specific user to all projects ...
rake gitlab:import_export:bump_version                          # GitLab | Import/Export | Bumps the Import/Export versi...
rake gitlab:import_export:data                                  # GitLab | Import/Export | Display exported DB structure
rake gitlab:import_export:export                                # GitLab | Import/Export | EXPERIMENTAL | Export large p...
rake gitlab:import_export:import                                # GitLab | Import/Export | EXPERIMENTAL | Import large p...
rake gitlab:import_export:version                               # GitLab | Import/Export | Show Import/Export version
rake gitlab:incoming_email:check                                # GitLab | Incoming Email | Check the configuration of R...
rake gitlab:ldap:rename_provider[old_provider,new_provider]     # GitLab | LDAP | Rename provider
rake gitlab:lfs:check                                           # GitLab | LFS | Check integrity of uploaded LFS objects
rake gitlab:lfs:migrate                                         # GitLab | LFS | Migrate LFS objects to remote storage
rake gitlab:orphans:check                                       # Gitlab | Orphans | Check for orphaned namespaces and r...
rake gitlab:orphans:check_namespaces                            # GitLab | Orphans | Check for orphaned namespaces in th...
rake gitlab:orphans:check_repositories                          # GitLab | Orphans | Check for orphaned repositories in ...
rake gitlab:packages:events:generate                            # GitLab | Packages | Events | Generate hll counter even...
rake gitlab:packages:migrate                                    # GitLab | Packages | Migrate packages files to remote s...
rake gitlab:praefect:replicas[project_id]                       # GitLab | Praefect | Check replicas
rake gitlab:seed:group_seed[subgroups_depth,username]           # Seed groups with sub-groups/projects/epics/milestones ...
rake gitlab:seed:issues                                         # GitLab | Seed | Seeds issues
rake gitlab:setup                                               # GitLab | Setup production application
rake gitlab:shell:build_missing_projects                        # GitLab | Shell | Build missing projects
rake gitlab:shell:install[repo]                                 # GitLab | Shell | Install or upgrade gitlab-shell
rake gitlab:shell:setup                                         # GitLab | Shell | Setup gitlab-shell
rake gitlab:sidekiq:check                                       # GitLab | Sidekiq | Check the configuration of Sidekiq
rake gitlab:snippets:list_non_migrated                          # GitLab | Show non migrated snippets
rake gitlab:snippets:migrate[ids]                               # GitLab | Migrate specific snippets to git
rake gitlab:snippets:migration_status                           # GitLab | Show whether there are snippet background mig...
rake gitlab:storage:hashed_attachments                          # Gitlab | Storage | Summary of project attachments usin...
rake gitlab:storage:hashed_projects                             # Gitlab | Storage | Summary of existing projects using ...
rake gitlab:storage:legacy_attachments                          # Gitlab | Storage | Summary of project attachments usin...
rake gitlab:storage:legacy_projects                             # Gitlab | Storage | Summary of existing projects using ...
rake gitlab:storage:list_hashed_attachments                     # Gitlab | Storage | List existing project attachments u...
rake gitlab:storage:list_hashed_projects                        # Gitlab | Storage | List existing projects using Hashed...
rake gitlab:storage:list_legacy_attachments                     # Gitlab | Storage | List existing project attachments u...
rake gitlab:storage:list_legacy_projects                        # Gitlab | Storage | List existing projects using Legacy...
rake gitlab:storage:migrate_to_hashed                           # GitLab | Storage | Migrate existing projects to Hashed...
rake gitlab:storage:rollback_to_legacy                          # GitLab | Storage | Rollback existing projects to Legac...
rake gitlab:tcp_check[host,port]                                # GitLab | Check TCP connectivity to a specific host and...
rake gitlab:test                                                # GitLab | Run all tests
rake gitlab:two_factor:disable_for_all_users                    # GitLab | 2FA | Disable Two-factor authentication (2FA)...
rake gitlab:two_factor:rotate_key:apply                         # GitLab | 2FA | Rotate Key | Encrypt user OTP secrets w...
rake gitlab:two_factor:rotate_key:rollback                      # GitLab | 2FA | Rotate Key | Rollback to secrets encryp...
rake gitlab:update_project_templates                            # GitLab | Update project templates
rake gitlab:update_templates                                    # GitLab | Update templates
rake gitlab:uploads:check                                       # GitLab | Uploads | Check integrity of uploaded files
rake gitlab:uploads:migrate                                     # GitLab | Uploads | Migrate the uploaded files of speci...
rake gitlab:uploads:migrate:all                                 # GitLab | Uploads | Migrate all uploaded files to objec...
rake gitlab:uploads:migrate_to_local                            # GitLab | Uploads | Migrate the uploaded files of speci...
rake gitlab:uploads:migrate_to_local:all                        # GitLab | Uploads | Migrate all uploaded files to local...
rake gitlab:uploads:sanitize:remove_exif                        # GitLab | Uploads | Remove EXIF from images
rake gitlab:usage_data:dump_sql_in_json                         # GitLab | UsageData | Generate raw SQLs for usage ping ...
rake gitlab:usage_data:dump_sql_in_yaml                         # GitLab | UsageData | Generate raw SQLs for usage ping ...
rake gitlab:web_hook:add                                        # GitLab | Webhook | Adds a webhook to the projects
rake gitlab:web_hook:list                                       # GitLab | Webhook | List webhooks
rake gitlab:web_hook:rm                                         # GitLab | Webhook | Remove a webhook from the projects
rake gitlab:workhorse:install[dir,repo]                         # GitLab | Workhorse | Install or upgrade gitlab-workhorse
rake gitlab:x509:update_signatures                              # GitLab | X509 | Update signatures when certificate sto...
rake grape:path_helpers                                         # Print route helper methods
rake grape:routes                                               # Print compiled grape routes
rake hipchat:send[message]                                      # Sends a HipChat message as a particular user
rake import:github[token,gitlab_username,project_path]          # GitLab | Import | Import a GitHub project - Example: i...
rake jira:generate_consumer_key                                 # Generate a consumer key for your application
rake jira:generate_public_cert                                  # Run the system call to generate a RSA public certificate
rake log:clear                                                  # Truncates all/specified *.log files in log/ to zero by...
rake metrics:setup_common_metrics                               # GitLab | Metrics | Setup common metrics
rake middleware                                                 # Prints out your Rack middleware stack
rake migrate_iids                                               # GitLab | Build internal ids for issues and merge requests
rake postgresql_md5_hash                                        # GitLab | Generate PostgreSQL Password Hash
rake raven:test[dsn]                                            # Send a test event to the remote Sentry server
rake restart                                                    # Restart app by touching tmp/restart.txt
rake secret                                                     # Generate a cryptographically secure secret key (this i...
rake setup                                                      # GitLab | Setup gitlab db
rake stats                                                      # Report code statistics (KLOCs, etc) from the applicati...
rake test                                                       # GitLab | Run all tests
rake test:db                                                    # Run tests quickly, but also reset db
rake test:system                                                # Run system tests only
rake time:zones[country_or_offset]                              # List all time zones, list by two-letter country code (...
rake tmp:clear                                                  # Clear cache, socket and screenshot files from tmp/ (na...
rake tmp:create                                                 # Creates tmp directories for cache, sockets, and pids
rake tokens:reset_all_email                                     # Reset all GitLab incoming email tokens
rake tokens:reset_all_feed                                      # Reset all GitLab feed tokens
rake yarn                                                       # Install Node dependencies with Yarn
rake yarn:available                                             # Ensure Yarn is installed
rake yarn:check                                                 # Ensure Node dependencies are installed
rake yarn:clobber                                               # Remove Node dependencies
rake yarn:install                                               # Install Node dependencies with Yarn / Install all Java...
rake zeitwerk:check                                             # Checks project structure for Zeitwerk compatibility

 

Posted by Uli Köhler in Ruby

How to use child_process.exec in Koa (async/await)

First install the child-process-promise library

npm i --save child-process-promise

Then you can use it like this:

const router = require('koa-router')();
const {exec} = require('child-process-promise');

router.get('/test', async ctx => {
  const [stdout, stderr] = await exec('python myscript.py')
  const ipv6 = stdout.toString();
  ctx.body = ipv6;
});

 

Posted by Uli Köhler in Javascript

How to format IPv6 address in groups of 32 bits in Python

def format_ipv6addr_group2(addr):
    """
    Format IPv6 addresses in 4 \n-separated groups of 32 bits
    Returns a string
    """
    addr_s = str(addr.exploded)
    return f"{addr_s[:10]}\n{addr_s[10:20]}\n{addr_s[20:30]}\n{addr_s[30:40]}"

Usage example:

addr = ipaddress.IPv6Address("fd66:6cbb:8c10:1234:4567:89ab:cdef:0123")
print(format_ipv6addr_group2(addr))

Output:

fd66:6cbb:
8c10:1234:
4567:89ab:
cdef:0123

 

Posted by Uli Köhler in Python

How to generate random IPv6 addresses in a given network using Python

This code generates random IPv6 addresses in a given network using Python’s ipaddress module:

import ipaddress
import random

def random_ipv6_addr(network):
    """
    Generate a random IPv6 address in the given network
    Example: random_ipv6_addr("fd66:6cbb:8c10::/48")
    Returns an IPv6Address object.
    """
    net = ipaddress.IPv6Network(network)
    # Which of the network.num_addresses we want to select?
    addr_no = random.randint(0, net.num_addresses)
    # Create the random address by converting to a 128-bit integer, adding addr_no and converting back
    network_int = int.from_bytes(net.network_address.packed, byteorder="big")
    addr_int = network_int + addr_no
    addr = ipaddress.IPv6Address(addr_int.to_bytes(16, byteorder="big"))
    return addr

# Usage example
print(random_ipv6_addr("fdce:4879:a1e9::/48"))
# Prints e.g. fdce:4879:a1e9:e351:1a01:be9:4d9a:157d

It works by first converting the IPv6 network address to binary and then adding a random host number. After that, it will be converted back to an IPv6Address object.

Posted by Uli Köhler in Networking, Python

How to modify file inside a ZIP file using Python

Python provides the zipfile module to read and write ZIP files. Our previous posts Python example: List files in ZIP archive and Downloading & reading a ZIP file in memory using Python show how to list and read files inside a ZIP file.

In this example, we will show how to copy the files from one ZIP file to another and modify one of the files in the progress. This is often the case if you want to use ZIP file formats like ODT or LBX as templates, replacing parts of the text content of a file.

import zipfile

with zipfile.ZipFile(srcfile) as inzip, zipfile.ZipFile(dstfile, "w") as outzip:
    # Iterate the input files
    for inzipinfo in inzip.infolist():
        # Read input file
        with inzip.open(inzipinfo) as infile:
            if inzipinfo.filename == "test.txt":
                content = infile.read()
                # Modify the content of the file by replacing a string
                content = content.replace("abc", "123")
                # Write conte
                outzip.writestr(inzipinfo.filename, content)
            else: # Other file, dont want to modify => just copy it
                

After opening both the input file and the output ZIP using

with zipfile.ZipFile(srcfile) as inzip, zipfile.ZipFile(dstfile, "w") as outzip:

we iterate through all the files in the input ZIP file:

for inzipinfo in inzip.infolist():

In case we’ve encountered the file we want to modify, which is identified by it’s filename test.txt:

if inzipinfo.filename == "test.txt":

we read and modify the content ….

with inzip.open(inzipinfo) as infile:
    content = infile.read().replace("abc", "123")

… and write the modified content to the output ZIP:

outzip.writestr("test.txt", content)

Otherwise, if the current file is not the file we want to modify,  we just copy the file to the output ZIP using

outzip.writestr(inzipinfo.filename, infile.read())

Note that the algorithm will always .read() the file from the input ZIP, hence its entire content will be temporarily stored in memory. Therefore, it doesn’t work well for files which are large when uncompressed.

Posted by Uli Köhler in Python

Where to find info about mbed mbed_app.json overridable parameters?

The first resource can have a look at is the platform configuration option page. Additionally, check the manual on how to use the mbed CLI to show configuration options.

Additionally, you can look at targets.json on GitHub:

For example, "Target" => "config" => "default-adc-vref" would need to be entered like this into mbed_app.json:

{
    "target_overrides": {
      "*": {
          "target.default-adc-vref": 3300
      }
    }
}

 

Posted by Uli Köhler in C/C++, mbed, PlatformIO

How to fix mbed AnalogIn.read_voltage() returning nan or 0.000 (PlatformIO)

Problem:

You are trying to read an ADC voltage in mbed / PlatformIO like this:

AnalogIn myADC(PA_5); 
// Read and print voltage, then return
float v = myADC.read_voltage();
printf("%f\n", v);

but this only prints nan or 0.000.

Solution:

mbed doesn’t know the reference voltage for your platform. The easiest method is to provide the referene voltage in the constructor of AnalogIn:

AnalogIn myADC(PA_5, 3.3);

This specifies a reference voltage of 3.3V. While this applies to most applications in their default configuration, note that the reference voltage might be different depending on the configuration of your microcontroller.

In my experience, it’s almost always better to experimentally verify the reference voltage instead of trying to theorize about it if it’s not immediately obvious.

Full example:

#include <mbed.h>

BufferedSerial pc(USBTX, USBRX, 115200); // tx, rx

AnalogIn   myADC(PA_5, 3.3);

FileHandle *mbed::mbed_override_console(int fd) {
    return &pc;
}

int main() {
  while(1) {
    float v = myADC.read_voltage();
    printf("%f\n", v);
    ThisThread::sleep_for(100ms);
  }
}
{
    "target_overrides": {
      "*": {
        "target.printf_lib": "std"
      }
    }
}
Posted by Uli Köhler in C/C++, mbed, PlatformIO

How to fix mbed printf() ignoring decimals in PlatformIO

Problem:

You are using code like

printf("%.2f\n", myFloat);

in your mbed/PlatformIO application, but instead of printing myFloat with 2 decimal places, it always prints it with 6 decimal places (like 0.000000).

Solution:

mbed uses the minimal-printf library by default which is configured to save space on the Microcontroller. Hence, float max decimals support is disabled by default. In order to get all printf features at the expense of more flash usage and much slower executing, us add mbed_app.json in the root directory of the PlatformIO project with "target.printf_lib": "std":

{
    "target_overrides": {
      "*": {
        "target.printf_lib": "std"
      }
    }
}

See the platform configuration option page for more details and similar options.

Posted by Uli Köhler in C/C++, mbed, PlatformIO

How to fix mbed printf() printing literal %f in PlatformIO

Problem:

You are using code like

printf("%f\n", myFloat);

in your mbed/PlatformIO application, but instead of printing myFloat it prints literal %f.

Solution:

mbed uses the minimal-printf library by default which is configured to save space on the Microcontroller. Hence, float support (i.e. %f support) is disabled by default. You need to enable it by adding mbed_app.json in the root directory of the PlatformIO project with "platform.minimal-printf-enable-floating-point": true:

{
    "target_overrides": {
      "*": {
        "platform.minimal-printf-enable-floating-point": true
      }
    }
}

See the platform configuration option page for more details and similar options.

Posted by Uli Köhler in C/C++, mbed, PlatformIO

How to fix Python asyncio RuntimeError: There is no current event loop in thread …

Problem:

You are trying to run your Python application, but you see an error message like

Traceback (most recent call last):
  [...]
  File "/mnt/KATranslationCheck/CrowdinLogin.py", line 38, in get_crowdin_tokens
    return asyncio.get_event_loop().run_until_complete(async_get_crowdin_tokens(username, password))
  File "/usr/lib/python3.6/asyncio/events.py", line 694, in get_event_loop
    return get_event_loop_policy().get_event_loop()
  File "/usr/lib/python3.6/asyncio/events.py", line 602, in get_event_loop
    % threading.current_thread().name)
RuntimeError: There is no current event loop in thread 'worker 3'.

Solution:

You are trying to run asyncio.get_event_loop() in some thread other than the main thread – however, asyncio only generates an event loop for the main thread.

Use this function instead of asyncio.get_event_loop():

import asyncio

def get_or_create_eventloop():
    try:
        return asyncio.get_event_loop()
    except RuntimeError as ex:
        if "There is no current event loop in thread" in str(ex):
            loop = asyncio.new_event_loop()
            asyncio.set_event_loop(loop)
            return asyncio.get_event_loop()

It will first try asyncio.get_event_loop(). In case that doesn’t work, it will generate a new event loop for the current thread using

loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)

and then returns this event loop.

Note that while this works well for generating the event loop, but depending on the way you use the event loop, you might encounter further error messages like

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/pyppeteer/launcher.py", line 305, in launch
    return await Launcher(options, **kwargs).launch()
  File "/usr/local/lib/python3.6/dist-packages/pyppeteer/launcher.py", line 157, in launch
    signal.signal(signal.SIGINT, _close_process)
  File "/usr/lib/python3.6/signal.py", line 47, in signal
    handler = _signal.signal(_enum_to_int(signalnum), _enum_to_int(handler))
ValueError: signal only works in main thread

that are usually not as easy to fix and require some restructuring of your program.

Posted by Uli Köhler in Python

C# minimal program that shows a message popup dialog

This program just shows a message box and exits:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Windows.Forms;

namespace MinimalMessageBoxProgram
{
    class MinimalMessageBoxProgram
    {
        public static void Main(string[] args)
        {
            MessageBox.Show("This is the minimal text shown in the message dialog.");
        }
    }
}

 

Posted by Uli Köhler in C#

How to fix pyppeteer pyppeteer.errors.BrowserError: Browser closed unexpectedly:

Problem:

You want to run your Pyppeteer application on Linux, but you see an error message like

Traceback (most recent call last):
  File "PyppeteerExample.py", line 15, in <module>
    asyncio.get_event_loop().run_until_complete(main())
  File "/usr/lib/python3.6/asyncio/base_events.py", line 484, in run_until_complete
    return future.result()
  File "PyppeteerExample.py", line 6, in main
    browser = await launch()
  File "/usr/local/lib/python3.6/dist-packages/pyppeteer/launcher.py", line 305, in launch
    return await Launcher(options, **kwargs).launch()
  File "/usr/local/lib/python3.6/dist-packages/pyppeteer/launcher.py", line 166, in launch
    self.browserWSEndpoint = get_ws_endpoint(self.url)
  File "/usr/local/lib/python3.6/dist-packages/pyppeteer/launcher.py", line 225, in get_ws_endpoint
    raise BrowserError('Browser closed unexpectedly:\n')
pyppeteer.errors.BrowserError: Browser closed unexpectedly:

Solution:

In most cases, the underlying error for this error message is Puppetteer’s libX11-xcb.so.1: cannot open shared object file: No such file or directory. In order to fix that, you need to install dependency libraries for Chromium which is used internally by Puppeteer / Pyppeteer:

sudo apt install -y gconf-service libasound2 libatk1.0-0 libc6 libcairo2 libcups2 libdbus-1-3 libexpat1 libfontconfig1 libgcc1 libgconf-2-4 libgdk-pixbuf2.0-0 libglib2.0-0 libgtk-3-0 libnspr4 libpango-1.0-0 libpangocairo-1.0-0 libstdc++6 libx11-6 libx11-xcb1 libxcb1 libxcomposite1 libxcursor1 libxdamage1 libxext6 libxfixes3 libxi6 libxrandr2 libxrender1 libxss1 libxtst6 ca-certificates fonts-liberation libappindicator1 libnss3 lsb-release xdg-utils wget

 

Posted by Uli Köhler in Pyppeteer, Python

How to get hostmask/netmask for given prefix length in Python

In order to get the host mask for e.g. a /112 IPv6 prefix, use:

import ipaddress
# Get netmask for a /112 prefix
ipaddress.IPv6Network("::/112").netmask

# Get host mask for a /112 prefix
ipaddress.IPv6Network("::/112").hostmask

 

Posted by Uli Köhler in Networking, Python

How to fix Python3 TypeError: unsupported operand type(s) for &: ‘bytes’ and ‘bytes’

Problem:

You want to perform bitwise boolean operations on bytes() arrays in Python, but you see an error message like

TypeError: unsupported operand type(s) for &: 'bytes' and 'bytes'

or

TypeError: unsupported operand type(s) for |: 'bytes' and 'bytes'

or

TypeError: unsupported operand type(s) for ^: 'bytes' and 'bytes'

Solution:

Python can’t perform bitwise operations directly on byte arrays. However, you can use the code from How to perform bitwise boolean operations on bytes() in Python3:

def bitwise_and_bytes(a, b):
    result_int = int.from_bytes(a, byteorder="big") & int.from_bytes(b, byteorder="big")
    return result_int.to_bytes(max(len(a), len(b)), byteorder="big")

def bitwise_or_bytes(a, b):
    result_int = int.from_bytes(a, byteorder="big") | int.from_bytes(b, byteorder="big")
    return result_int.to_bytes(max(len(a), len(b)), byteorder="big")

def bitwise_xor_bytes(a, b):
    result_int = int.from_bytes(a, byteorder="big") ^ int.from_bytes(b, byteorder="big")
    return result_int.to_bytes(max(len(a), len(b)), byteorder="big")

# Example usage:

a = bytes([0x00, 0x01, 0x02, 0x03])
b = bytes([0x03, 0x02, 0x01, 0xff])

print(bitwise_and_bytes(a, b)) # b'\x00\x00\x00\x03'
print(bitwise_or_bytes(a, b)) # b'\x03\x03\x03\xff'
print(bitwise_xor_bytes(a, b)) # b'\x03\x03\x03\xfc'
Posted by Uli Köhler in Python

How to perform bitwise boolean operations on bytes() in Python3

Performing bitwise operations on bytes() instances in Python3.2+ is easy but not straightforward:

  1. Use int.from_bytes(...) to acquire an integer representing the byte array
  2. Perform bitwise operations with said integer
  3. Use result.to_bytes(...) to convert back the integer to a bytes() array

Note that for the result to make any sense, you need to ensure that both bytes() instances have the same length.

Python code:

def bitwise_and_bytes(a, b):
    result_int = int.from_bytes(a, byteorder="big") & int.from_bytes(b, byteorder="big")
    return result_int.to_bytes(max(len(a), len(b)), byteorder="big")

def bitwise_or_bytes(a, b):
    result_int = int.from_bytes(a, byteorder="big") | int.from_bytes(b, byteorder="big")
    return result_int.to_bytes(max(len(a), len(b)), byteorder="big")

def bitwise_xor_bytes(a, b):
    result_int = int.from_bytes(a, byteorder="big") ^ int.from_bytes(b, byteorder="big")
    return result_int.to_bytes(max(len(a), len(b)), byteorder="big")

Example usage:

a = bytes([0x00, 0x01, 0x02, 0x03])
b = bytes([0x03, 0x02, 0x01, 0xff])

print(bitwise_and_bytes(a, b)) # b'\x00\x00\x00\x03'
print(bitwise_or_bytes(a, b)) # b'\x03\x03\x03\xff'
print(bitwise_xor_bytes(a, b)) # b'\x03\x03\x03\xfc'

 

Posted by Uli Köhler in Python

Bitwise operation with IPv6 addresses and networks in Python

Python3 features the easy-to-use ipaddress library providing many calculations. However, bitwise boolean operators are not available for addresses.

This post shows you how to perform bitwise operations with IPv6Address() objects. We’ll use the following strategy:

  1. Use .packed to get a binary bytes() instance of the IP address
  2. Use int.from_bytes() to acquire an integer representing the binary address
  3. Perform bitwise operations with said integer
  4. Use result.to_bytes(16, ...) to convert back the integer to a bytes() array in the correct byte order
  5. Construct an IPv6Address() object from the resulting byte array.

Python code:

import ipaddress

def bitwise_and_ipv6(addr1, addr2):
    result_int = int.from_bytes(addr1.packed, byteorder="big") & int.from_bytes(addr2.packed, byteorder="big")
    return ipaddress.IPv6Address(result_int.to_bytes(16, byteorder="big"))

def bitwise_or_ipv6(addr1, addr2):
    result_int = int.from_bytes(addr1.packed, byteorder="big") | int.from_bytes(addr2.packed, byteorder="big")
    return ipaddress.IPv6Address(result_int.to_bytes(16, byteorder="big"))

def bitwise_xor_ipv6(addr1, addr2):
    result_int = int.from_bytes(addr1.packed, byteorder="big") ^ int.from_bytes(addr2.packed, byteorder="big")
    return ipaddress.IPv6Address(result_int.to_bytes(16, byteorder="big"))

Example usage:

a = ipaddress.IPv6Address('2001:16b8:2703:8835:9ec7:a6ff:febe:96b1')
b = ipaddress.IPv6Address('2001:16b8:2703:4241:9ec7:a6ff:febe:96b1')

print(bitwise_and_ipv6(a, b)) # IPv6Address('2001:16b8:2703:1:9ec7:a6ff:febe:96b1')
print(bitwise_or_ipv6(a, b)) # IPv6Address('2001:16b8:2703:ca75:9ec7:a6ff:febe:96b1')
print(bitwise_xor_ipv6(a, b)) # IPv6Address('0:0:0:ca74::')

Similarly, you can use the code in order to manipulate IPv6Network() instances:

a = ipaddress.IPv6Network('2001:16b8:2703:8835:9ec7:a6ff:febe::/112')
b = ipaddress.IPv6Network('2001:16b8:2703:4241:9ec7:a6ff:febe::/112')

print(bitwise_and_ipv6(a.network_address, b.network_address)) # IPv6Address('2001:16b8:2703:1:9ec7:a6ff:febe:0')
print(bitwise_or_ipv6(a.network_address, b.network_address)) # IPv6Address('2001:16b8:2703:ca75:9ec7:a6ff:febe:0')
print(bitwise_xor_ipv6(a.network_address, b.network_address)) # IPv6Address('0:0:0:ca74::')

Note that the return type will always be IPv6Address() and never IPv6Network() since the result of the bitwise operation doesn’t have any netmask associated with it.

Besides .network_address you can also use other properties of IPv6Address() instances like .broadcast_address or .hostmask or .netmask.

Posted by Uli Köhler in Networking, Python

How to get current page URL in pyppeteer

In pyppeteer you can use

url = await page.evaluate("() => window.location.href")

in order to get the current URL. Note that page.evaluate() runs whatever Javascript your give it – hence you can use your Javascript skills in order to create the desired effect.

Full example

import asyncio
from pyppeteer import launch

async def main():
    browser = await launch()
    page = await browser.newPage()
    await page.goto('https://www.techoverflow.net')

    # Get the URL and print it
    url = await page.evaluate("() => window.location.href")
    print(url) # prints https://www.techoverflow.net/

    # Cleanuip
    await browser.close()

asyncio.get_event_loop().run_until_complete(main())

 

Posted by Uli Köhler in Pyppeteer, Python

How simulate click using pyppeteer

In order to click a button or a link using the the pyppeteer library, you can use page.evaluate().

If you have an <button> element or a link (<a>) like

<button id="mybutton">

you can use

# Now click the search button    
await page.evaluate(f"""() => {{
    document.getElementById('mybutton').dispatchEvent(new MouseEvent('click', {{
        bubbles: true,
        cancelable: true,
        view: window
    }}));
}}""")

in order to generate a MouseEvent that simulates a click. Note that page.evaluate() will run any Javascript code you pass to it, so you can use your Javascript skills in order to create the desired effect

Also see https://gomakethings.com/how-to-simulate-a-click-event-with-javascript/ for more details on how to simulate mouse clicks in pure Javascript without relying on jQuery.

Note that page.evaluate() will just run any Javascript code you give it, so you can put your Javascript skills to use in order to manipulate the page.

Full example

This example will open https://techoverflow.net, enter a search term into the search field, click the search button and then create a screenshot

import asyncio
from pyppeteer import launch

async def main():
    browser = await launch()
    page = await browser.newPage()
    await page.goto('https://techoverflow.net')

    # Fill content into the search field
    content = "pypetteer"
    await page.evaluate(f"""() => {{
        document.getElementById('s').value = '{content}';
    }}""")

    # Now click the search button    
    await page.evaluate(f"""() => {{
        document.getElementById('searchsubmit').dispatchEvent(new MouseEvent('click', {{
            bubbles: true,
            cancelable: true,
            view: window
        }}));
    }}""")

    # Wait until search results page has been loaded
    await page.waitForSelector(".archive-title")

    # Now take screenshot and exit
    await page.screenshot({'path': 'screenshot.png'})
    await browser.close()

asyncio.get_event_loop().run_until_complete(main())

The result will look like this:

Posted by Uli Köhler in Pyppeteer, Python

How to fill <input> field using pyppeteer

In order to fill an input field using the pyppeteer library, you can use page.evaluate().

If you have an <input> element like

<input name="myinput" id="myinput" type="text">

you can use

content = "My content" # This will be filled into <input id="myinput"> !
await page.evaluate(f"""() => {{
    document.getElementById('myinput').value = '{content}';
}}""")

Note that page.evaluate() will just run any Javascript code you give it, so you can put your Javascript skills to use in order to manipulate the page.

Full example

This example will open https://techoverflow.net, enter a search term into the search field and then create a screenshot

#!/usr/bin/env python3
import asyncio
from pyppeteer import launch

async def main():
    browser = await launch()
    page = await browser.newPage()
    await page.goto('https://techoverflow.net')
    
    # This example fills content into the search field
    content = "My search term"
    await page.evaluate(f"""() => {{
        document.getElementById('s').value = '{content}';
    }}""")

    # Make screenshot
    await page.screenshot({'path': 'screenshot.png'})
    await browser.close()

asyncio.get_event_loop().run_until_complete(main())

The result will look like this:

Posted by Uli Köhler in Pyppeteer, Python

pypetteer is a spelling mistake. It’s called pyppeteer !

If you found this page when looking for the pyppeteer library, you must have spelled it wrong.

The correct spelling is pyppeteer: Two ps and only one t!

Posted by Uli Köhler in Pyppeteer, Python