Programming languages

How to fix Python ValueError: unsupported format character ‘ ‘ (0x20) at index 3

Problem:

You are trying to use Python format strings like

"Hello %.3 world" % 1.234

but you see an error message like

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-4-7cbe94e4525d> in <module>
----> 1 "Hello %.3 world" % 1.234

ValueError: unsupported format character ' ' (0x20) at index 9

Solution:

Your format string %.3 is incomplete ! You need to specify the type of value to format, e.g. f for floating point, d for integers, s for strings etc. So instead of %.3, write %.3f, or instead of just % write e.g. %f, %d or %s depending on the data type of the variable you want to insert there.

Posted by Uli Köhler in Python

How to auto-fit Pandas pd.to_excel() XLSX column width

If you export XLSX data using df.to_excel(), the column widths in the spreadsheet are left as default and are not adjusted automatically:

# Load example dataset
df = pd.read_csv("https://datasets.techoverflow.net/timeseries-example.csv", parse_dates=["Timestamp"])
df.set_index("Timestamp", inplace=True)

# Export dataset to XLSX
df.to_excel("example.xlsx")

Solution

You can use UliPlot‘s auto_adjust_xlsx_column_width in order to automatically adjust the column width.

pip install UliPlot

Then use it like this in order to  export the XLSX:

from UliPlot.XLSX import auto_adjust_xlsx_column_width

# Load example dataset
df = pd.read_csv("https://datasets.techoverflow.net/timeseries-example.csv", parse_dates=["Timestamp"])
df.set_index("Timestamp", inplace=True)

# Export dataset to XLSX
with pd.ExcelWriter("example.xlsx") as writer:
    df.to_excel(writer, sheet_name="MySheet")
    auto_adjust_xlsx_column_width(df, writer, sheet_name="MySheet", margin=0)

Note that the algorithm currently tends to oversize the columns a bit, but in most cases, every type of column will fit.

Posted by Uli Köhler in pandas, Python

Minimal ESP32 NTP client example using NTPClient_Generic and PlatformIO

Example of using NTPClient_Generic

main.cpp

#include <Arduino.h>
#include <WiFi.h>
#include <NTPClient_Generic.h>

#define TIME_ZONE_OFFSET_HRS 1 // UTC+1 for Germany, winter time
#define NTP_UPDATE_INTERVAL_MS 60000L // Update every 60s automatically
WiFiUDP ntpUDP;
NTPClient ntpClient(ntpUDP, "europe.pool.ntp.org", (3600 * TIME_ZONE_OFFSET_HRS), NTP_UPDATE_INTERVAL_MS);

void ntpUpdateTask(void* param) {
  while(true) {
    // Update NTP. This will only ACTUALLY update if
    // the NTP client has not updated for NTP_UPDATE_INTERVAL_MS
    ntpClient.update();
    vTaskDelay(1000 / portTICK_PERIOD_MS);
  }
}

void setup() {
  Serial.begin(115200);
  // Force reset Wifi to avoid failure to connect
  WiFi.disconnect(true);
  // Set hostname
  WiFi.config(INADDR_NONE, INADDR_NONE, INADDR_NONE);
  WiFi.setHostname("ESP-NTP");
  // Connect to Wifi
  WiFi.begin("MyWifiSSID", "MyWifiPassword");
  while (WiFi.status() != WL_CONNECTED) {
      delay(100);
      Serial.println("Wifi connecting...");
  }

  // Start NTP client (i.e. start listening for NTP packets)
  ntpClient.begin();
  // Create task to automatically update NTP in the background
  xTaskCreate(ntpUpdateTask, "NTP update", 2000, nullptr, 1, nullptr);
}

void loop() {
  delay(1000);
  if (ntpClient.updated()) {
    Serial.println("# Time in sync with NTP server");
  } else {
    Serial.println("# TIME NOT IN SYNC WITH NTP SERVER !");
    return; // Do not print time
  }

  Serial.println("UTC : " + ntpClient.getFormattedUTCTime());
  Serial.println("UTC : " + ntpClient.getFormattedUTCDateTime());
  Serial.println("LOC : " + ntpClient.getFormattedTime());
}

platformio.ini

[env:nodemcu-32s]
platform = espressif32
board = nodemcu-32s
framework = arduino
monitor_speed = 115200
lib_deps =
    khoih.prog/NTPClient_Generic @ ^3.2.2
    Time

Example output:

Wifi connecting...
Wifi connecting...
Wifi connecting...
# TIME NOT IN SYNC WITH NTP SERVER !
# Time in sync with NTP server
UTC : 01:22:07
UTC : 01:22:07 Sun 07 Feb 2021
LOC : 02:22:07
# Time in sync with NTP server
UTC : 01:22:08
UTC : 01:22:08 Sun 07 Feb 2021
LOC : 02:22:08
# Time in sync with NTP server
UTC : 01:22:09
UTC : 01:22:09 Sun 07 Feb 2021
LOC : 02:22:09

 

Posted by Uli Köhler in C/C++, ESP8266/ESP32, PlatformIO

How to get current gateway IP address on ESP8266/ESP32

In order to get the current gateway IP address on the ESP8266 or ESP32, use:

WiFi.gatewayIP();

In order to print the gateway IP address on the serial port, use

Serial.println("Gateway IP address: ");
Serial.println(WiFi.gatewayIP());

In order to get the gateway IP address as string, use

String gatewayIP = WiFi.gatewayIP().toString();

 

Posted by Uli Köhler in C/C++, Embedded, ESP8266/ESP32, PlatformIO

ESP32 minimal WebSocket example (ESPAsyncWebserver / PlatformIO)

Minimal firmware to use WebSockets on the ESP32 using ESPAsyncWebserver:

main.cpp

#include <Arduino.h>
#include <WiFi.h>
#include <ESPAsyncWebServer.h>
#include <ArduinoJson.h>

AsyncWebServer server(80);
AsyncWebSocket ws("/ws");

AsyncWebSocketClient* wsClient;
 
void onWsEvent(AsyncWebSocket * server, AsyncWebSocketClient * client, AwsEventType type, void * arg, uint8_t *data, size_t len){
  if(type == WS_EVT_CONNECT){
    wsClient = client;
  } else if(type == WS_EVT_DISCONNECT){
    wsClient = nullptr;
  }
}

/**
 * Wait for WiFi connection, and, if not connected, reboot
 */
void waitForWiFiConnectOrReboot(bool printOnSerial=true) {
  uint32_t notConnectedCounter = 0;
  while (WiFi.status() != WL_CONNECTED) {
      delay(100);
      if(printOnSerial) {
        Serial.println("Wifi connecting...");
      }
      notConnectedCounter++;
      if(notConnectedCounter > 50) { // Reset board if not connected after 5s
          if(printOnSerial) {
            Serial.println("Resetting due to Wifi not connecting...");
          }
          ESP.restart();
      }
  }
  if(printOnSerial) {
    // Print wifi IP addess
    Serial.println("IP address: ");
    Serial.println(WiFi.localIP());
  }
}


void setup() { 
  Serial.begin(115200);
  WiFi.config(INADDR_NONE, INADDR_NONE, INADDR_NONE);
  WiFi.setHostname("ESP-Websocket-Test");

  WiFi.begin("MyWifiSSID", "MyWifiPassword");
  waitForWiFiConnectOrReboot();

  // Start webserver
  ws.onEvent(onWsEvent);
  server.addHandler(&ws);
  server.begin();
}

uint64_t counter = 0;
void loop() {
  // If client is connected ...
  if(wsClient != nullptr && wsClient->canSend()) {
    // .. send hello message :-)
    wsClient->text("Hello client");
  }

  // Wait 10 ms
  delay(10);
}

platformio.ini:

[env:nodemcu-32s]
platform = espressif32
board = nodemcu-32s
framework = arduino
monitor_speed = 115200
lib_deps =
    ESP Async [email protected]
    [email protected]

Python code for testing:

import websocket
ws = websocket.WebSocket()
ws.connect("ws://192.168.1.211/ws")
while True:
    result = ws.recv()
    print(result)

This will print Hello Client many times a second.

Posted by Uli Köhler in C/C++, ESP8266/ESP32, PlatformIO

How to fix Python WebSocket ValueError: scheme http is invalid

Problem:

You want to use Python to connect to your WebSocket server:

import websocket
ws = websocket.WebSocket()
ws.connect("http://192.168.1.211/ws")

But you see an error message like

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-2-5108525d0b43> in <module>
      1 import websocket
      2 ws = websocket.WebSocket()
----> 3 ws.connect("http://10.228.152.103/ws")
      4 while True:
      5     result = ws.recv()

c:\python38\lib\site-packages\websocket\_core.py in connect(self, url, **options)
    220         # FIXME: "header", "cookie", "origin" and "host" too
    221         self.sock_opt.timeout = options.get('timeout', self.sock_opt.timeout)
--> 222         self.sock, addrs = connect(url, self.sock_opt, proxy_info(**options),
    223                                    options.pop('socket', None))
    224 

c:\python38\lib\site-packages\websocket\_http.py in connect(url, options, proxy, socket)
    106         return _open_proxied_socket(url, options, proxy)
    107 
--> 108     hostname, port, resource, is_secure = parse_url(url)
    109 
    110     if socket:

c:\python38\lib\site-packages\websocket\_url.py in parse_url(url)
     61             port = 443
     62     else:
---> 63         raise ValueError("scheme %s is invalid" % scheme)
     64 
     65     if parsed.path:

ValueError: scheme http is invalid

Solution:

The websocket-client library wants you to use ws://host/path instead of http://host/path. So instead of

ws.connect("http://10.228.152.103/ws")

use

ws.connect("http://10.228.152.103/ws")
Posted by Uli Köhler in Python

How to fix Python WebSocket TypeError: __init__() missing 3 required positional arguments: ‘environ’, ‘socket’, and ‘rfile’

Problem:

You are trying to connect a WebSocket in Python:

import websocket
ws = websocket.WebSocket()

but you see an error message like

TypeError                                 Traceback (most recent call last)
<ipython-input-3-5108525d0b43> in <module>
      1 import websocket
----> 2 ws = websocket.WebSocket()
      3 ws.connect("ws://192.168.1.211/ws")
      4 while True:
      5     result = ws.recv()

TypeError: __init__() missing 3 required positional arguments: 'environ', 'socket', and 'rfile'

Solution:

You have installed the wrong library ! You need to use websocket-client instead of websocket !

First, uninstall websocket:

pip uninstall websocket

Now install websocket-client:

pip install websocket-client

Now try again – your code should work now.

Posted by Uli Köhler in Python

How to fix ESP32 too few arguments to function ‘esp_err_t esp_wifi_sta_wpa2_ent_enable(const esp_wpa2_config_t*)’

Problem:

While trying to compile your ESP32 application, you see this compiler error:

src\main.cpp: In function 'void wifiConnectWPAEAP(const char*, const char*, const char*, const char*)':
src\main.cpp:23:32: error: too few arguments to function 'esp_err_t esp_wifi_sta_wpa2_ent_enable(const esp_wpa2_config_t*)'   esp_wifi_sta_wpa2_ent_enable();
                                ^

Solution:

Replace

esp_wifi_sta_wpa2_ent_enable();

by

esp_wpa2_config_t config = WPA2_CONFIG_INIT_DEFAULT();
esp_wifi_sta_wpa2_ent_enable(&config);
Posted by Uli Köhler in C/C++, Embedded, ESP8266/ESP32

How to fix ESP32 fatal error: ESP8266WiFi.h: No such file or directory

Problem:

While trying to compile your ESP32 application, you see this compiler error:

fatal error: ESP8266WiFi.h: No such file or directory

Solution:

ESP8266WiFi.h is the Wifi header for the ESP8266. For the ESP32, it’s named WiFi.h. In order to fix the issue, replae

#include <ESP8266WiFi.h>

by

#include <WiFi.h>
Posted by Uli Köhler in C/C++, Embedded, ESP8266/ESP32

How to fix Alpine Linux fatal error: stdio.h: No such file or directory

Problem:

When trying to compile a C/C++ program or library on Alpine Linux, you see an error message like

/home/user/test.cpp:5:10: fatal error: stdio.h: No such file or directory
  123 | #include <stdio.h>
      |          ^~~~~~~~~

Solution:

Install the libc headers using

apk add musl-dev

 

Posted by Uli Köhler in C/C++, GCC errors, Linux

How to fix cfgrib TypeError: open_dataset() got an unexpected keyword argument ‘filter_by_keys’

Problem:

You want to open a GRB2 file using the cfgrib library using code like

import xarray as xr
ds = xr.open_dataset('myfile.grb2', engine='cfgrib', filter_by_keys={'typeOfLevel': 'atmosphere'})

But you see an exception like

TypeError: open_dataset() got an unexpected keyword argument 'filter_by_keys'

Solution:

You can’t use filter_by_keys=… as argument to xr.open_dataset() directly. You need to use backend_kwargs like this:

ds = xr.open_dataset('myfile.grb2', engine='cfgrib', backend_kwargs={'filter_by_keys': {'typeOfLevel': 'atmosphere'}})

 

Posted by Uli Köhler in Data science, Python

Minimal ESP8266 WiFi example

#include <Arduino.h>
#include <ESP8266WiFi.h>

void setup() {
  Serial.begin(115200);
  WiFi.begin("MySSID", "MyPassword");
  while (WiFi.status() != WL_CONNECTED) {
    Serial.println("Wifi connecting...");
    delay(500);
  }
  Serial.println("Wifi connected");
}

void loop() {
  // put your main code here, to run repeatedly:
}

 

Posted by Uli Köhler in C/C++, ESP8266/ESP32

How to fix .NET Core: Could not execute because the application was not found or a compatible .NET SDK is not installed.

Problem:

You are trying to run a .NET core command using e.g.

dotnet new console

but you see this error message:

Could not execute because the application was not found or a compatible .NET SDK is not installed.
Possible reasons for this include:
  * You intended to execute a .NET program:
      The application 'new' does not exist.
  * You intended to execute a .NET SDK command:
      It was not possible to find any installed .NET SDKs.
      Install a .NET SDK from:
        https://aka.ms/dotnet-download

Solution:

You didn’t install a .NET Core SDK, only the .NET Core host.

Install the SDK by using e.g. (this command works on Debian & Ubuntu)

sudo apt -y install dotnet-sdk-5.0

You might need to select the correct version (5.0 in this example).

Posted by Uli Köhler in C#

How to grep for WordPress DB_NAME, DB_USER, DB_PASSWORD and DB_HOST in wp-config.php

This grep statement filters out the DB_NAME, DB_USER, DB_PASSWORD and DB_HOST contents from a wp-config.php:

grep -oP "define\(['\"]DB_NAME['\"],\s*['\"]\K[^'\"]+(?=[\'\"]\s*\)\s*;)" wp-config.php

For example, from the wp-config.php line

define('DB_NAME', 'techoverflow');

it will extract techoverflow.

Here are the statements for DB_NAME, DB_USER, DB_PASSWORD and DB_HOST:

DB_NAME=$(grep -oP "define\(['\"]DB_NAME['\"],\s*['\"]\K[^'\"]+(?=[\'\"]\s*\)\s*;)" wp-config.php)
DB_USER=$(grep -oP "define\(['\"]DB_USER['\"],\s*['\"]\K[^'\"]+(?=[\'\"]\s*\)\s*;)" wp-config.php)
DB_PASSWORD=$(grep -oP "define\(['\"]DB_PASSWORD['\"],\s*['\"]\K[^'\"]+(?=[\'\"]\s*\)\s*;)" wp-config.php)
DB_HOST=$(grep -oP "define\(['\"]DB_HOST['\"],\s*['\"]\K[^'\"]+(?=[\'\"]\s*\)\s*;)" wp-config.php)

You can use this, for example, in automatic backup scripts without manually copying the password to the backup script

Posted by Uli Köhler in PHP, Wordpress

Best practice Angular 11 production build command

This is the command I use to make production builds of Angular 11+ webapps:
ng build --prod --aot --build-optimizer --common-chunk --vendor-chunk --named-chunks

 

While it will consume quite some CPU and RAM during the build, it will produce a highly efficient compiled output.

Posted by Uli Köhler in Angular, Javascript

How to load sample dataset in Pandas

This code loads an example dataset in Pandas but requires an internet connection:

import pandas as pd
iris = pd.read_csv('https://raw.githubusercontent.com/mwaskom/seaborn-data/master/iris.csv')

This is how the iris dataset looks like:

For a more detailed answer with other options on how to import example datasets, see this StackOverflow post.

Posted by Uli Köhler in pandas, Python

How to list gitlab-rake tasks

You can list the tasks available in your GitLab’s gitlab-rake using

gitlab-rake -T

In case you’re running a docker-compose based configuration, you can use

docker-compose exec gitlab gitlab-rake -T

where gitlab is the name of the docker-compose service in docker-compose.yml.

This will show you a list like

rake about                                                      # List versions of all Rails frameworks and the environment
rake acts_as_taggable_on_engine:install:migrations              # Copy migrations from acts_as_taggable_on_engine to app...
rake acts_as_taggable_on_engine:tag_names:collate_bin           # Forcing collate of tag names to utf8_bin
rake acts_as_taggable_on_engine:tag_names:collate_ci            # Forcing collate of tag names to utf8_general_ci
rake app:template                                               # Applies the template supplied by LOCATION=(/path/to/te...
rake app:update                                                 # Update configs and some other initially generated file...
rake assets:clean[keep]                                         # Remove old compiled assets
rake assets:clobber                                             # Remove compiled assets
rake assets:environment                                         # Load asset compile environment
rake assets:precompile                                          # Compile all the assets named in config.assets.precompile
rake brakeman                                                   # Security check via brakeman
rake cache:clear:redis                                          # GitLab | Cache | Clear redis cache
rake cache_digests:dependencies                                 # Lookup first-level dependencies for TEMPLATE (like mes...
rake cache_digests:nested_dependencies                          # Lookup nested dependencies for TEMPLATE (like messages...
rake ci:cleanup:builds                                          # GitLab | CI | Clean running builds
rake clean                                                      # Remove any temporary products
rake clobber                                                    # Remove any generated files
rake config_lint                                                # Checks syntax for shell scripts and nginx config files...
rake danger_local                                               # Run local Danger rules
rake db:create                                                  # Creates the database from DATABASE_URL or config/datab...
rake db:drop                                                    # Drops the database from DATABASE_URL or config/databas...
rake db:environment:set                                         # Set the environment value for the database
rake db:fixtures:load                                           # Loads fixtures into the current environment's database
rake db:migrate                                                 # Migrate the database (options: VERSION=x, VERBOSE=fals...
rake db:migrate:status                                          # Display status of migrations
rake db:obsolete_ignored_columns                                # Show a list of obsolete `ignored_columns`
rake db:prepare                                                 # Runs setup if database does not exist, or runs migrati...
rake db:rollback                                                # Rolls the schema back to the previous version (specify...
rake db:schema:cache:clear                                      # Clears a db/schema_cache.yml file
rake db:schema:cache:dump                                       # Creates a db/schema_cache.yml file
rake db:schema:dump                                             # Creates a db/schema.rb file that is portable against a...
rake db:schema:load                                             # Loads a schema.rb file into the database
rake db:seed                                                    # Loads the seed data from db/seeds.rb
rake db:seed:replant                                            # Truncates tables of each database for current environm...
rake db:seed_fu                                                 # Loads seed data for the current environment
rake db:setup                                                   # Creates the database, loads the schema, and initialize...
rake db:structure:dump                                          # Dumps the database structure to db/structure.sql
rake db:structure:load                                          # Recreates the databases from the structure.sql file
rake db:version                                                 # Retrieves the current schema version number
rake dev:load                                                   # GitLab | Dev | Eager load application
rake dev:setup                                                  # GitLab | Dev | Setup developer environment (db, fixtures)
rake downtime_check                                             # Checks if migrations in a branch require downtime
rake file_hooks:validate                                        # Validate existing file hooks
rake gemojione:aliases                                          # Generates Emoji SHA256 digests
rake gemojione:install_assets                                   # Install Emoji Image Assets
rake gettext:add_language[language]                             # add a new language
rake gettext:find                                               # Update pot/po files
rake gettext:lint                                               # Lint all po files in `locale/
rake gettext:pack                                               # Create mo-files
rake gettext:po_to_json                                         # Convert PO files to JS files
rake gettext:regenerate                                         # Regenerate gitlab.pot file
rake gettext:store_model_attributes                             # write the model attributes to <locale_path>/model_attr...
rake gitlab:app:check                                           # GitLab | App | Check the configuration of the GitLab R...
rake gitlab:artifacts:check                                     # GitLab | Artifacts | Check integrity of uploaded job a...
rake gitlab:artifacts:migrate                                   # GitLab | Artifacts | Migrate files for artifacts to co...
rake gitlab:assets:check_page_bundle_mixins_css_for_sideeffects # GitLab | Assets | Check that scss mixins do not introd...
rake gitlab:assets:clean                                        # GitLab | Assets | Clean up old compiled frontend assets
rake gitlab:assets:compile                                      # GitLab | Assets | Compile all frontend assets
rake gitlab:assets:compile_webpack_if_needed                    # GitLab | Assets | Compile all Webpack assets
rake gitlab:assets:fix_urls                                     # GitLab | Assets | Fix all absolute url references in CSS
rake gitlab:assets:purge                                        # GitLab | Assets | Remove all compiled frontend assets
rake gitlab:assets:purge_modules                                # GitLab | Assets | Uninstall frontend dependencies
rake gitlab:assets:vendor                                       # GitLab | Assets | Compile vendor assets
rake gitlab:backup:create                                       # GitLab | Backup | Create a backup of the GitLab system
rake gitlab:backup:restore                                      # GitLab | Backup | Restore a previously created backup
rake gitlab:check                                               # GitLab | Check the configuration of GitLab and its env...
rake gitlab:cleanup:block_removed_ldap_users                    # GitLab | Cleanup | Block users that have been removed ...
rake gitlab:cleanup:moved                                       # GitLab | Cleanup | Delete moved repositories
rake gitlab:cleanup:orphan_job_artifact_files                   # GitLab | Cleanup | Clean orphan job artifact files
rake gitlab:cleanup:orphan_lfs_file_references                  # GitLab | Cleanup | Clean orphan LFS file references
rake gitlab:cleanup:orphan_lfs_files                            # GitLab | Cleanup | Clean orphan LFS files
rake gitlab:cleanup:project_uploads                             # GitLab | Cleanup | Clean orphaned project uploads
rake gitlab:cleanup:remote_upload_files                         # GitLab | Cleanup | Clean orphan remote upload files th...
rake gitlab:cleanup:sessions:active_sessions_lookup_keys        # GitLab | Cleanup | Sessions | Clean ActiveSession look...
rake gitlab:container_registry:configure                        # GitLab | Container Registry | Configure
rake gitlab:db:clean_structure_sql                              # This adjusts and cleans db/structure.sql - it runs aft...
rake gitlab:db:composite_primary_keys_add                       # GitLab | DB | Adds primary keys to tables that only ha...
rake gitlab:db:composite_primary_keys_drop                      # GitLab | DB | Removes previously added composite prima...
rake gitlab:db:configure                                        # GitLab | DB | Configures the database by running migra...
rake gitlab:db:create_dynamic_partitions                        # Create missing dynamic database partitions
rake gitlab:db:downtime_check[ref]                              # GitLab | DB | Checks if migrations require downtime or...
rake gitlab:db:drop_tables                                      # GitLab | DB | Drop all tables
rake gitlab:db:dump_custom_structure                            # This dumps GitLab specific database details - it runs ...
rake gitlab:db:load_custom_structure                            # This loads GitLab specific database details - runs aft...
rake gitlab:db:mark_migration_complete[version]                 # GitLab | DB | Manually insert schema migration version
rake gitlab:db:reindex[index_name]                              # reindex a regular (non-unique) index without downtime ...
rake gitlab:db:setup_ee                                         # GitLab | DB | Sets up EE specific database functionality
rake gitlab:db:unattended                                       # GitLab | DB | Run database migrations and print `unatt...
rake gitlab:doctor:secrets                                      # GitLab | Check if the database encrypted values can be...
rake gitlab:env:info                                            # GitLab | Env | Show information about GitLab and its e...
rake gitlab:exclusive_lease:clear[scope]                        # GitLab | Exclusive Lease | Clear existing exclusive le...
rake gitlab:external_diffs:force_object_storage                 # Override external diffs in file storage to be in objec...
rake gitlab:features:enable_rugged                              # GitLab | Features | Enable direct Git access via Rugge...
rake gitlab:generate_sample_prometheus_data[environment_id]     # GitLab | Generate Sample Prometheus Data
rake gitlab:git:fsck                                            # GitLab | Git | Check all repos integrity
rake gitlab:gitaly:check                                        # GitLab | Gitaly | Check the health of Gitaly
rake gitlab:gitaly:install[dir,storage_path,repo]               # GitLab | Gitaly | Install or upgrade gitaly
rake gitlab:gitlab_shell:check                                  # GitLab | GitLab Shell | Check the configuration of Git...
rake gitlab:import:all_users_to_all_groups                      # GitLab | Import | Add all users to all groups (admin u...
rake gitlab:import:all_users_to_all_projects                    # GitLab | Import | Add all users to all projects (admin...
rake gitlab:import:repos[import_path]                           # GitLab | Import | Import bare repositories from reposi...
rake gitlab:import:user_to_groups[email]                        # GitLab | Import | Add a specific user to all groups (a...
rake gitlab:import:user_to_projects[email]                      # GitLab | Import | Add a specific user to all projects ...
rake gitlab:import_export:bump_version                          # GitLab | Import/Export | Bumps the Import/Export versi...
rake gitlab:import_export:data                                  # GitLab | Import/Export | Display exported DB structure
rake gitlab:import_export:export                                # GitLab | Import/Export | EXPERIMENTAL | Export large p...
rake gitlab:import_export:import                                # GitLab | Import/Export | EXPERIMENTAL | Import large p...
rake gitlab:import_export:version                               # GitLab | Import/Export | Show Import/Export version
rake gitlab:incoming_email:check                                # GitLab | Incoming Email | Check the configuration of R...
rake gitlab:ldap:rename_provider[old_provider,new_provider]     # GitLab | LDAP | Rename provider
rake gitlab:lfs:check                                           # GitLab | LFS | Check integrity of uploaded LFS objects
rake gitlab:lfs:migrate                                         # GitLab | LFS | Migrate LFS objects to remote storage
rake gitlab:orphans:check                                       # Gitlab | Orphans | Check for orphaned namespaces and r...
rake gitlab:orphans:check_namespaces                            # GitLab | Orphans | Check for orphaned namespaces in th...
rake gitlab:orphans:check_repositories                          # GitLab | Orphans | Check for orphaned repositories in ...
rake gitlab:packages:events:generate                            # GitLab | Packages | Events | Generate hll counter even...
rake gitlab:packages:migrate                                    # GitLab | Packages | Migrate packages files to remote s...
rake gitlab:praefect:replicas[project_id]                       # GitLab | Praefect | Check replicas
rake gitlab:seed:group_seed[subgroups_depth,username]           # Seed groups with sub-groups/projects/epics/milestones ...
rake gitlab:seed:issues                                         # GitLab | Seed | Seeds issues
rake gitlab:setup                                               # GitLab | Setup production application
rake gitlab:shell:build_missing_projects                        # GitLab | Shell | Build missing projects
rake gitlab:shell:install[repo]                                 # GitLab | Shell | Install or upgrade gitlab-shell
rake gitlab:shell:setup                                         # GitLab | Shell | Setup gitlab-shell
rake gitlab:sidekiq:check                                       # GitLab | Sidekiq | Check the configuration of Sidekiq
rake gitlab:snippets:list_non_migrated                          # GitLab | Show non migrated snippets
rake gitlab:snippets:migrate[ids]                               # GitLab | Migrate specific snippets to git
rake gitlab:snippets:migration_status                           # GitLab | Show whether there are snippet background mig...
rake gitlab:storage:hashed_attachments                          # Gitlab | Storage | Summary of project attachments usin...
rake gitlab:storage:hashed_projects                             # Gitlab | Storage | Summary of existing projects using ...
rake gitlab:storage:legacy_attachments                          # Gitlab | Storage | Summary of project attachments usin...
rake gitlab:storage:legacy_projects                             # Gitlab | Storage | Summary of existing projects using ...
rake gitlab:storage:list_hashed_attachments                     # Gitlab | Storage | List existing project attachments u...
rake gitlab:storage:list_hashed_projects                        # Gitlab | Storage | List existing projects using Hashed...
rake gitlab:storage:list_legacy_attachments                     # Gitlab | Storage | List existing project attachments u...
rake gitlab:storage:list_legacy_projects                        # Gitlab | Storage | List existing projects using Legacy...
rake gitlab:storage:migrate_to_hashed                           # GitLab | Storage | Migrate existing projects to Hashed...
rake gitlab:storage:rollback_to_legacy                          # GitLab | Storage | Rollback existing projects to Legac...
rake gitlab:tcp_check[host,port]                                # GitLab | Check TCP connectivity to a specific host and...
rake gitlab:test                                                # GitLab | Run all tests
rake gitlab:two_factor:disable_for_all_users                    # GitLab | 2FA | Disable Two-factor authentication (2FA)...
rake gitlab:two_factor:rotate_key:apply                         # GitLab | 2FA | Rotate Key | Encrypt user OTP secrets w...
rake gitlab:two_factor:rotate_key:rollback                      # GitLab | 2FA | Rotate Key | Rollback to secrets encryp...
rake gitlab:update_project_templates                            # GitLab | Update project templates
rake gitlab:update_templates                                    # GitLab | Update templates
rake gitlab:uploads:check                                       # GitLab | Uploads | Check integrity of uploaded files
rake gitlab:uploads:migrate                                     # GitLab | Uploads | Migrate the uploaded files of speci...
rake gitlab:uploads:migrate:all                                 # GitLab | Uploads | Migrate all uploaded files to objec...
rake gitlab:uploads:migrate_to_local                            # GitLab | Uploads | Migrate the uploaded files of speci...
rake gitlab:uploads:migrate_to_local:all                        # GitLab | Uploads | Migrate all uploaded files to local...
rake gitlab:uploads:sanitize:remove_exif                        # GitLab | Uploads | Remove EXIF from images
rake gitlab:usage_data:dump_sql_in_json                         # GitLab | UsageData | Generate raw SQLs for usage ping ...
rake gitlab:usage_data:dump_sql_in_yaml                         # GitLab | UsageData | Generate raw SQLs for usage ping ...
rake gitlab:web_hook:add                                        # GitLab | Webhook | Adds a webhook to the projects
rake gitlab:web_hook:list                                       # GitLab | Webhook | List webhooks
rake gitlab:web_hook:rm                                         # GitLab | Webhook | Remove a webhook from the projects
rake gitlab:workhorse:install[dir,repo]                         # GitLab | Workhorse | Install or upgrade gitlab-workhorse
rake gitlab:x509:update_signatures                              # GitLab | X509 | Update signatures when certificate sto...
rake grape:path_helpers                                         # Print route helper methods
rake grape:routes                                               # Print compiled grape routes
rake hipchat:send[message]                                      # Sends a HipChat message as a particular user
rake import:github[token,gitlab_username,project_path]          # GitLab | Import | Import a GitHub project - Example: i...
rake jira:generate_consumer_key                                 # Generate a consumer key for your application
rake jira:generate_public_cert                                  # Run the system call to generate a RSA public certificate
rake log:clear                                                  # Truncates all/specified *.log files in log/ to zero by...
rake metrics:setup_common_metrics                               # GitLab | Metrics | Setup common metrics
rake middleware                                                 # Prints out your Rack middleware stack
rake migrate_iids                                               # GitLab | Build internal ids for issues and merge requests
rake postgresql_md5_hash                                        # GitLab | Generate PostgreSQL Password Hash
rake raven:test[dsn]                                            # Send a test event to the remote Sentry server
rake restart                                                    # Restart app by touching tmp/restart.txt
rake secret                                                     # Generate a cryptographically secure secret key (this i...
rake setup                                                      # GitLab | Setup gitlab db
rake stats                                                      # Report code statistics (KLOCs, etc) from the applicati...
rake test                                                       # GitLab | Run all tests
rake test:db                                                    # Run tests quickly, but also reset db
rake test:system                                                # Run system tests only
rake time:zones[country_or_offset]                              # List all time zones, list by two-letter country code (...
rake tmp:clear                                                  # Clear cache, socket and screenshot files from tmp/ (na...
rake tmp:create                                                 # Creates tmp directories for cache, sockets, and pids
rake tokens:reset_all_email                                     # Reset all GitLab incoming email tokens
rake tokens:reset_all_feed                                      # Reset all GitLab feed tokens
rake yarn                                                       # Install Node dependencies with Yarn
rake yarn:available                                             # Ensure Yarn is installed
rake yarn:check                                                 # Ensure Node dependencies are installed
rake yarn:clobber                                               # Remove Node dependencies
rake yarn:install                                               # Install Node dependencies with Yarn / Install all Java...
rake zeitwerk:check                                             # Checks project structure for Zeitwerk compatibility

 

Posted by Uli Köhler in Ruby

How to use child_process.exec in Koa (async/await)

First install the child-process-promise library

npm i --save child-process-promise

Then you can use it like this:

const router = require('koa-router')();
const {exec} = require('child-process-promise');

router.get('/test', async ctx => {
  const [stdout, stderr] = await exec('python myscript.py')
  const ipv6 = stdout.toString();
  ctx.body = ipv6;
});

 

Posted by Uli Köhler in Javascript

How to format IPv6 address in groups of 32 bits in Python

def format_ipv6addr_group2(addr):
    """
    Format IPv6 addresses in 4 \n-separated groups of 32 bits
    Returns a string
    """
    addr_s = str(addr.exploded)
    return f"{addr_s[:10]}\n{addr_s[10:20]}\n{addr_s[20:30]}\n{addr_s[30:40]}"

Usage example:

addr = ipaddress.IPv6Address("fd66:6cbb:8c10:1234:4567:89ab:cdef:0123")
print(format_ipv6addr_group2(addr))

Output:

fd66:6cbb:
8c10:1234:
4567:89ab:
cdef:0123

 

Posted by Uli Köhler in Python