Technologies

Traefik API config example using Wildcard certificates and HTTP username/password (basic auth)

Based on our previous post Simple Traefik docker-compose setup with Lets Encrypt Cloudflare DNS-01 & TLS-ALPN-01 & HTTP-01 challenges this is the TOML config I use to allow access to the Traefik dashboard using HTTPS and my Let’s encrypt wildcard certificate. In this example, it will be reachable under traefik.mydomain.com. Place the config in /opt/traefik/conf/api.toml assuming you have setup Traefik based on our example.

[http.routers.traefik-api]
rule = "Host(`traefik.mydomain.com`)"
service = "[email protected]"
middlewares = ["auth"]
[http.routers.traefik-api.tls]
certresolver = "cloudflare"
[[http.routers.traefik-api.tls.domains]]
main = "mydomain.com"
sans = ["*.mydomain.com"]

[http.middlewares.auth.basicAuth]
users = [
  "admin:$apr1$ocvmQb0w$Bwlbz3V2VVRZlcu46X0zK0",
]

Create a new password string using htpasswd:

htpasswd -n admin

When prompted, enter the password and then copy the password such as admin:$apr1$ocvmQb0w$Bwlbz3V2VVRZlcu46X0zK0 to the [http.middlewares.auth.basicAuth] section:

[http.middlewares.auth.basicAuth]
users = [
  "COPY IT HERE"
]

It should look like our full example above.

Posted by Uli Köhler in Traefik

Traefik docker container labels for wildcard certificate using Cloudflare

See our previous post Simple Traefik docker-compose setup with Lets Encrypt Cloudflare DNS-01 & TLS-ALPN-01 & HTTP-01 challenges for the general config we’re using to deploy Traefik.

If you want to automatically connect Traefik to your docker container, use labels like

   labels:
     - "traefik.enable=true"
     - "traefik.http.routers.homeassistant.rule=Host(`homeassistant.mydomain.com`)"
     - "traefik.http.routers.homeassistant.entrypoints=websecure"
     - "traefik.http.routers.homeassistant.tls.certresolver=cloudflare"
     - "traefik.http.routers.homeassistant.tls.domains[0].main=mydomain.com"
     - "traefik.http.routers.homeassistant.tls.domains[0].sans=*.mydomain.com"
     - "traefik.http.services.homeassistant.loadbalancer.server.port=80"

 

Posted by Uli Köhler in Traefik

How to fix bellows Error: Invalid value for “-D” / “–database”: File “/home/user/.config/bellows/app.db” does not exist.

Problem:

When running

bellows devices

you see this error message:

Error: Invalid value for "-D" / "--database": File "/home/uli/.config/bellows/app.db" does not exist.

Solution:

Just create the directory and create an empty file for the database:

mkdir -p ~/.config/bellows
touch ~/.config/bellows/app.db

Then retry running your original command like

bellows devices
Posted by Uli Köhler in Python, Zigbee

Traefik wildcard Lets Encrypt certificate reverse proxy example

The following example builds on our config from Simple Traefik docker-compose setup with Lets Encrypt Cloudflare DNS-01 & TLS-ALPN-01 & HTTP-01 challenges

This config (placed in /etc/traefik/conf/myservice.toml – which is mapped to ./conf/myservice.toml i.e. /opt/traefik/conf/myservice.toml in our docker-compose example) generates a wildcard certificate for *.mydomain.com (also including just mydomain.com) using the cloudflare certificate provider and uses said wildcard certificate for myservice.mydomain.com and any other *.mydomain.com backends you have configured.

This config will reverse proxy all traffic on myservice.mydomain.com to 192.168.178.233:8080

# Host
[http.routers.myservice]
rule = "Host(`myservice.mydomain.com`)"
service = "myservice"

# Backend
[http.services]
[http.services.myservice.loadBalancer]
[[http.services.myservice.loadBalancer.servers]]
url = "http://192.168.178.233:8080/"

# Certificates
[http.routers.myservice.tls]
certresolver = "cloudflare"
[[http.routers.myservice.tls.domains]]
main = "mydomain.com"
sans = ["*.mydomain.com"]

Note that cloudflare in certresolver = "cloudflare" refers to the provider configured using

--certificatesresolvers.cloudflare....

but you can choose any other name with the cloudflare method such as --certificatesresolvers.myprovider.acme.dnschallenge.provider=cloudflare in which case the provider will be referred to as myprovider !

Posted by Uli Köhler in Networking, Traefik

Simple Traefik docker-compose setup with Lets Encrypt Cloudflare DNS-01 & TLS-ALPN-01 & HTTP-01 challenges

This is my setup using docker-compose to start Traefik, supporting all major encryption providers. I recommend to create the /opt/traefikdirectory and save the following file as /opt/traefik/docker-compose.yml. This config has the fileand docker providers enabled by default.

version: "3.3"
services:
  traefik:
    image: "traefik:v2.5"
    network_mode: "host" 
    command:
      - "--api.insecure=true"
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--providers.file.directory=/etc/traefik/conf"
      - "--providers.file.watch=true"
      - "--entrypoints.web.address=:80"
      - "--entryPoints.web.http.redirections.entryPoint.to=websecure"
      - "--entryPoints.web.http.redirections.entryPoint.scheme=https"
      - "--entrypoints.websecure.address=:443"
      - "--log.level=info"
      - "--serversTransport.insecureSkipVerify=true"
#      - "--pilot.token=PILOT_TOKEN_HERE"
#
      - "--certificatesresolvers.cloudflare.acme.dnschallenge=true"
      - "--certificatesresolvers.cloudflare.acme.dnschallenge.provider=cloudflare"
      - "--certificatesresolvers.cloudflare.acme.dnschallenge.resolvers=1.1.1.1:53,1.0.0.1:53"
      - "--certificatesresolvers.cloudflare.acme.caserver=https://acme-v02.api.letsencrypt.org/directory"
      - "[email protected]in.com"
      - "--certificatesresolvers.cloudflare.acme.KeyType=EC256"
      - "--certificatesresolvers.cloudflare.acme.storage=/letsencrypt/acme.json"
#
      - "--certificatesresolvers.cloudflare-staging.acme.dnschallenge=true"
      - "--certificatesresolvers.cloudflare-staging.acme.dnschallenge.provider=cloudflare"
      - "--certificatesresolvers.cloudflare-staging.acme.dnschallenge.resolvers=1.1.1.1:53,1.0.0.1:53"
      - "--certificatesresolvers.cloudflare-staging.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory"
      - "--[email protected]mydomain.com"
      - "--certificatesresolvers.cloudflare-staging.acme.KeyType=EC256"
      - "--certificatesresolvers.cloudflare-staging.acme.storage=/letsencrypt/acme.json"
#
      - "--certificatesresolvers.alpn.acme.tlsChallenge=true"
      - "--certificatesresolvers.alpn.acme.caserver=https://acme-v02.api.letsencrypt.org/directory"
      - "[email protected]"
      - "--certificatesresolvers.alpn.acme.KeyType=EC256"
      - "--certificatesresolvers.alpn.acme.storage=/letsencrypt/acme.json"
#
      - "--certificatesresolvers.alpn-staging.acme.tlsChallenge=true"
      - "--certificatesresolvers.alpn-staging.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory"
      - "[email protected]main.com"
      - "--certificatesresolvers.alpn-staging.acme.KeyType=EC256"
      - "--certificatesresolvers.alpn-staging.acme.storage=/letsencrypt/acme.json"
    environment:
      - [email protected]
      - CLOUDFLARE_API_KEY=XYZABC123
    volumes:
      - "./letsencrypt:/letsencrypt"
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
      - "./conf:/etc/traefik/conf:ro"

Replace [email protected] by the Email address to register certificates to. Also ensure to change

- [email protected]
- CLOUDFLARE_API_KEY=XYZABC123

Optionally, create a Pilot token and set it (don’t forget to un-comment the line) using

# - "--pilot.token=PILOT_TOKEN_HERE"

Now let’s make the service autostart on boot (and start it right now) using the method detailed in docker-compose systemd .service generator: Run the following in /opt/traefik

curl -fsSL https://techoverflow.net/scripts/create-docker-compose-service.sh | sudo bash /dev/stdin

We will detail how to get access to the Traefik API in followup posts.

 

 

Posted by Uli Köhler in Networking, Traefik

How to fix Oracle cloud Create Instance: Can’t select Ubuntu image

When I was trying to create a Oracle Cloud instance using Chrome, when I selected the Ubuntu image, the select image dialog closed and the image did not get selected.

However, the solution is simple: Use Firefox to create the instance. After that, you can continue to work with the instance using Chrome.

Posted by Uli Köhler in Cloud

Simple XenOrchestra setup using docker-compose

Create a directory such as /opt/xenorchestra and create docker-compose.yml:

version: '3'
services:
    xen-orchestra:
        restart: unless-stopped
        image: ronivay/xen-orchestra:latest
        container_name: xen-orchestra
        network_mode: host
        stop_grace_period: 1m
        environment:
          - HTTP_PORT=1780
        cap_add:
          - SYS_ADMIN
        security_opt:
          - apparmor:unconfined
        volumes:
          - ./xo-data:/var/lib/xo-server
          - ./redis-data:/var/lib/redis

You can choose any HTTP port you want using HTTP_PORT=1780. In this case, we opted for using network_mode: host to bypass the docker networking, since XenOrchestra seems to work better with full network access instead of the container having an own IP.

Now you can use our script from Create a systemd service for your docker-compose project in 10 seconds to automatically start XenOrchestra on boot (and start it immediately):

curl -fsSL https://techoverflow.net/scripts/create-docker-compose-service.sh | sudo bash /dev/stdin

Now access https://<IP of controller>:1780 (or your custom HTTP port) to get started with the XO setup.

Posted by Uli Köhler in Docker, Virtualization

How to get page HTML source code in Puppeteer

In order to get the current page HTML source code (i.e. not the source code received from the server, but the currently loaded source code including Javascript modifications), use

await page.content()

Full example based on Puppeteer minimal example:

// Minimal puppeteer get page HTML source code example
const puppeteer = require('puppeteer');
(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();
  await page.goto('https://techoverflow.net', {waitUntil: 'domcontentloaded'});
  // Wait for 5 seconds
  console.log(await page.content());
  // Take screenshot
  await browser.close();
})();

 

Posted by Uli Köhler in Javascript, NodeJS, Puppeteer

How to sleep for X seconds in Puppeteer

In order to sleep for 5 seconds, use

await page.waitForTimeout(5000);

Full example based on Puppeteer minimal example:

// Minimal puppeteer example
const puppeteer = require('puppeteer');
(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();
  await page.goto('https://techoverflow.net', {waitUntil: 'domcontentloaded'});
  // Wait for 5 seconds
  await page.waitForTimeout(5000);
  // Take screenshot
  await page.screenshot({path: 'screenshot.png'});
  await browser.close();
})();

 

Posted by Uli Köhler in Javascript, NodeJS, Puppeteer

How to optimize MySQL/MariaDB tables in docker-compose

If your MariaDB / MySQL root password is stored in .env , use this command:

source .env && docker-compose exec mariadb mysqlcheck -uroot -p$MARIADB_ROOT_PASSWORD --auto-repair --optimize --all-databases

You can also directly use the root password in the command:

docker-compose exec mariadb mysqlcheck -uroot -phoox8AiFahuniPaivatoh2iexighee --auto-repair --optimize --all-databases

 

Posted by Uli Köhler in Container, Databases, Docker

What image / distribution should you use on Oracle Cloud?

I recommend to always use either CentOS or Ubuntu. This way you will be compatible with other hosting providers and cloud vendors. Additionally, there is much more documentation regarding Ubuntu out there than for distributions like Oracle Linux.

Posted by Uli Köhler in Cloud

Oracle Cloud free tier VM.Standard.E2.1.Micro /proc/cpuinfo

processor       : 0
vendor_id       : AuthenticAMD
cpu family      : 23
model           : 1
model name      : AMD EPYC 7551 32-Core Processor
stepping        : 2
microcode       : 0x1000065
cpu MHz         : 1996.246
cache size      : 512 KB
physical id     : 0
siblings        : 2
core id         : 0
cpu cores       : 1
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm rep_good nopl cpuid extd_apicid tsc_known_freq pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd ibpb vmmcall fsgsbase tsc_adjust bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 xsaves clzero xsaveerptr virt_ssbd arat arch_capabilities
bugs            : sysret_ss_attrs null_seg spectre_v1 spectre_v2 spec_store_bypass
bogomips        : 3992.49
TLB size        : 1024 4K pages
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

processor       : 1
vendor_id       : AuthenticAMD
cpu family      : 23
model           : 1
model name      : AMD EPYC 7551 32-Core Processor
stepping        : 2
microcode       : 0x1000065
cpu MHz         : 1996.246
cache size      : 512 KB
physical id     : 0
siblings        : 2
core id         : 0
cpu cores       : 1
apicid          : 1
initial apicid  : 1
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm rep_good nopl cpuid extd_apicid tsc_known_freq pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd ibpb vmmcall fsgsbase tsc_adjust bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 xsaves clzero xsaveerptr virt_ssbd arat arch_capabilities
bugs            : sysret_ss_attrs null_seg spectre_v1 spectre_v2 spec_store_bypass
bogomips        : 3992.49
TLB size        : 1024 4K pages
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

 

Posted by Uli Köhler in Cloud, Networking

How to disable virtual cloud network firewall on Oracle Cloud

When running VM instances on Oracle Cloud, you might want to use all ports, not just the few ports that are open by default. This post shows how to disable the Virtual Cloud Network firewall altogether. Additionally, you need to configure the instance firewall e.g. via SSH. For Ubuntu, see How to disable instance firewall on Ubuntu on Oracle Cloud.

First login to the cloud network dashboard at https://cloud.oracle.com/networking/vcns

Now click the virtual cloud network:

Now click Security lists on the bottom left:

 

Click the Default security list (which has been automatically created)

Click Add Ingress Rules

Enter source 0.0.0.0/0 (i.e. all IP addresses) and IP protocol All protocols:

Now click Save changes and don’t forget to configure your instance firewall

Posted by Uli Köhler in Cloud, Networking

How to disable instance firewall on Ubuntu on Oracle Cloud

Note: This describes how to disable the firewall on an Ubuntu instance. You additionally need to configure the cloud network security list! See How to disable virtual cloud network firewall on Oracle Cloud for details!

The Oracle firewall is iptables based. We can disable the Ubuntu instance firewall using

sudo iptables -F
sudo netfilter-persistent save

Explanation:

  • iptables -F: Flush (remove all) iptables rules
  • netfilter-persistent save Save empty ruleset to disk so it will be reloaded on reboot.
Posted by Uli Köhler in Cloud, Networking

How to fix Nextcloud 4047 InnoDB refuses to write tables with ROW_FORMAT=COMPRESSED or KEY_BLOCK_SIZE.

Problem:

When trying to maintenance:repair your Nextcloud instance, e.g. during a utf8mb4 upgrade, you see an error message like

ERROR: An exception occurred while executing a query: SQLSTATE[HY000]: General error: 4047 InnoDB refuses to write tables with ROW_FORMAT=COMPRESSED or KEY_BLOCK_SIZE.

Solution:

You need to turn off innodb-read-only-compressed. I do this by starting my MySQL docker with

--skip-innodb-read-only-compressed

Full command line which worked for me:

--transaction-isolation=READ-COMMITTED --binlog-format=ROW --innodb-file-per-table=1 --skip-innodb-read-only-compressed

Full docker-compose.yml nextcloud section:

nextcloud-db:
  image: mariadb
  command: --transaction-isolation=READ-COMMITTED --binlog-format=ROW --innodb-file-per-table=1 --skip-innodb-read-only-compressed
  restart: always
  volumes:
    - ./nextcloud-db:/var/lib/mysql
  environment:
    - MYSQL_ROOT_PASSWORD=${MYSQL_ROOT_PASSWORD}
    - MYSQL_PASSWORD=${MYSQL_PASSWORD}

 

Posted by Uli Köhler in Cloud, Databases, Technologies

How to fix elasticsearch.exceptions.RequestError: RequestError(400, ‘resource_already_exists_exception’, ‘index […] already exists’) in Python

Problem:

You want to create an ElasticSearch index in Python using code like

es.indices.create("nodes") # Create an index names "nodes"

but you see the following error message:

Traceback (most recent call last):
  File "estest.py", line 22, in <module>
    es.indices.create("nodes")
  File "/usr/local/lib/python3.8/dist-packages/elasticsearch/client/utils.py", line 168, in _wrapped
    return func(*args, params=params, headers=headers, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/elasticsearch/client/indices.py", line 123, in create
    return self.transport.perform_request(
  File "/usr/local/lib/python3.8/dist-packages/elasticsearch/transport.py", line 415, in perform_request
    raise e
  File "/usr/local/lib/python3.8/dist-packages/elasticsearch/transport.py", line 381, in perform_request
    status, headers_response, data = connection.perform_request(
  File "/usr/local/lib/python3.8/dist-packages/elasticsearch/connection/http_urllib3.py", line 277, in perform_request
    self._raise_error(response.status, raw_data)
  File "/usr/local/lib/python3.8/dist-packages/elasticsearch/connection/base.py", line 330, in _raise_error
    raise HTTP_EXCEPTIONS.get(status_code, TransportError)(
elasticsearch.exceptions.RequestError: RequestError(400, 'resource_already_exists_exception', 'index [nodes/mXAiBt0wTKK4Y31HpshVbw] already exists')

Solution:

The error message tells you that the index you are trying to create already exists!

The simples solution is to use the code from our post on How to create ElasticSearch index if it doesn’t already exist in Python:

def es_create_index_if_not_exists(es, index):
    """Create the given ElasticSearch index and ignore error if it already exists"""
    try:
        es.indices.create(index)
    except elasticsearch.exceptions.RequestError as ex:
        if ex.error == 'resource_already_exists_exception':
            pass # Index already exists. Ignore.
        else: # Other exception - raise it
            raise ex

and use that function to create your index:

es_create_index_if_not_exists(es, "nodes") # Creates the "nodes" index ; doesn't fail if it already exists

 

Posted by Uli Köhler in Databases, ElasticSearch, Python

How to create ElasticSearch index if it doesn’t already exist in Python

The following utility will create an index if it doesn’t exist already by ignoring any resource_already_exists_exception

def es_create_index_if_not_exists(es, index):
    """Create the given ElasticSearch index and ignore error if it already exists"""
    try:
        es.indices.create(index)
    except elasticsearch.exceptions.RequestError as ex:
        if ex.error == 'resource_already_exists_exception':
            pass # Index already exists. Ignore.
        else: # Other exception - raise it
            raise ex

# Example usage: Create "nodes" index
es_create_index_if_not_exists(es, "nodes")

 

Posted by Uli Köhler in Databases, ElasticSearch, Python

What is the ElasticSearch equivalent to an SQL table?

In ElasticSearch, the concept which closely resembles an SQL table is called an index.

Compared to an SQL table, the index does not neccessarily need to have a predefined structure – the ElasticSearch index is more similar to a MongoDB collection.

However, an elasticsearch index has many features similar to SQL tables such as indices (which you typically don’t need to create explicity – ElasticSearch takes care of that for you).

Typically, indices contain lots of similar documents that have (mostly) the same properties.

Posted by Uli Köhler in Databases, ElasticSearch

How to fix Elasticsearch Python elasticsearch.exceptions.NotFoundError: NotFoundError(404, ‘index_not_found_exception’, ‘no such index [node]’, node, index_or_alias)

Problem:

You want to update ElasticSearch index settings in Python using code like

es.indices.put_settings(index="node", body={
    "index.mapping.total_fields.limit": 100000
})

but you see an error message like this one:

Traceback (most recent call last):
  File "estest.py", line 11, in <module>
    es.indices.put_settings(index="node", body={
  File "/usr/local/lib/python3.8/dist-packages/elasticsearch/client/utils.py", line 168, in _wrapped
    return func(*args, params=params, headers=headers, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/elasticsearch/client/indices.py", line 786, in put_settings
    return self.transport.perform_request(
  File "/usr/local/lib/python3.8/dist-packages/elasticsearch/transport.py", line 415, in perform_request
    raise e
  File "/usr/local/lib/python3.8/dist-packages/elasticsearch/transport.py", line 381, in perform_request
    status, headers_response, data = connection.perform_request(
  File "/usr/local/lib/python3.8/dist-packages/elasticsearch/connection/http_urllib3.py", line 277, in perform_request
    self._raise_error(response.status, raw_data)
  File "/usr/local/lib/python3.8/dist-packages/elasticsearch/connection/base.py", line 330, in _raise_error
    raise HTTP_EXCEPTIONS.get(status_code, TransportError)(
elasticsearch.exceptions.NotFoundError: NotFoundError(404, 'index_not_found_exception', 'no such index [node]', node, index_or_alias)

Solution:

The index needs to be created first in order to be able to put settings. First, double-check if you spelled the index name correctly! You can see the index name in the exception: no such index [node] means that the index is called node

The direct way to create an index is

es.indices.create("node")

But note that this will fail if the index already exists. In order to work around this issue, I recommend to use the code from our previous post How to create ElasticSearch index if it doesn’t already exist in Python:

def es_create_index_if_not_exists(es, index):
    """Create the given ElasticSearch index and ignore error if it already exists"""
    try:
        es.indices.create(index)
    except elasticsearch.exceptions.RequestError as ex:
        if ex.error == 'resource_already_exists_exception':
            pass # Index already exists. Ignore.
        else: # Other exception - raise it
            raise exnodes

and use the es_create_index_if_not_exists()  function to create your index:

es_create_index_if_not_exists(es, "node") # Creates the "node" index ; doesn't fail if it already exists

 

Posted by Uli Köhler in Databases, ElasticSearch, Python

How to set index setting using Python ElasticSearch client

You can set index settings using the official ElasticSearch python client library by using:

es.indices.put_settings(index="my-index", body={
    # Put your index settings here
    # Example: "index.mapping.total_fields.limit": 100000
})

Full example:

from elasticsearch import Elasticsearch

es = Elasticsearch()

es.indices.put_settings(index="ways", body={
    "index.mapping.total_fields.limit": 100000
})
Posted by Uli Köhler in Databases, ElasticSearch, Python