Technologies

Traefik container labels for the Unifi controller via docker-compose

For the basic configuration & setup of the Unifi controller via docker-compose, see Simple Unifi controller setup using docker-compose ! This post just covers the Traefik label part.

This setup is based on our previous post on the Unifi docker-compose setup. Furthermore, our traefik configuration is discussed in more detail in our post on Simple Traefik docker-compose setup with Lets Encrypt Cloudflare DNS-01 & TLS-ALPN-01 & HTTP-01 challenges.

For this example, we’ll use a wildcart Let’s Encrypt certificate for the domain *.mydomain.com via the Traefik certificate provider named cloudflare, with the Unifi controller running on unifi.mydomain.com

Here’s the container label config:

labels:
  - "traefik.enable=true"
  - "traefik.http.routers.unifi.rule=Host(`unifi.mydomain.com`)"
  - "traefik.http.routers.unifi.entrypoints=websecure"
  - "traefik.http.routers.unifi.tls.certresolver=cloudflare"
  - "traefik.http.routers.unifi.tls.domains[0].main=mydomain.com"
  - "traefik.http.routers.unifi.tls.domains[0].sans=*.mydomain.com"
  - "traefik.http.services.unifi.loadbalancer.server.port=8443"
  - "traefik.http.services.unifi.loadbalancer.server.scheme=https"

Note particularly these lines which make Traefik access the Unifi controller via HTTPS:

- "traefik.http.services.unifi.loadbalancer.server.port=8443"
- "traefik.http.services.unifi.loadbalancer.server.scheme=https"

Complete example

version: '2.3'
services:
  mongo_unifi:
    image: mongo:3.6
    network_mode: host
    restart: always
    volumes:
      - ./mongo_db:/data/db
      - ./mongo/dbcfg:/data/configdb
    command: mongod --port 29718
  controller:
    image: "jacobalberty/unifi:latest"
    depends_on:
      - mongo_unifi
    init: true
    network_mode: host
    restart: always
    volumes:
      - ./unifi_dir:/unifi
      - ./unifi_data:/unifi/data
      - ./unifi_log:/unifi/log
      - ./unifi_cert:/unifi/cert
      - ./unifi_init:/unifi/init.d
      - ./unifi_run:/var/run/unifi
      - ./unifi_backup:/unifi/data/backup
#    sysctls:
#      net.ipv4.ip_unprivileged_port_start: 0
    environment:
      - DB_URI=mongodb://localhost:29718/unifi
      - STATDB_URI=mongodb://localhost:29718/unifi_stat
      - DB_NAME=unifi
      - UNIFI_HTTP_PORT=8090
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.unifi.rule=Host(`unifi.mydomain.com`)"
      - "traefik.http.routers.unifi.entrypoints=websecure"
      - "traefik.http.routers.unifi.tls.certresolver=cloudflare"
      - "traefik.http.routers.unifi.tls.domains[0].main=mydomain.com"
      - "traefik.http.routers.unifi.tls.domains[0].sans=*.mydomain.com"
      - "traefik.http.services.unifi.loadbalancer.server.port=8443"
      - "traefik.http.services.unifi.loadbalancer.server.scheme=https"
# Ports commentet out since network mode is set to "host"
#    ports:
#      - "3478:3478/udp" # STUN
#      - "6789:6789/tcp" # Speed test
#      - "8080:8080/tcp" # Device/ controller comm.
#      - "8443:8443/tcp" # Controller GUI/API as seen in a web browser
#      - "8880:8880/tcp" # HTTP portal redirection
#      - "8843:8843/tcp" # HTTPS portal redirection
#      - "10001:10001/udp" # AP discovery
  logs:
    image: bash
    depends_on:
      - controller
    command: bash -c 'tail -F /unifi/log/*.log'
    restart: always
    volumes:
      - ./unifi_log:/unifi/log
Posted by Uli Köhler in Networking, Traefik

How to disable XCP-NG Windows Update PCIe device on the command line

This post shows you how to disable the XCP-NG windows update device on the command line. This prevents automatic installation of the Citrix drivers, enabling manual install of a custom version.

Note that you can easily disable the Windows update PCIe device in XenOrchestra using a single click, but not in XCP-NG center!

Prerequisite: Shut down the VM in question – usually you need to disable the device before installing Windows!

First, get the UUID of the VM usinjg

xe vm-list

which will output, for each virtual machine, something like:

uuid ( RO)           : 98002b8d-070f-9638-071c-be7e6c82f6a3
     name-label ( RW): CoreOS
    power-state ( RO): running

From that, copy the UUID such as 98002b8d-070f-9638-071c-be7e6c82f6a3.

Now run:

xe vm-param-set uuid=YOURUUID has-vendor-device=false

for example,

xe vm-param-set uuid=98002b8d-070f-9638-071c-be7e6c82f6a3 has-vendor-device=false

Now you can startup your VM with the driver installation PCIe device being disabled.

Posted by Uli Köhler in Networking, Virtualization

MikroTik webinterface reverse proxy using Traefik

The following Traefik .toml file which reverse proxies a MikroTik router’s WebFig webinterface is based on our Traefik setup from Simple Traefik docker-compose setup with Lets Encrypt Cloudflare DNS-01 & TLS-ALPN-01 & HTTP-01 challenges. It assumes that the MikroTik router is reachable at 10.1.2.3 via HTTP.

No Authentication beyond the MikroTik router’s WebFig internal authentication is performed. However – at least when using our Traefik config from our previous post it enforces HTTPS i.e. encrypted access.

Save the following file under config/mikrotik01.toml. Traefik will automatically reload, no restart will be required.

[http.routers.mikrotik01]
rule = "Host(`mikrotik01.mydomain.com`)"
service = "mikrotik01"

[http.routers.mikrotik01.tls]
certresolver = "cloudflare"

[[http.routers.mikrotik01.tls.domains]]
main = "mydomain.com"
sans = ["*.mydomain.com"]

[http.services]
[http.services.mikrotik01.loadBalancer]
[[http.services.mikrotik01.loadBalancer.servers]]
url = "http://10.1.2.3.4/"

 

Posted by Uli Köhler in MikroTik, Networking, Traefik

XenOrchestra docker-compose setup with Traefik labels

Based on Simple XenOrchestra setup using docker-compose, this extension of our config from that post features Traefik container labels. For the Traefik configuration, see for example our previous post Simple Traefik docker-compose setup with Lets Encrypt Cloudflare DNS-01 & TLS-ALPN-01 & HTTP-01 challenges

This setup uses a Wildcard certificate but you can also use a non-wildcard cert (e.g. if you don’t have access to the DNS for the DNS01 challenge) by just deleting both traefik.http.routers.xenorchestra.tls.domains... lines and selecting a suitable resolver.

version: '3'
services:
    xen-orchestra:
        restart: unless-stopped
        image: ronivay/xen-orchestra:latest
        container_name: xen-orchestra
        network_mode: host
        stop_grace_period: 1m
        environment:
            - HTTP_PORT=1780
        cap_add:
          - SYS_ADMIN
        security_opt:
          - apparmor:unconfined
        volumes:
          - ./xo-data:/var/lib/xo-server
          - ./redis-data:/var/lib/redis
        labels:
          - "traefik.enable=true"
          - "traefik.http.routers.xenorchestra.rule=Host(`xenorchestra.mydomain.com`)"
          - "traefik.http.routers.xenorchestra.entrypoints=websecure"
          - "traefik.http.routers.xenorchestra.tls.certresolver=cloudflare"
          - "traefik.http.routers.xenorchestra.tls.domains[0].main=mydomain.com"
          - "traefik.http.routers.xenorchestra.tls.domains[0].sans=*.mydomain.com"
          - "traefik.http.services.xenorchestra.loadbalancer.server.port=1780"

 

Posted by Uli Köhler in MikroTik, Networking, Virtualization

How to install tailscale on XCP-NG host

By installing tailscale on XCP-NG hosts, you can provide easier access to your virtualization host using VPN.

Run the following commands via SSH as root on the XCP-NG host:

sudo yum-config-manager --add-repo https://pkgs.tailscale.com/stable/centos/7/tailscale.repo
sudo yum -y install tailscale

and enable & start the tailscale daemon tailscaled:

systemctl enable --now tailscaled

 

Posted by Uli Köhler in Headscale, Networking, Virtualization, VPN

How to test if MongoDB database exists on command line (bash)

Use this command to test if a given MongoDB database exists:

mongo --quiet --eval 'db.getMongo().getDBNames().indexOf("mydb")'

This will return an index such as 0 or 241 if the database is found. On the other hand, it will return -1 if the database does not exist.

docker-compose version:

docker-compose exec mongodb mongo --quiet --eval 'db.getMongo().getDBNames().indexOf("mydb")'

where mongodb is the name of your container.

Now we can put it together in a bash script to test if the database exists:

# Query if DB exists in MongoDB
mongo_indexof_db=$(mongo --quiet --eval 'db.getMongo().getDBNames().indexOf("mydb")')
if [ $mongo_indexof_db -ne "-1" ]; then
    echo "MongoDB database exists"
else
    echo "MongoDB database does not exist"
fi

 

docker-compose variant:

# Query if DB exists in MongoDB
mongo_indexof_db=$(docker-compose -f inspect.yml exec -T mongodb mongo --quiet --eval 'db.getMongo().getDBNames().indexOf("mydb")')
if [ $mongo_indexof_db -ne "-1" ]; then
    echo "MongoDB database exists"
else
    echo "MongoDB database does not exist"
fi

 

Posted by Uli Köhler in MongoDB, Shell

How to fix Python MongoDB TypeError: Object of type ObjectId is not JSON serializable

Problem:

When trying to export data as JSON that has originally been queried from MongoDB using code like

with open("alle.json", "w") as outfile:
    json.dump(alle, outfile)

you see the following error message:

File /usr/lib/python3.9/json/__init__.py:179, in dump(obj, fp, skipkeys, ensure_ascii, check_circular, allow_nan, cls, indent, separators, default, sort_keys, **kw)
    173     iterable = cls(skipkeys=skipkeys, ensure_ascii=ensure_ascii,
    174         check_circular=check_circular, allow_nan=allow_nan, indent=indent,
    175         separators=separators,
    176         default=default, sort_keys=sort_keys, **kw).iterencode(obj)
    177 # could accelerate with writelines in some versions of Python, at
    178 # a debuggability cost
--> 179 for chunk in iterable:
    180     fp.write(chunk)

File /usr/lib/python3.9/json/encoder.py:429, in _make_iterencode.<locals>._iterencode(o, _current_indent_level)
    427     yield _floatstr(o)
    428 elif isinstance(o, (list, tuple)):
--> 429     yield from _iterencode_list(o, _current_indent_level)
    430 elif isinstance(o, dict):
    431     yield from _iterencode_dict(o, _current_indent_level)

File /usr/lib/python3.9/json/encoder.py:325, in _make_iterencode.<locals>._iterencode_list(lst, _current_indent_level)
    323         else:
    324             chunks = _iterencode(value, _current_indent_level)
--> 325         yield from chunks
    326 if newline_indent is not None:
    327     _current_indent_level -= 1

File /usr/lib/python3.9/json/encoder.py:405, in _make_iterencode.<locals>._iterencode_dict(dct, _current_indent_level)
    403         else:
    404             chunks = _iterencode(value, _current_indent_level)
--> 405         yield from chunks
    406 if newline_indent is not None:
    407     _current_indent_level -= 1

File /usr/lib/python3.9/json/encoder.py:438, in _make_iterencode.<locals>._iterencode(o, _current_indent_level)
    436         raise ValueError("Circular reference detected")
    437     markers[markerid] = o
--> 438 o = _default(o)
    439 yield from _iterencode(o, _current_indent_level)
    440 if markers is not None:

File /usr/lib/python3.9/json/encoder.py:179, in JSONEncoder.default(self, o)
    160 def default(self, o):
    161     """Implement this method in a subclass such that it returns
    162     a serializable object for ``o``, or calls the base implementation
    163     (to raise a ``TypeError``).
   (...)
    177 
    178     """
--> 179     raise TypeError(f'Object of type {o.__class__.__name__} '
    180                     f'is not JSON serializable')

TypeError: Object of type ObjectId is not JSON serializable

Solution:

This error occurs because objects queried from PyMongo always contain _id which is of type ObjectId and the normal JSON library (or drop-in replacements like simplejson do not know how to create JSON representations of Objects of type ObjectId).

In order to fix this, use pymongo‘s json_util instead of json. Note that the bson.json_util package contains dumps but does not contain dump, so use the following snippet to write to a file:

 

import bson.json_util as json_util

with open("alle.json", "w") as outfile:
    outfile.write(json_util.dumps(alle))

 

Posted by Uli Köhler in MongoDB, Python

How to install gcsfuse on Ubuntu in 15 seconds

export GCSFUSE_REPO=gcsfuse-`lsb_release -c -s`
echo "deb http://packages.cloud.google.com/apt $GCSFUSE_REPO main" | sudo tee /etc/apt/sources.list.d/gcsfuse.list
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
sudo apt-get update
sudo apt-get install gcsfuse

If that doesn’t work (such as for Ubuntu 21.10 impish at the time of writing this post), use the following method:

curl -L -O https://github.com/GoogleCloudPlatform/gcsfuse/releases/download/v0.39.2/gcsfuse_0.39.2_amd64.deb
sudo dpkg --install gcsfuse_0.39.2_amd64.deb
rm gcsfuse_0.39.2_amd64.deb

 

 

This is a summary from the official docs.

Posted by Uli Köhler in Cloud, Linux

How to fix gsutil 401 Anonymous caller does not have storage.objects.list access to the Google Cloud Storage bucket

Problem:

While running a command like

gsutil rsync my-folder gs://my-bucket

you see an error message like

Building synchronization state...
Caught non-retryable exception while listing gs://mfwh-backups/: ServiceException: 401 Anonymous caller does not have storage.objects.list access to the Google Cloud Storage bucket.
CommandException: Caught non-retryable exception - aborting rsync

Solution:

This error is basically telling you that you are not logged in !

First, create a service account for the project on Google cloud: Direct link to service account page. You need to figure out depending on your setup what roles you want to assign to the service account. If you are lost and don’t know what to select, just assign it admin rights on the storage, but be aware that this might have security implications, as this account may also delete or create storage buckets etc.

Then open the page for that service account and create a new key!

This will give you a JSON file such as my-project-4d267a915c4e.json. Save it on the server or computer where you want to run gsutil. I recommend to save it in ~ (the user’s home folder) with the original filename, for example ~/my-project-4d267a915c4e.json.

Then you need to activate that service account using

gcloud auth activate-service-account --key-file [path to JSON file]

such as

gcloud auth activate-service-account --key-file [path to JSON file]

 

Posted by Uli Köhler in Cloud

How to install gcloud on Ubuntu in 10 seconds

sudo snap install google-cloud-cli --classic

This is the summary from the official docs. I recommend to install it as snap package as opposed to a deb package since it will auto-update, it’s much easier to use and just works better out of the box in my experience.

Posted by Uli Köhler in Cloud, Linux

How to put text into input element in Pyppeteer

In Pyppeteer, if you have an input like this one:

<input id="myInput">

you can fill with text like abc123 by using page.type() like in this snippet:

await page.type('#myInput', 'abc123')

Full example

This example fetches techoverflow.net and puts my search into the search query input on the top right:

#!/usr/bin/env python3
import asyncio
from pyppeteer import launch

async def main():
    browser = await launch()
    page = await browser.newPage()
    await page.goto('https://www.techoverflow.net')
    # Fill text with input
    await page.type('.search-form-input', 'my search')
    # Make screenshot
    await page.screenshot({'path': 'screenshot.png'})
    # Cleanup
    await browser.close()

asyncio.get_event_loop().run_until_complete(main())

 

Posted by Uli Köhler in Pyppeteer, Python

How to disable SSL certificate verification in Pyppeteer

If you see an error message like

pyppeteer.errors.PageError: net::ERR_CERT_AUTHORITY_INVALID at https://10.9.5.12/

in Pyppeteer and you are sure that you just want to skip certificate verification change

browser = await launch()

to

browser = await launch({"ignoreHTTPSErrors": True})

or add "ignoreHTTPSErrors": True to the list of parameters to launch if you already have other parameters there. This will just ignore the net::ERR_CERT_AUTHORITY_INVALID and other, related HTTPS errors.

Posted by Uli Köhler in Pyppeteer, Python

Pyppeteer minimal screenshot example

This script is a minimal example on how to use Pyppeteer to fetch a web page and save a screenshot to screenshot.png:

#!/usr/bin/env python3
import asyncio
from pyppeteer import launch

async def main():
    browser = await launch()
    page = await browser.newPage()
    await page.goto('https://www.techoverflow.net')
    # Make screenshot
    await page.screenshot({'path': 'screenshot.png'})
    # Cleanup
    await browser.close()

asyncio.get_event_loop().run_until_complete(main())

How to run:

sudo pip3 install pyppeteer
python3 PyppeteerScreenshotExample.py

 

Posted by Uli Köhler in Pyppeteer

How to S3 concepts relate to standard filesystem concepts: Access keys, objects, …

The following mapping is often useful

  • Objects are essentially files
  • Object keys are filenames
  • An Access Key is essentially a username used for access to the S3 storage
  • A Secret Key is essentially a password for a given access key = username
  • Prefixes are folders
  • region is conceptually a fileserver although in practice it consists of multiple servers linked together
Posted by Uli Köhler in S3

How to sort files on S3 by timestamp in filename using boto3 & Python

Let’s assume we have backup objects in an S3 directory like:

production-backup-2022-03-29_14-40-16.xz
production-backup-2022-03-29_14-50-16.xz
production-backup-2022-03-29_15-00-03.xz
production-backup-2022-03-29_15-10-04.xz
production-backup-2022-03-29_15-20-06.xz
production-backup-2022-03-29_15-30-06.xz
production-backup-2022-03-29_15-40-00.xz
production-backup-2022-03-29_15-50-07.xz
production-backup-2022-03-29_16-00-06.xz
production-backup-2022-03-29_16-10-12.xz
production-backup-2022-03-29_16-20-18.xz
production-backup-2022-03-29_16-30-18.xz
production-backup-2022-03-29_16-40-00.xz
production-backup-2022-03-29_16-50-09.xz
production-backup-2022-03-29_17-00-18.xz
production-backup-2022-03-29_17-10-13.xz
production-backup-2022-03-29_17-20-18.xz
production-backup-2022-03-29_17-30-18.xz
production-backup-2022-03-29_17-40-06.xz
production-backup-2022-03-29_17-50-21.xz
production-backup-2022-03-29_18-00-06.xz

And we want to identify the newest one. Often in these situations, you can’t really rely on modification timestamps as these can change when syncing old files or when changing folder structures or names.

Hence the best way is to rely on the timestamp from the filename as a reference point. The date timestamp we’re using here is based on our post How to generate filename containing date & time on the command line ; if you’re using a different object key format, you might need to adjust the date_regex accordingly.

The following example script iterates all objects within a specific S3 folder, sorting them by the timestamp from the filename and choses the latest one, downloading it from S3 to the local filesystem.

This script is based on a few of our previous posts, including:

#!/usr/bin/env python3
import boto3
import re
import os.path
from collections import namedtuple
from datetime import datetime

# Create connection to Wasabi / S3
s3 = boto3.resource('s3',
    endpoint_url = 'https://minio.mydomain.com',
    aws_access_key_id = 'my-access-key',
    aws_secret_access_key = 'my-password'
)

# Get bucket object
backups = s3.Bucket('mfwh-backup')

date_regex = re.compile(r"(?P<year>\d{4})-(?P<month>\d{2})-(?P<day>\d{2})_(?P<hour>\d{2})-(?P<minute>\d{2})-(?P<second>\d{2})")

DatedObject =  namedtuple("DatedObject", ["Date", "Object"])
entries = []
# Iterate over objects in bucket
for obj in backups.objects.filter(Prefix="production/"):
    date_match = date_regex.search(obj.key)
    # Ignore other files (without date stamp) if any
    if date_match is None:
        continue
    dt = datetime(year=int(date_match.group("year")), month=int(date_match.group("month")),
        day=int(date_match.group("day")), hour=int(date_match.group("hour")), minute=int(date_match.group("minute")),
        second=int(date_match.group("second")))
    entries.append(DatedObject(dt, obj))
# Sort entries by date
entries.sort(key=lambda entry: entry.Date)

newest_date, newest_obj = entries[-1]
#print(f"Downloading {newest_obj.key} from {newest_date.isoformat()}")
filename = os.path.basename(newest_obj.key)

with open(filename, "wb") as outfile:
    backups.download_fileobj(newest_obj.key, outfile)

# Print filename for automation purposes
print(filename)
Posted by Uli Köhler in S3

How iterate all documents in MongoDB collection using pymongo

This example will connect to the MongoDB running at localhost (on the default port 27017) without any username or password and open the database named mydb (also see Python MongoDB minimal connect example using pymongo), open the collection mycollectionand iterate all the documents in said collection, printing each document.

from pymongo import MongoClient
client = MongoClient("mongodb://localhost")
db = client["mydb"]
mycollection = db["mycollection"]

for doc in mycollection.find():
    print(doc)

This will print, for example,

{'_id': 123, 'name': 'John', 'phone': '+123456789'}

 

Posted by Uli Köhler in Databases, MongoDB

How to list MongoDB collection names in Python using pymongo

This example will connect to the MongoDB running at localhost (on the default port 27017) without any username or password and open the database named mydb (also see Python MongoDB minimal connect example using pymongo) and list all the collection names in mydb:

from pymongo import MongoClient
client = MongoClient("mongodb://localhost")
db = client["mydb"]

print(db.list_collection_names())

This will print, for example,

['people', 'salaries']

 

Posted by Uli Köhler in Databases, MongoDB

Python MongoDB minimal connect example using pymongo

This example will connect to the MongoDB running at localhost (on the default port 27017) without any username or password and open the database named mydb

from pymongo import MongoClient
client = MongoClient("mongodb://localhost")
db = client["mydb"]

 

Posted by Uli Köhler in Databases, MongoDB

A working Traefik & docker-compose minio setup with console

The following config works by using two domains: minio.mydomain.com and console.minio.mydomain.com.

For the basic Traefik setup this is based on, see Simple Traefik docker-compose setup with Lets Encrypt Cloudflare DNS-01 & TLS-ALPN-01 & HTTP-01 challenges. Regarding this setup, the important part is to enabled the docker autodiscovery and defining the certificate resolve (we’re using the ALPN resolver).

Be sure to choose a random MINIO_ROOT_PASSWORD!

version: '3.5'
services:
   minio:
       image: quay.io/minio/minio:latest
       command: server --console-address ":9001" /data
       volumes:
          - ./data:/data
          - ./config:/root/.minio
       environment:
          - MINIO_ROOT_USER=minioadmin
          - MINIO_ROOT_PASSWORD=uikui5choRith0ZieV2zohN5aish5r
          - MINIO_DOMAIN=minio.mydomain.com
          - MINIO_SERVER_URL=https://minio.mydomain.com
          - MINIO_BROWSER_REDIRECT_URL=https://console.minio.mydomain.com
       labels:
          - "traefik.enable=true"
          # Console
          - "traefik.http.routers.minio-console.rule=Host(`console.minio.mydomain.com`)"
          - "traefik.http.routers.minio-console.entrypoints=websecure"
          - "traefik.http.routers.minio-console.tls.certresolver=alpn"
          - "traefik.http.routers.minio-console.service=minio-console"
          - "traefik.http.services.minio-console.loadbalancer.server.port=9001"
          # APi
          - "traefik.http.routers.minio.rule=Host(`minio.mydomain.com`)"
          - "traefik.http.routers.minio.entrypoints=websecure"
          - "traefik.http.routers.minio.tls.certresolver=alpn"
          - "traefik.http.routers.minio.service=minio"
          - "traefik.http.services.minio.loadbalancer.server.port=9000"

 

Posted by Uli Köhler in Container, Docker, S3, Traefik