VPN

Headscale docker-compose config for Traefik HTTPS reverse proxy

This config is based on our previous post How to setup headscale server in 5 minutes using docker-compose and our Traefik configuration with Cloudflare wildcard certs (see Simple Traefik docker-compose setup with Lets Encrypt Cloudflare DNS-01 & TLS-ALPN-01 & HTTP-01 challenges)

version: '3.5'
services:
  headscale:
    image: headscale/headscale:latest
    volumes:
      - ./config:/etc/headscale/
      - ./data:/var/lib/headscale
    ports:
      # - 27896:8080
      - 9090:9090
      - 3478:3478/udp
    command: headscale serve
    restart: unless-stopped
    depends_on:
      - postgres
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.headscale.rule=Host(`headscale.mydomain.com`)"
      - "traefik.http.routers.headscale.entrypoints=websecure"
      - "traefik.http.routers.headscale.tls.certresolver=cloudflare"
      - "traefik.http.routers.headscale.tls.domains[0].main=mydomain.com"
      - "traefik.http.routers.headscale.tls.domains[0].sans=*.mydomain.com"
      - "traefik.http.services.headscale.loadbalancer.server.port=8080"

  postgres:
    image: postgres:14
    restart: unless-stopped
    volumes:
      - ./pg_data:/var/lib/postgresql/data
    environment:
      - POSTGRES_PASSWORD=${POSTGRES_PASSWORD}
      - POSTGRES_DB=${POSTGRES_DB}
      - POSTGRES_USER=${POSTGRES_USER}

 

Posted by Uli Köhler in Headscale, Networking, VPN

How to re-resolve DNS for Wireguard on Alpine Linux

In our previous post How to automatically re-resolve DNS in Wireguard on Linux we explored how to use the reresolve-dns.sh script that is included with wireguard-tools on Ubuntu to re-resolve DNS entries in wireguard config files.

Add the following to the root crontab using crontab -e:

*       *       *       *       *       /usr/local/bin/reresolve-dns.sh /etc/wireguard/mywg.conf

You need to add this for every wireguard config (mywg.conf for this example).

Posted by Uli Köhler in Alpine Linux, Wireguard

How to autostart Wireguard (wg-quick) on boot on Alpine Linux

If you have a Wireguard config file such as /etc/wireguard/wghome.conf, which you can start manually using wg-quick up wghome, this is how you autostart it on boot. Thanks to Justin Ludwig on Serverfault for the template for that init script

Create /etc/init.d/wg-quick-wghome:

#!/sbin/openrc-run
description="wg-quick wghome"

depend() {
    need net
    need localmount
}

start() {
    wg-quick up schlaftier
}

stop() {
    wg-quick down schlaftier
}

Then make it executable:

chmod a+x /etc/init.d/wg-quick-wghome

and enable it to start on boot:

rc-update add wg-quick-wghome default

and start it right now if desired:

/etc/init.d/wg-quick-wghome start

Of course you can add multiple scripts like this. Just ensure to name them differently and perform all the steps required to enable startup on boot.

Posted by Uli Köhler in Alpine Linux, Networking, Wireguard

How to fix Alpine Linux wg-quick (no such package)

Problem:

While trying to install wg-quick on Alpine Linux, you see the following error message:

ERROR: unable to select packages:
  wg-quick (no such package):
    required by: world[wg-quick]

Solution:

You need to install the wireguard-tools package which also contains wg-quick:

apk add wireguard-tools

 

Posted by Uli Köhler in Alpine Linux, Wireguard

How to fix headscale startup error “failed to configure TLS (sslmode is invalid)”

Problem:

When starting headscale 0.17.0 or newer, you see repeating error messages like

headscale_1  | 2022-11-26T20:03:39Z FTL go/src/headscale/cmd/headscale/cli/server.go:21 > Error initializing error="cannot parse `host=postgres dbname=headscale user=headscale sslmode= port=5432 password=xxxxx`: failed to configure TLS (sslmode is invalid)"

Solution:

This error occurs due to a change starting from headscale 0.17.0. In your config.yaml, add

db_ssl: false

after db_type: ...

Complete database section config example:

# # Postgres config
db_type: postgres
db_host: postgres
db_port: 5432
db_name: headscale
db_user: headscale
db_pass: Paep6AhJiZeedie5fiefieV0quohro

db_ssl: false

After that, restart headscale and the error message should disappear.

Posted by Uli Köhler in Headscale

How to leave netmaker network

Once you have used netclient join to join a netmaker network, you can use

netclient leave --network mynetwork

to permanently leave the network named mynetwork (until you re-join of course).

 

Posted by Uli Köhler in Networking, VPN

Is tailscale available for OpenWRT 19.07?

No, tailscale can’t be installed using opkg on OpenWRT 19.xx. I have experimentally verified this using a MIPSBE router with OpenWRT 19.07.10.

However, tailscale is available on OpenWRT starting from version 21.02 – source: tailscale package page on OpenWRT.

Posted by Uli Köhler in Headscale, Networking, OpenWRT

How to install Netmaker client (netclient) on Ubuntu in 15 seconds

Just run these commands in your Shell in Ubuntu to install netclient

curl -sL 'https://apt.netmaker.org/gpg.key' | sudo tee /etc/apt/trusted.gpg.d/netclient.asc
curl -sL 'https://apt.netmaker.org/debian.deb.txt' | sudo tee /etc/apt/sources.list.d/netclient.list
sudo apt update
sudo apt install netclient

Source: Netmaker installation page

Posted by Uli Köhler in Networking, VPN, Wireguard

iperf3 benchmark of ZeroTier vs Netmaker vs Tailscale vs direct switched connection

In our setup, a virtual machine (running on an XCP-NG host) on  was connected to my Desktop (HP Z240, i7-6700 @3.4 GHz running Ubuntu 22.04) in a purely switched network with 1Gbit links. Both devices were connected using a MikroTik 10G switch (Marvell chip

I ran iperf3 -s on the VM and ran iperf3 -c [IP address] on the desktop. Reverse tests have not been performed.

Direct switched connection (no VPN)

Connecting to host 10.9.2.103, port 5201
[  5] local 10.9.2.10 port 56848 connected to 10.9.2.103 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  92.8 MBytes   779 Mbits/sec    0    444 KBytes       
[  5]   1.00-2.00   sec  90.7 MBytes   761 Mbits/sec    0    543 KBytes       
[  5]   2.00-3.00   sec  88.6 MBytes   743 Mbits/sec    0    816 KBytes       
[  5]   3.00-4.00   sec  90.0 MBytes   755 Mbits/sec    0    816 KBytes       
[  5]   4.00-5.00   sec  90.0 MBytes   755 Mbits/sec    0    856 KBytes       
[  5]   5.00-6.00   sec  88.8 MBytes   744 Mbits/sec    0    946 KBytes       
[  5]   6.00-7.00   sec  88.8 MBytes   745 Mbits/sec    0    946 KBytes       
[  5]   7.00-8.00   sec  90.0 MBytes   755 Mbits/sec    0    993 KBytes       
[  5]   8.00-9.00   sec  90.0 MBytes   755 Mbits/sec    0    993 KBytes       
[  5]   9.00-10.00  sec  88.8 MBytes   744 Mbits/sec    0    993 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   898 MBytes   754 Mbits/sec    0             sender
[  5]   0.00-10.01  sec   896 MBytes   751 Mbits/sec                  receiver

ZeroTier

Connecting to host 10.80.246.34, port 5201
[  5] local 10.80.246.38 port 35474 connected to 10.80.246.34 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  59.9 MBytes   503 Mbits/sec  338    102 KBytes       
[  5]   1.00-2.00   sec  60.2 MBytes   505 Mbits/sec  313    188 KBytes       
[  5]   2.00-3.00   sec  63.9 MBytes   536 Mbits/sec  176   99.3 KBytes       
[  5]   3.00-4.00   sec  74.3 MBytes   623 Mbits/sec  174    113 KBytes       
[  5]   4.00-5.00   sec  67.7 MBytes   568 Mbits/sec  197   83.2 KBytes       
[  5]   5.00-6.00   sec  72.5 MBytes   609 Mbits/sec  218    228 KBytes       
[  5]   6.00-7.00   sec  61.3 MBytes   514 Mbits/sec  281   77.8 KBytes       
[  5]   7.00-8.00   sec  72.0 MBytes   604 Mbits/sec  213   91.2 KBytes       
[  5]   8.00-9.00   sec  65.4 MBytes   549 Mbits/sec  309    156 KBytes       
[  5]   9.00-10.00  sec  53.9 MBytes   453 Mbits/sec  190    121 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   651 MBytes   546 Mbits/sec  2409             sender
[  5]   0.00-10.01  sec   650 MBytes   545 Mbits/sec                  receiver

NetMaker

Netmaker internally uses a normal (kernel-based) wireguard connection, so in some respect this is a test of Wireguard performance

Connecting to host 10.230.113.3, port 5201
[  5] local 10.230.113.1 port 35534 connected to 10.230.113.3 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   105 MBytes   881 Mbits/sec    0   1.01 MBytes       
[  5]   1.00-2.00   sec   104 MBytes   870 Mbits/sec   86    422 KBytes       
[  5]   2.00-3.00   sec   101 MBytes   849 Mbits/sec    0    488 KBytes       
[  5]   3.00-4.00   sec  98.8 MBytes   828 Mbits/sec    0    535 KBytes       
[  5]   4.00-5.00   sec  98.8 MBytes   828 Mbits/sec    0    584 KBytes       
[  5]   5.00-6.00   sec   104 MBytes   870 Mbits/sec    0    615 KBytes       
[  5]   6.00-7.00   sec  97.5 MBytes   818 Mbits/sec    7    472 KBytes       
[  5]   7.00-8.00   sec   104 MBytes   870 Mbits/sec    0    522 KBytes       
[  5]   8.00-9.00   sec   101 MBytes   849 Mbits/sec    0    580 KBytes       
[  5]   9.00-10.00  sec   102 MBytes   860 Mbits/sec    0    606 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1016 MBytes   852 Mbits/sec   93             sender
[  5]   0.00-10.00  sec  1014 MBytes   850 Mbits/sec                  receiver

 Tailscale

Tailscale 1.28.0 has been used for this test.

During this test, I ensured that the tailscale connection was established using the switched network and was not going through a DERP server or the routed network.

$ tailscale ping 100.64.0.3
pong from vm (fd5d:7b60:4742::3) via 10.9.2.103:41641 in 1ms

Results:

Connecting to host 100.64.0.3, port 5201
[  5] local 100.64.0.2 port 40690 connected to 100.64.0.3 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  38.3 MBytes   321 Mbits/sec  389   60.0 KBytes       
[  5]   1.00-2.00   sec  37.6 MBytes   315 Mbits/sec  366   43.2 KBytes       
[  5]   2.00-3.00   sec  36.7 MBytes   308 Mbits/sec  431   52.8 KBytes       
[  5]   3.00-4.00   sec  38.5 MBytes   323 Mbits/sec  488   80.3 KBytes       
[  5]   4.00-5.00   sec  29.3 MBytes   246 Mbits/sec  356   38.4 KBytes       
[  5]   5.00-6.00   sec  31.0 MBytes   260 Mbits/sec  351   86.3 KBytes       
[  5]   6.00-7.00   sec  27.1 MBytes   227 Mbits/sec  287   50.4 KBytes       
[  5]   7.00-8.00   sec  26.1 MBytes   219 Mbits/sec  210   46.8 KBytes       
[  5]   8.00-9.00   sec  27.1 MBytes   227 Mbits/sec  261   39.6 KBytes       
[  5]   9.00-10.00  sec  27.5 MBytes   231 Mbits/sec  222   40.8 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   319 MBytes   268 Mbits/sec  3361             sender
[  5]   0.00-10.01  sec   318 MBytes   267 Mbits/sec                  receiver

Summary

The approximate performance expectation in this specific scenario is:

  • Tailscale: 300 Mbit/s
  • ZeroTier: 550 Mbit/s
  • Netmaker: 850 Mbit/s
  • Direct switched network: 750 Mbit/s

Curiously, netmaker performed better than the direct connection. The reason for this is not known at this point, but a similar effect has been observed in this medium.com article.

Generally, one can see that Tailscale (which internally uses software wireguard) is approximately half the speed of ZeroTier, which in turn is outperformed significantly by Netmaker.

In a followup post I will describe advantages and disadvantages of those solutions and explore under which scenarios I would use the solutions.

Posted by Uli Köhler in Headscale, Networking, Wireguard, ZeroTier

How to install tailscale on XCP-NG host

By installing tailscale on XCP-NG hosts, you can provide easier access to your virtualization host using VPN.

Run the following commands via SSH as root on the XCP-NG host:

sudo yum-config-manager --add-repo https://pkgs.tailscale.com/stable/centos/7/tailscale.repo
sudo yum -y install tailscale

and enable & start the tailscale daemon tailscaled:

systemctl enable --now tailscaled

 

Posted by Uli Köhler in Headscale, Networking, Virtualization, VPN

How to install tailscale on Fedora CoreOS

In order to install tailscale, on Fedora CoreOS (this post has been tested on Fedora CoreOS 35), you can use this sequence of commands:

sudo curl -o /etc/yum.repos.d/tailscale.repo https://pkgs.tailscale.com/stable/fedora/tailscale.repo
sudo rpm-ostree install tailscale

Now reboot using

sudo systemctl reboot

Once rebooted, you can enable the service using

sudo systemctl enable --now tailscaled

and then configure tailscale as usual:

sudo tailscale up --login-server .... --authkey ...

Also see our post on How to connect tailscale to headscale server on Linux

Posted by Uli Köhler in CoreOS, Headscale, VPN

How to connect Synology NAS to Headscale

First, install the Tailscale App using the Synology Package manager. Don’t try to initialize using the UI since this will only work with the commercial tailscale service, not with headscale.

Then login to the NAS using SSH (I’m using the admin account) and run sudo su to run

You should see the following shell prompt:

ash-4.4#

Now you can initialize tailscale using the tailscale command similar to our previous post How to connect tailscale to headscale server on Linux. In my case, I needed to use the --reset flag in order for the command to work.

tailscale up --reset --login-server https://headscale.mydomain.com --authkey ... --accept-routes

This will login to your server just like the normal (non-synology) tailscale client does.

Posted by Uli Köhler in Headscale, VPN

ufw: How to allow traffic to all ports on specific interface

sudo ufw allow in on tailscale0 to any

This will allow any traffic (including routed traffic, if packet forwarding is enabled) coming from the tailscale0 interface.

Posted by Uli Köhler in Linux, Networking, VPN

How to install tailscale on Ubuntu

In order to instal tailscale, on any Ubuntu version, you can use the official tailscale install command:

sudo apt -y install curl apt-transport-https
curl -fsSL https://tailscale.com/install.sh | sh
Posted by Uli Köhler in Headscale, Raspberry Pi, VPN

How to install tailscale on Raspberry Pi

Just use the official install command from the tailscale website:

curl -fsSL https://tailscale.com/install.sh | sh

 

Posted by Uli Köhler in Headscale, Raspberry Pi, VPN

Headscale nginx reverse proxy config

Reverse proxying headscale using nginx is extremly simple. You don’t need any special config.

Here is my config with Let’s Encrypt enabled (part of the config is auto-generated using certbot --nginx).

Ensure to set the port (27896) to match the one mapped to the Headscale port.

server {
    server_name  headscale.mydomain.com;

    location / {
        proxy_pass http://localhost:27896/;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "Upgrade";
        proxy_redirect http:// https://;
        proxy_buffering off;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $http_x_forwarded_proto;
        add_header Strict-Transport-Security "max-age=15552000; includeSubDomains" always;
    }

    listen [::]:443 ssl; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/mydomain-wildcard/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/mydomain-wildcard/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot


}
server {
    if ($host = headscale.mydomain.com) {
        return 301 https://$host$request_uri;
    } # managed by Certbot

    server_name  headscale.mydomain.com;

    listen [::]:80; # managed by Certbot
    return 404; # managed by Certbot
}

The proxy_… upgrade stuff is for proxying websockets. I have no idea if it’s required or not because I have not tried, but it doesn’t really hurt to keep in in there.

Posted by Uli Köhler in Headscale, Networking, nginx

Headscale docker-compose config with PostgreSQL

This config is intended for larger installations than our sqlite based standard config. It tends to be slightly easier to back up correctly and will be faster for larger workloads. However, it will consume more RAM especially for low-workload installations and you have two docker containers to worry about during maintenance (though they are managed using a single docker-compose instance). I do not recommend using a shared postgres server although this is certainly possible.

First, create a random password using

echo POSTGRES_PASSWORD=$(pwgen 30 1) > .env

The docker-compose.yml looks like this:

version: '3.5'
services:
  headscale:
    image: headscale/headscale:latest
    volumes:
      - ./config:/etc/headscale/
      - ./data:/var/lib/headscale
    ports:
      - 27896:8080
    command: headscale serve
    restart: unless-stopped
    depends_on:
      - postgres
  postgres:
    image: postgres
    restart: unless-stopped
    volumes:
      - ./pg_data:/var/lib/postgresql/data
    environment:
      - POSTGRES_PASSWORD=${POSTGRES_PASSWORD}
      - POSTGRES_DB=headscale
      - POSTGRES_USER=headscale

 

Now we create the default headscale config:

mkdir -p ./config
curl https://raw.githubusercontent.com/juanfont/headscale/main/config-example.yaml -o ./config/config.yaml

In config/config.yaml we need to make these changes:

Set server URL:

server_url: https://headscale.mydomain.com

Comment out the sqlite database (add # to the front of every line):

# SQLite config
# db_type: sqlite3
# db_path: /var/lib/headscale/db.sqlite

And uncomment and configure postgres:

# Postgres config
db_type: postgres
db_host: postgres
db_port: 5432
db_name: headscale
db_user: headscale
db_pass: ohngooFaciice2hooGoo1Ahvif3ahl

Make sure all of these are uncommented and you copy the password from .env . It is extremely important that you use a unique password here to prevent attacks from unprivileged host processes to the docker containers.

My recommendation is to reverse proxy headscale using traefik or nginx instead of using the builtin Let’s Encrypt / ACME support. This will allow not only sharing the port & IP address with other services, standard services like Traefik and/or nginx are much more well tested regarding exposure to the internet and hence provide a potential security benefit. Additionally, they make it easier to manage certificates in a service-independent manner and provide an additional layer for debugging etc.

You might also configure custom IP address ranges:

ip_prefixes:
  - fd5d:7b60:4742::/48
  - 100.64.0.0/10

but this is optional.

For more info regarding autostart etc, see How to setup headscale server in 5 minutes using docker-compose

Posted by Uli Köhler in Headscale, Networking, VPN

How to create namespace on headscale server

Currently you need to create a namespace using the command line.

If running without a container

headscale namespaces create -n mynamespace

If running using docker-compose:

docker-compose exec headscale headscale namespaces create mynamespace

If successful, this will show

Namespace created
Posted by Uli Köhler in Headscale, Networking, VPN

How to list headscale namespaces

To list namespaces (which are comparable to accounts) in headscale, run

headscale namespaces list

If you are using headscale using docker-compose, use e.g.

docker-compose exec headscale headscale namespaces list

Example output

ID | Name | Created            
1  | uli  | 2022-01-16 19:54:55
Posted by Uli Köhler in Headscale, Networking, VPN

How to connect tailscale to headscale server on Linux

Also see our guide on How to setup headscale server in 5 minutes using docker-compose

Assuming you are running your headscale server at https://headscale.mydomain.com and you have already created a namespace named mynamespace, use one of the following methods:

Pre-Authkeys method (recommended)

First, create a pre-authkey token which is valid for 24h on the server:

headscale preauthkeys create -e 24h -n mynamespace

or (docker-compose version)

docker-compose exec headscale headscale preauthkeys create -e 24h -n mynamespace

This will generate a pre-auth key such as 3215a1ce7967c11e8ea844b3e199d3c46f9f5e7b660b48fb which you can send to the user.

Now login on the client using

tailscale up --login-server https://headscale.mydomain.com --authkey 3215a1ce7967c11e8ea844b3e199d3c46f9f5e7b660b48fb

Direct login method

tailscale up --login-server https://headscale.mydomain.com

On the client, this will show you an URL to access using your browser on the headscale server. This will in turn give you a command that you need to run on the host running the headscale container. If running headscale using docker-compose, prepend docker-compose exec headscale to the command and replace NAMESPACE by the name of your namespace.

The only reason why this method is not recommended by me is because it requires back-and-forth interaction between the user and the administrator which I don’t consider practical.

Posted by Uli Köhler in Headscale, Linux, Networking, VPN