iperf3 benchmark of ZeroTier vs Netmaker vs Tailscale vs direct switched connection

In our setup, a virtual machine (running on an XCP-NG host) on  was connected to my Desktop (HP Z240, i7-6700 @3.4 GHz running Ubuntu 22.04) in a purely switched network with 1Gbit links. Both devices were connected using a MikroTik 10G switch (Marvell chip

I ran iperf3 -s on the VM and ran iperf3 -c [IP address] on the desktop. Reverse tests have not been performed.

Direct switched connection (no VPN)

Connecting to host 10.9.2.103, port 5201
[  5] local 10.9.2.10 port 56848 connected to 10.9.2.103 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  92.8 MBytes   779 Mbits/sec    0    444 KBytes       
[  5]   1.00-2.00   sec  90.7 MBytes   761 Mbits/sec    0    543 KBytes       
[  5]   2.00-3.00   sec  88.6 MBytes   743 Mbits/sec    0    816 KBytes       
[  5]   3.00-4.00   sec  90.0 MBytes   755 Mbits/sec    0    816 KBytes       
[  5]   4.00-5.00   sec  90.0 MBytes   755 Mbits/sec    0    856 KBytes       
[  5]   5.00-6.00   sec  88.8 MBytes   744 Mbits/sec    0    946 KBytes       
[  5]   6.00-7.00   sec  88.8 MBytes   745 Mbits/sec    0    946 KBytes       
[  5]   7.00-8.00   sec  90.0 MBytes   755 Mbits/sec    0    993 KBytes       
[  5]   8.00-9.00   sec  90.0 MBytes   755 Mbits/sec    0    993 KBytes       
[  5]   9.00-10.00  sec  88.8 MBytes   744 Mbits/sec    0    993 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   898 MBytes   754 Mbits/sec    0             sender
[  5]   0.00-10.01  sec   896 MBytes   751 Mbits/sec                  receiver

ZeroTier

Connecting to host 10.80.246.34, port 5201
[  5] local 10.80.246.38 port 35474 connected to 10.80.246.34 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  59.9 MBytes   503 Mbits/sec  338    102 KBytes       
[  5]   1.00-2.00   sec  60.2 MBytes   505 Mbits/sec  313    188 KBytes       
[  5]   2.00-3.00   sec  63.9 MBytes   536 Mbits/sec  176   99.3 KBytes       
[  5]   3.00-4.00   sec  74.3 MBytes   623 Mbits/sec  174    113 KBytes       
[  5]   4.00-5.00   sec  67.7 MBytes   568 Mbits/sec  197   83.2 KBytes       
[  5]   5.00-6.00   sec  72.5 MBytes   609 Mbits/sec  218    228 KBytes       
[  5]   6.00-7.00   sec  61.3 MBytes   514 Mbits/sec  281   77.8 KBytes       
[  5]   7.00-8.00   sec  72.0 MBytes   604 Mbits/sec  213   91.2 KBytes       
[  5]   8.00-9.00   sec  65.4 MBytes   549 Mbits/sec  309    156 KBytes       
[  5]   9.00-10.00  sec  53.9 MBytes   453 Mbits/sec  190    121 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   651 MBytes   546 Mbits/sec  2409             sender
[  5]   0.00-10.01  sec   650 MBytes   545 Mbits/sec                  receiver

NetMaker

Netmaker internally uses a normal (kernel-based) wireguard connection, so in some respect this is a test of Wireguard performance

Connecting to host 10.230.113.3, port 5201
[  5] local 10.230.113.1 port 35534 connected to 10.230.113.3 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   105 MBytes   881 Mbits/sec    0   1.01 MBytes       
[  5]   1.00-2.00   sec   104 MBytes   870 Mbits/sec   86    422 KBytes       
[  5]   2.00-3.00   sec   101 MBytes   849 Mbits/sec    0    488 KBytes       
[  5]   3.00-4.00   sec  98.8 MBytes   828 Mbits/sec    0    535 KBytes       
[  5]   4.00-5.00   sec  98.8 MBytes   828 Mbits/sec    0    584 KBytes       
[  5]   5.00-6.00   sec   104 MBytes   870 Mbits/sec    0    615 KBytes       
[  5]   6.00-7.00   sec  97.5 MBytes   818 Mbits/sec    7    472 KBytes       
[  5]   7.00-8.00   sec   104 MBytes   870 Mbits/sec    0    522 KBytes       
[  5]   8.00-9.00   sec   101 MBytes   849 Mbits/sec    0    580 KBytes       
[  5]   9.00-10.00  sec   102 MBytes   860 Mbits/sec    0    606 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1016 MBytes   852 Mbits/sec   93             sender
[  5]   0.00-10.00  sec  1014 MBytes   850 Mbits/sec                  receiver

Tailscale

Tailscale 1.28.0 has been used for this test.

During this test, I ensured that the tailscale connection was established using the switched network and was not going through a DERP server or the routed network.

$ tailscale ping 100.64.0.3
pong from vm (fd5d:7b60:4742::3) via 10.9.2.103:41641 in 1ms

Results:

Connecting to host 100.64.0.3, port 5201
[  5] local 100.64.0.2 port 40690 connected to 100.64.0.3 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  38.3 MBytes   321 Mbits/sec  389   60.0 KBytes       
[  5]   1.00-2.00   sec  37.6 MBytes   315 Mbits/sec  366   43.2 KBytes       
[  5]   2.00-3.00   sec  36.7 MBytes   308 Mbits/sec  431   52.8 KBytes       
[  5]   3.00-4.00   sec  38.5 MBytes   323 Mbits/sec  488   80.3 KBytes       
[  5]   4.00-5.00   sec  29.3 MBytes   246 Mbits/sec  356   38.4 KBytes       
[  5]   5.00-6.00   sec  31.0 MBytes   260 Mbits/sec  351   86.3 KBytes       
[  5]   6.00-7.00   sec  27.1 MBytes   227 Mbits/sec  287   50.4 KBytes       
[  5]   7.00-8.00   sec  26.1 MBytes   219 Mbits/sec  210   46.8 KBytes       
[  5]   8.00-9.00   sec  27.1 MBytes   227 Mbits/sec  261   39.6 KBytes       
[  5]   9.00-10.00  sec  27.5 MBytes   231 Mbits/sec  222   40.8 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   319 MBytes   268 Mbits/sec  3361             sender
[  5]   0.00-10.01  sec   318 MBytes   267 Mbits/sec                  receiver

Summary

The approximate performance expectation in this specific scenario is:

Curiously, netmaker performed better than the direct connection. The reason for this is not known at this point, but a similar effect has been observed in this medium.com article.

Generally, one can see that Tailscale (which internally uses software wireguard) is approximately half the speed of ZeroTier, which in turn is outperformed significantly by Netmaker.

In a followup post I will describe advantages and disadvantages of those solutions and explore under which scenarios I would use the solutions.