iperf3 benchmark of ZeroTier vs Netmaker vs Tailscale vs direct switched connection
In our setup, a virtual machine (running on an XCP-NG host) on was connected to my Desktop (HP Z240, i7-6700 @3.4 GHz running Ubuntu 22.04) in a purely switched network with 1Gbit links. Both devices were connected using a MikroTik 10G switch (Marvell chip
I ran iperf3 -s on the VM and ran iperf3 -c [IP address] on the desktop. Reverse tests have not been performed.
Direct switched connection (no VPN)
Connecting to host 10.9.2.103, port 5201
[ 5] local 10.9.2.10 port 56848 connected to 10.9.2.103 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 92.8 MBytes 779 Mbits/sec 0 444 KBytes
[ 5] 1.00-2.00 sec 90.7 MBytes 761 Mbits/sec 0 543 KBytes
[ 5] 2.00-3.00 sec 88.6 MBytes 743 Mbits/sec 0 816 KBytes
[ 5] 3.00-4.00 sec 90.0 MBytes 755 Mbits/sec 0 816 KBytes
[ 5] 4.00-5.00 sec 90.0 MBytes 755 Mbits/sec 0 856 KBytes
[ 5] 5.00-6.00 sec 88.8 MBytes 744 Mbits/sec 0 946 KBytes
[ 5] 6.00-7.00 sec 88.8 MBytes 745 Mbits/sec 0 946 KBytes
[ 5] 7.00-8.00 sec 90.0 MBytes 755 Mbits/sec 0 993 KBytes
[ 5] 8.00-9.00 sec 90.0 MBytes 755 Mbits/sec 0 993 KBytes
[ 5] 9.00-10.00 sec 88.8 MBytes 744 Mbits/sec 0 993 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 898 MBytes 754 Mbits/sec 0 sender
[ 5] 0.00-10.01 sec 896 MBytes 751 Mbits/sec receiver
ZeroTier
Connecting to host 10.80.246.34, port 5201
[ 5] local 10.80.246.38 port 35474 connected to 10.80.246.34 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 59.9 MBytes 503 Mbits/sec 338 102 KBytes
[ 5] 1.00-2.00 sec 60.2 MBytes 505 Mbits/sec 313 188 KBytes
[ 5] 2.00-3.00 sec 63.9 MBytes 536 Mbits/sec 176 99.3 KBytes
[ 5] 3.00-4.00 sec 74.3 MBytes 623 Mbits/sec 174 113 KBytes
[ 5] 4.00-5.00 sec 67.7 MBytes 568 Mbits/sec 197 83.2 KBytes
[ 5] 5.00-6.00 sec 72.5 MBytes 609 Mbits/sec 218 228 KBytes
[ 5] 6.00-7.00 sec 61.3 MBytes 514 Mbits/sec 281 77.8 KBytes
[ 5] 7.00-8.00 sec 72.0 MBytes 604 Mbits/sec 213 91.2 KBytes
[ 5] 8.00-9.00 sec 65.4 MBytes 549 Mbits/sec 309 156 KBytes
[ 5] 9.00-10.00 sec 53.9 MBytes 453 Mbits/sec 190 121 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 651 MBytes 546 Mbits/sec 2409 sender
[ 5] 0.00-10.01 sec 650 MBytes 545 Mbits/sec receiver
NetMaker
Netmaker internally uses a normal (kernel-based) wireguard connection, so in some respect this is a test of Wireguard performance
Connecting to host 10.230.113.3, port 5201
[ 5] local 10.230.113.1 port 35534 connected to 10.230.113.3 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 105 MBytes 881 Mbits/sec 0 1.01 MBytes
[ 5] 1.00-2.00 sec 104 MBytes 870 Mbits/sec 86 422 KBytes
[ 5] 2.00-3.00 sec 101 MBytes 849 Mbits/sec 0 488 KBytes
[ 5] 3.00-4.00 sec 98.8 MBytes 828 Mbits/sec 0 535 KBytes
[ 5] 4.00-5.00 sec 98.8 MBytes 828 Mbits/sec 0 584 KBytes
[ 5] 5.00-6.00 sec 104 MBytes 870 Mbits/sec 0 615 KBytes
[ 5] 6.00-7.00 sec 97.5 MBytes 818 Mbits/sec 7 472 KBytes
[ 5] 7.00-8.00 sec 104 MBytes 870 Mbits/sec 0 522 KBytes
[ 5] 8.00-9.00 sec 101 MBytes 849 Mbits/sec 0 580 KBytes
[ 5] 9.00-10.00 sec 102 MBytes 860 Mbits/sec 0 606 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 1016 MBytes 852 Mbits/sec 93 sender
[ 5] 0.00-10.00 sec 1014 MBytes 850 Mbits/sec receiver
Tailscale
Tailscale 1.28.0 has been used for this test.
During this test, I ensured that the tailscale connection was established using the switched network and was not going through a DERP server or the routed network.
$ tailscale ping 100.64.0.3
pong from vm (fd5d:7b60:4742::3) via 10.9.2.103:41641 in 1ms
Results:
Connecting to host 100.64.0.3, port 5201
[ 5] local 100.64.0.2 port 40690 connected to 100.64.0.3 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 38.3 MBytes 321 Mbits/sec 389 60.0 KBytes
[ 5] 1.00-2.00 sec 37.6 MBytes 315 Mbits/sec 366 43.2 KBytes
[ 5] 2.00-3.00 sec 36.7 MBytes 308 Mbits/sec 431 52.8 KBytes
[ 5] 3.00-4.00 sec 38.5 MBytes 323 Mbits/sec 488 80.3 KBytes
[ 5] 4.00-5.00 sec 29.3 MBytes 246 Mbits/sec 356 38.4 KBytes
[ 5] 5.00-6.00 sec 31.0 MBytes 260 Mbits/sec 351 86.3 KBytes
[ 5] 6.00-7.00 sec 27.1 MBytes 227 Mbits/sec 287 50.4 KBytes
[ 5] 7.00-8.00 sec 26.1 MBytes 219 Mbits/sec 210 46.8 KBytes
[ 5] 8.00-9.00 sec 27.1 MBytes 227 Mbits/sec 261 39.6 KBytes
[ 5] 9.00-10.00 sec 27.5 MBytes 231 Mbits/sec 222 40.8 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 319 MBytes 268 Mbits/sec 3361 sender
[ 5] 0.00-10.01 sec 318 MBytes 267 Mbits/sec receiver
Summary
The approximate performance expectation in this specific scenario is:
- Tailscale:
300 Mbit/s - ZeroTier:
550 Mbit/s - Netmaker:
850 Mbit/s - Direct switched network:
750 Mbit/s
Curiously, netmaker performed better than the direct connection. The reason for this is not known at this point, but a similar effect has been observed in this medium.com article.
Generally, one can see that Tailscale (which internally uses software wireguard) is approximately half the speed of ZeroTier, which in turn is outperformed significantly by Netmaker.
In a followup post I will describe advantages and disadvantages of those solutions and explore under which scenarios I would use the solutions.