Wie ich docker panic: assertion failed: write: circular dependency occurred behoben habe
Problem:
Beim Starten von Docker auf einer VM, die zuvor während eines Stromausfalls plötzlich ausgeschaltet wurde, startete der Docker-Daemon nicht und zeigte das folgende Fehlerprotokoll:
Jan 05 15:15:14 CoreOS-Haar dockerd[5333]: time="2022-01-05T15:15:14.946815577Z" level=info msg="Starting up"
Jan 05 15:15:14 CoreOS-Haar dockerd[5333]: time="2022-01-05T15:15:14.947842629Z" level=info msg="detected 127.0.0.53 nameserver, assuming systemd-resolved, so using resolv.conf: /run/systemd/resolve/resolv.conf"
Jan 05 15:15:14 CoreOS-Haar dockerd[5333]: time="2022-01-05T15:15:14.949500623Z" level=info msg="parsed scheme: \"unix\"" module=grpc
Jan 05 15:15:14 CoreOS-Haar dockerd[5333]: time="2022-01-05T15:15:14.949617127Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
Jan 05 15:15:14 CoreOS-Haar dockerd[5333]: time="2022-01-05T15:15:14.949705114Z" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/containerd.sock <nil> 0 <nil>}] <nil> <nil>}" module=grpc
Jan 05 15:15:14 CoreOS-Haar dockerd[5333]: time="2022-01-05T15:15:14.949776371Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
Jan 05 15:15:14 CoreOS-Haar dockerd[5333]: time="2022-01-05T15:15:14.950747679Z" level=info msg="parsed scheme: \"unix\"" module=grpc
Jan 05 15:15:14 CoreOS-Haar dockerd[5333]: time="2022-01-05T15:15:14.950788173Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
Jan 05 15:15:14 CoreOS-Haar dockerd[5333]: time="2022-01-05T15:15:14.950806216Z" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/containerd.sock <nil> 0 <nil>}] <nil> <nil>}" module=grpc
Jan 05 15:15:14 CoreOS-Haar dockerd[5333]: time="2022-01-05T15:15:14.950815090Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
Jan 05 15:15:15 CoreOS-Haar dockerd[5333]: time="2022-01-05T15:15:15.012683899Z" level=info msg="[graphdriver] using prior storage driver: overlay2"
Jan 05 15:15:15 CoreOS-Haar dockerd[5333]: time="2022-01-05T15:15:15.027806434Z" level=warning msg="Docker could not enable SELinux on the host system"
Jan 05 15:15:15 CoreOS-Haar dockerd[5333]: time="2022-01-05T15:15:15.176098505Z" level=info msg="Loading containers: start."
Jan 05 15:15:15 CoreOS-Haar dockerd[5333]: time="2022-01-05T15:15:15.728503609Z" level=info msg="Removing stale sandbox 22e03a9f65217fa0ce1603fa1d6326b7bf412777be94e930b02dbe6554549084 (154aa4bd403045e229b39cc4dda1d16a72b45d18671cab7c993bff4eaee9c2c9)"
Jan 05 15:15:15 CoreOS-Haar dockerd[5333]: panic: assertion failed: write: circular dependency occurred
Jan 05 15:15:15 CoreOS-Haar dockerd[5333]: goroutine 1 [running]:
Jan 05 15:15:15 CoreOS-Haar dockerd[5333]: github.com/docker/docker/vendor/go.etcd.io/bbolt._assert(...)
Jan 05 15:15:15 CoreOS-Haar dockerd[5333]: /builddir/build/BUILD/moby-20.10.11/src/github.com/docker/docker/vendor/go.etcd.io/bbolt/db.go:1172
Jan 05 15:15:15 CoreOS-Haar dockerd[5333]: github.com/docker/docker/vendor/go.etcd.io/bbolt.(*node).write(0xc0001e16c0, 0xc001381000)
Jan 05 15:15:15 CoreOS-Haar dockerd[5333]: /builddir/build/BUILD/moby-20.10.11/src/github.com/docker/docker/vendor/go.etcd.io/bbolt/node.go:233 +0x3c5
Jan 05 15:15:15 CoreOS-Haar dockerd[5333]: github.com/docker/docker/vendor/go.etcd.io/bbolt.(*node).spill(0xc0001e16c0, 0xc0012da078, 0x1)
Jan 05 15:15:15 CoreOS-Haar dockerd[5333]: /builddir/build/BUILD/moby-20.10.11/src/github.com/docker/docker/vendor/go.etcd.io/bbolt/node.go:374 +0x225
Jan 05 15:15:15 CoreOS-Haar dockerd[5333]: github.com/docker/docker/vendor/go.etcd.io/bbolt.(*node).spill(0xc0001e1650, 0xc000fefaa0, 0xc000d9dc50)
Jan 05 15:15:15 CoreOS-Haar dockerd[5333]: /builddir/build/BUILD/moby-20.10.11/src/github.com/docker/docker/vendor/go.etcd.io/bbolt/node.go:346 +0xbc
Jan 05 15:15:15 CoreOS-Haar dockerd[5333]: github.com/docker/docker/vendor/go.etcd.io/bbolt.(*Bucket).spill(0xc000d6f180, 0xc000fefa00, 0xc000d9de40)
Jan 05 15:15:15 CoreOS-Haar dockerd[5333]: /builddir/build/BUILD/moby-20.10.11/src/github.com/docker/docker/vendor/go.etcd.io/bbolt/bucket.go:570 +0x49a
Jan 05 15:15:15 CoreOS-Haar dockerd[5333]: github.com/docker/docker/vendor/go.etcd.io/bbolt.(*Bucket).spill(0xc000fca2b8, 0x310b759f, 0x55dfe7bf17e0)
Jan 05 15:15:15 CoreOS-Haar dockerd[5333]: /builddir/build/BUILD/moby-20.10.11/src/github.com/docker/docker/vendor/go.etcd.io/bbolt/bucket.go:537 +0x3f6
Jan 05 15:15:15 CoreOS-Haar dockerd[5333]: github.com/docker/docker/vendor/go.etcd.io/bbolt.(*Tx).Commit(0xc000fca2a0, 0x0, 0x0)
Jan 05 15:15:15 CoreOS-Haar dockerd[5333]: /builddir/build/BUILD/moby-20.10.11/src/github.com/docker/docker/vendor/go.etcd.io/bbolt/tx.go:160 +0xe8
Jan 05 15:15:15 CoreOS-Haar dockerd[5333]: github.com/docker/docker/vendor/go.etcd.io/bbolt.(*DB).Update(0xc000d6bc00, 0xc000d9e078, 0x0, 0x0)
Jan 05 15:15:15 CoreOS-Haar dockerd[5333]: /builddir/build/BUILD/moby-20.10.11/src/github.com/docker/docker/vendor/go.etcd.io/bbolt/db.go:701 +0x105
Jan 05 15:15:15 CoreOS-Haar dockerd[5333]: github.com/docker/docker/vendor/github.com/docker/libkv/store/boltdb.(*BoltDB).AtomicPut(0xc000a862d0, 0xc000fd32d0, 0x65, 0xc000d46300, 0x179, 0x180, 0xc000fef890, 0x0, 0x55dfe5407900, 0x0, ...)
Jan 05 15:15:15 CoreOS-Haar dockerd[5333]: /builddir/build/BUILD/moby-20.10.11/src/github.com/docker/docker/vendor/github.com/docker/libkv/store/boltdb/boltdb.go:371 +0x225
Jan 05 15:15:15 CoreOS-Haar dockerd[5333]: github.com/docker/docker/vendor/github.com/docker/libnetwork/datastore.(*datastore).PutObjectAtomic(0xc000c3bbc0, 0x55dfe6bd1398, 0xc000efad20, 0x0, 0x0)
Jan 05 15:15:15 CoreOS-Haar dockerd[5333]: /builddir/build/BUILD/moby-20.10.11/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/datastore/datastore.go:415 +0x3ca
Jan 05 15:15:15 CoreOS-Haar dockerd[5333]: github.com/docker/docker/vendor/github.com/docker/libnetwork/drivers/bridge.(*driver).storeUpdate(0xc0000c3c00, 0x55dfe6bd1398, 0xc000efad20, 0x0, 0x0)
Jan 05 15:15:15 CoreOS-Haar dockerd[5333]: /builddir/build/BUILD/moby-20.10.11/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/drivers/bridge/bridge_store.go:106 +0x6e
Jan 05 15:15:15 CoreOS-Haar dockerd[5333]: github.com/docker/docker/vendor/github.com/docker/libnetwork/drivers/bridge.(*driver).RevokeExternalConnectivity(0xc0000c3c00, 0xc000f19d80, 0x40, 0xc00111c300, 0x40, 0x0, 0x0)
Jan 05 15:15:15 CoreOS-Haar dockerd[5333]: /builddir/build/BUILD/moby-20.10.11/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/drivers/bridge/bridge.go:1405 +0x1f1
Jan 05 15:15:15 CoreOS-Haar dockerd[5333]: github.com/docker/docker/vendor/github.com/docker/libnetwork.(*endpoint).sbLeave(0xc000d41e40, 0xc000f63200, 0x1, 0x0, 0x0, 0x0, 0x0, 0xc001b91ce0)
Jan 05 15:15:15 CoreOS-Haar dockerd[5333]: /builddir/build/BUILD/moby-20.10.11/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/endpoint.go:751 +0x1326
Jan 05 15:15:15 CoreOS-Haar dockerd[5333]: github.com/docker/docker/vendor/github.com/docker/libnetwork.(*endpoint).Delete(0xc000d41b80, 0xc000f19d01, 0x0, 0x0)
Jan 05 15:15:15 CoreOS-Haar dockerd[5333]: /builddir/build/BUILD/moby-20.10.11/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/endpoint.go:842 +0x374
Jan 05 15:15:15 CoreOS-Haar dockerd[5333]: github.com/docker/docker/vendor/github.com/docker/libnetwork.(*sandbox).delete(0xc000f63200, 0x1, 0x55dfe61ee216, 0x1e)
Jan 05 15:15:15 CoreOS-Haar dockerd[5333]: /builddir/build/BUILD/moby-20.10.11/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/sandbox.go:229 +0x191
Jan 05 15:15:15 CoreOS-Haar dockerd[5333]: github.com/docker/docker/vendor/github.com/docker/libnetwork.(*controller).sandboxCleanup(0xc000432a00, 0x0)
Jan 05 15:15:15 CoreOS-Haar dockerd[5333]: /builddir/build/BUILD/moby-20.10.11/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/sandbox_store.go:278 +0xdae
Jan 05 15:15:15 CoreOS-Haar dockerd[5333]: github.com/docker/docker/vendor/github.com/docker/libnetwork.New(0xc0000c3b80, 0x9, 0x10, 0xc000878090, 0xc001e69e60, 0xc0000c3b80, 0x9)
Jan 05 15:15:15 CoreOS-Haar dockerd[5333]: /builddir/build/BUILD/moby-20.10.11/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/controller.go:248 +0x726
Jan 05 15:15:15 CoreOS-Haar dockerd[5333]: github.com/docker/docker/daemon.(*Daemon).initNetworkController(0xc00000c1e0, 0xc00021a000, 0xc001e69e60, 0xc000151270, 0xc00000c1e0, 0xc000686a10, 0xc001e69e60)
Jan 05 15:15:15 CoreOS-Haar dockerd[5333]: /builddir/build/BUILD/moby-20.10.11/src/github.com/docker/docker/daemon/daemon_unix.go:855 +0xac
Jan 05 15:15:15 CoreOS-Haar dockerd[5333]: github.com/docker/docker/daemon.(*Daemon).restore(0xc00000c1e0, 0xc00045c4c0, 0xc00023e2a0)
Jan 05 15:15:15 CoreOS-Haar dockerd[5333]: /builddir/build/BUILD/moby-20.10.11/src/github.com/docker/docker/daemon/daemon.go:490 +0x52c
Jan 05 15:15:15 CoreOS-Haar dockerd[5333]: github.com/docker/docker/daemon.NewDaemon(0x55dfe6bad710, 0xc00045c4c0, 0xc00021a000, 0xc000878090, 0x0, 0x0, 0x0)
Jan 05 15:15:15 CoreOS-Haar dockerd[5333]: /builddir/build/BUILD/moby-20.10.11/src/github.com/docker/docker/daemon/daemon.go:1150 +0x2c1d
Jan 05 15:15:15 CoreOS-Haar dockerd[5333]: main.(*DaemonCli).start(0xc000b62720, 0xc00009a8a0, 0x0, 0x0)
Jan 05 15:15:15 CoreOS-Haar dockerd[5333]: /builddir/build/BUILD/moby-20.10.11/src/github.com/docker/docker/cmd/dockerd/daemon.go:195 +0x785
Jan 05 15:15:15 CoreOS-Haar dockerd[5333]: main.runDaemon(...)
Jan 05 15:15:15 CoreOS-Haar dockerd[5333]: /builddir/build/BUILD/moby-20.10.11/src/github.com/docker/docker/cmd/dockerd/docker_unix.go:13
Jan 05 15:15:15 CoreOS-Haar dockerd[5333]: main.newDaemonCommand.func1(0xc0000bc2c0, 0xc0000b2000, 0x0, 0xc, 0x0, 0x0)
Jan 05 15:15:15 CoreOS-Haar dockerd[5333]: /builddir/build/BUILD/moby-20.10.11/src/github.com/docker/docker/cmd/dockerd/docker.go:34 +0x7d
Jan 05 15:15:15 CoreOS-Haar dockerd[5333]: github.com/docker/docker/vendor/github.com/spf13/cobra.(*Command).execute(0xc0000bc2c0, 0xc00011caa0, 0xc, 0xc, 0xc0000bc2c0, 0xc00011caa0)
Jan 05 15:15:15 CoreOS-Haar dockerd[5333]: /builddir/build/BUILD/moby-20.10.11/src/github.com/docker/docker/vendor/github.com/spf13/cobra/command.go:850 +0x472
Jan 05 15:15:15 CoreOS-Haar dockerd[5333]: github.com/docker/docker/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc0000bc2c0, 0x0, 0x0, 0x10)
Jan 05 15:15:15 CoreOS-Haar dockerd[5333]: /builddir/build/BUILD/moby-20.10.11/src/github.com/docker/docker/vendor/github.com/spf13/cobra/command.go:958 +0x375
Jan 05 15:15:15 CoreOS-Haar dockerd[5333]: github.com/docker/docker/vendor/github.com/spf13/cobra.(*Command).Execute(...)
Jan 05 15:15:15 CoreOS-Haar dockerd[5333]: /builddir/build/BUILD/moby-20.10.11/src/github.com/docker/docker/vendor/github.com/spf13/cobra/command.go:895
Jan 05 15:15:15 CoreOS-Haar dockerd[5333]: main.main()
Jan 05 15:15:15 CoreOS-Haar dockerd[5333]: /builddir/build/BUILD/moby-20.10.11/src/github.com/docker/docker/cmd/dockerd/docker.go:97 +0x185
Jan 05 15:15:15 CoreOS-Haar systemd[1]: docker.service: Main process exited, code=exited, status=2/INVALIDARGUMENTLösung
Zuerst versuchen, /var/lib/docker/network/files/local-kv.db zu verschieben:
sudo mv /var/lib/docker/network/files/local-kv.db /opt/old-docker-local-kv.dbund dann docker neu starten, z.B. mit sudo systemctl restart docker, was für viele Benutzer auf GitHub funktioniert hat.
Nur wenn das nicht funktioniert, die folgende Brute-Force-Methode versuchen!
Das Verzeichnis /var/lib/docker an einen neuen Ort für ein Backup verschieben, danach konnte Docker wieder starten:
sudo mv /var/lib/docker /opt/old-var-lib-dockerDies ist eine ziemlich brute-force-Methode, aber für mich hat sie einwandfrei funktioniert, da mein Setup keine Volumes, sondern lokale Verzeichnisse verwendet hat. Falls Volumes aktiv verwendet werden, müssen die Volumes möglicherweise manuell aus /opt/old-var-lib-docker wiederhergestellt werden!