Container

How to automatically remove docker images that are not associated to a container daily

Note: This will not only remove docker images without a tag but all docker images not associated to a running or stopped container. See our previous post How to automatically cleanup (prune) docker images daily in case this is not the desired behaviour.

docker image prune provides an easy way to remove “unused” docker images from a system and hence fixes or significantly delays docker eating up all your disk space on e.g. automated disk space.

I created a systemd-timer based daily image removal routine using TechOverflow’s Simple systemd timer generator.

Quick install using

wget -qO- https://techoverflow.net/scripts/install-cleanup-docker-all.sh | sudo bash

This is the script which automatically creates & installs both systemd config files.

#!/bin/sh
# This script installs automated docker cleanup.
# onto systemd-based systems.
# See https://techoverflow.net/2020/02/04/how-to-remove-all-docker-images-that-are-not-associated-to-a-container/
# for details on what images are removed.
# It requires that docker is installed properly

cat >/etc/systemd/system/PruneDockerAll.service <<EOF
[Unit]
Description=PruneDockerAll

[Service]
Type=oneshot
ExecStart=/bin/bash -c "docker image ls --format '{{.ID}}' | xargs docker image rm ; true"
WorkingDirectory=/tmp
EOF

cat >/etc/systemd/system/PruneDockerAll.timer <<EOF
[Unit]
Description=PruneDockerAll

[Timer]
OnCalendar=daily
Persistent=true

[Install]
WantedBy=timers.target
EOF

# Enable and start service
systemctl enable PruneDockerAll.timer && sudo systemctl start PruneDockerAll.timer

 

To view the logs, use

journalctl -xfu PruneDockerAll.service

To view the status, use

sudo systemctl status PruneDockerAll.timer

To immediately cleanup your docker images, use

sudo systemctl start PruneDockerAll.service
Posted by Uli Köhler in Docker

How to remove all stopped docker containers

Removing all docker containers that are currently stopped is simple:

docker container prune

In case you want to skip the Are you sure you want to continue? [y/N] confirmation, use:

docker container prune -f

Also see our post on How to remove all docker images that are not associated to a container.

Posted by Uli Köhler in Docker

How to remove all docker images that are not associated to a container

In our previous post we showed how to prune docker images to free up space on your hard drive. However, this approach will not remove images that have tags (i.e. names) associated with them.

Often you want to remove all images that are not required by one of the containers (both running and stopped containers).

This is pretty easy:

docker image ls --format '{{.ID}}' | xargs docker image rm

This command will list all image IDs using docker image ls --format '{{.ID}}' and run docker image rm for every image ID.

Since docker image rm will fail for images that are associated to either a running or a stopped container (hence that image won’t be deleted), this will only delete those images that are not associated to any container.

In case you get a lot of those error messages as output from the command:

Error response from daemon: conflict: unable to delete 1f9cfa8dc305 (cannot be forced) - image is being used by running container 22a27af7d595
Error response from daemon: conflict: unable to delete 9af515ad5c74 (must be forced) - image is being used by stopped container 2ebcbd936841

don’t worry, that’s fine, that just means one of your images is associated to a container and hence won’t be deleted.

Posted by Uli Köhler in Docker

How to print only image ID in ‘docker image ls’

Use --format  '{{.ID}}':

docker image ls --format '{{.ID}}'

 

Posted by Uli Köhler in Docker

How to fix bitnami mariadb ‘mkdir: cannot create directory ‘/bitnami/mariadb’: Permission denied’

Problem:

You are trying to run a docker bitnami/mariadb container but when you try to start it up, you see an error message like

mariadb_1   | mkdir: cannot create directory '/bitnami/mariadb': Permission denied

Solution:

bitnami containers mostly are non-root-containers, hence you need to adjust the permissions for the data directory mapped onto the host.

First, find out what directory your /bitnami is mapped to on the host. For example, for

services:
     mariadb:
         image: 'bitnami/mariadb:latest'
         environment:
             - ALLOW_EMPTY_PASSWORD=yes
         volumes:
             - '/var/lib/my_docker/mariadb_data:/bitnami'

it is mapped to /var/lib/my_docker/mariadb_data.

Now chown this directory to 1001:1001 since the image is using UID 1001 as the user running the command:

sudo chown -R 1001:1001 [directory]

for example

sudo chown -R 1001:1001 /var/lib/my_docker/mariadb_data

 

Posted by Uli Köhler in Docker

How to remove all docker networks

In order to remove all docker network interfaces (including all the bridge interfaces), run

docker network rm $(docker network ls -q)

in your favourite shell.

This will attempt to remove some pre-defined networks that can’t be removed and hence show you these error messages:

Error response from daemon: bridge is a pre-defined network and cannot be removed
Error response from daemon: host is a pre-defined network and cannot be removed
Error response from daemon: none is a pre-defined network and cannot be removed

You can safely ignore those error messages.

Posted by Uli Köhler in Container, Docker

How to fix ‘docker: invalid reference format.’

Problem:

You want to start docker container but you see an error message like this:

docker: invalid reference format.

Solution:

Docker is telling you that the syntax of the docker image name (& version) is wrong. Note that this is not the same as docker not being able to find the image in the registry. Docker will not even be able to lookup the image in the registry if you see an invalid reference format error!

Common causes include:

  • You used a colon at the end of the image name, e.g. ubuntu: – omit the colon ; using just ubuntu will refer to ubuntu:latest
  • You used a dash at the end of the image name, e.g. ubuntu- – omit the dash ; using just ubuntu will refer to ubuntu:latest
  • You used variables like ubuntu:$VERSION but $VERSION is not set. Ensure you have set $VERSION to an appropriate (non-empty) value like latest or 18.04!
  • You used multiple colons like ubuntu:18.04:2 or ubuntu:ubuntu:18.04. Use only a single colon in the image name!
  • You mixed up the order of command line arguments, so another argument is being interpreted as the image name. Check the syntax of your docker command, e.g. docker run and compare some examples with your command.
Posted by Uli Köhler in Container, Docker

How to install & use nano in a running Docker container

If you want to interactively edit a file in a docker container, you might want to install an editor like GNU nano (for example to debug your config files) in your Docker container that allows you direct access to the container’s file system.

docker exec -it [container name or ID] bash -c 'apt-get -y update && apt -y install nano'

This will work for most debian/Ubuntu-based containers ; for other containers you might need to use a package manager other than apt

Now you can use docker exec -it to interactively edit a config file, e.g.:

docker exec -it [container name or ID] nano /etc/host.conf

 

Posted by Uli Köhler in Container, Docker

What’s the difference between ‘docker exec’ and ‘docker run’?

docker exec runs a program in a container that’s already running. For example, you can use it to create a backup in a container that currently runs your database server.

docker run starts a new container and runs a command in it. For example, your can use it to run a specific script in a container.

Posted by Uli Köhler in Container, Docker

How to automatically cleanup (prune) docker images daily

Note: This will only remove docker images without a tag, but not all images not associated to a running or stopped container. See our post on How to automatically cleanup (prune) docker images daily in case this is not the desired behaviour.

docker image prune provides an easy way to remove “unused” (i.e. untagged) docker images from a system and hence fixes or significantly delays docker eating up all your disk space on.

I created a systemd-timer based daily prune routine using TechOverflow’s Simple systemd timer generator.

Quick install using

wget -qO- https://techoverflow.net/scripts/install-cleanup-docker.sh | sudo bash

This is the script which automatically creates & installs both systemd config files.

#!/bin/sh
# This script installs automated docker cleanup via "docker image prune"
# onto systemd-based systems.
# It requires that docker is installed properly

cat >/etc/systemd/system/PruneDocker.service <<EOF
[Unit]
Description=PruneDocker

[Service]
Type=oneshot
ExecStart=/usr/bin/docker image prune -f
WorkingDirectory=/tmp
EOF

cat >/etc/systemd/system/PruneDocker.timer <<EOF
[Unit]
Description=PruneDocker

[Timer]
OnCalendar=daily
Persistent=true

[Install]
WantedBy=timers.target
EOF

# Enable and start service
systemctl enable PruneDocker.timer && sudo systemctl start PruneDocker.timer

To view logs, use

journalctl -xfu PruneDocker.service

To view the status, use

sudo systemctl status PruneDocker.timer

To immediately cleanup your docker images, use

sudo systemctl start PruneDocker.service
Posted by Uli Köhler in Container, Docker

How to remove all unused docker images

Use

docker image prune -f

to remove all unused/old (“dangling”) docker images. -f will skip the confirmation prompt.

This command will remove all images that a) don’t have a tag themselves and b) no image with a tag depends on the image.

Example output:

Deleted Images:
deleted: sha256:b6e0e69c6ba1e811063921d18b52627a9fb905e0bdfd80e226e86958831df636
[...]
deleted: sha256:6a40d04ddf00ad0f1806df7b2d4a2d44a6d8031cab5c369a4bf3d1694d5c48b4

Total reclaimed space: 985.3MB

 

Posted by Uli Köhler in Container, Docker

How to fix docker-compose start ERROR: No containers to start

Problem:

When running docker-compose start, you see an error message like this:

Starting mongodb     ... error
Starting myapp       ... error

ERROR: No containers to start

Solution:

In order to start your containers, use docker-compose up, instead of docker-compose start!

Posted by Uli Köhler in Container, Docker

The super-simple docker-compose cheatsheet

Run these commands in the directory (or git repo) where docker-compose.yml is located!

Start all services

docker-compose up -d

-d means run in background (= daemonize).

Stop all services

docker-compose down

Restart all services

docker-compose restart

Update containers

docker-compose pull
docker-compose restart

View logs

docker-compose logs

To view and follow use

docker-compose logs -f

 

Start a specific service (and all the services it depends on)

docker-compose start myservice

Show info about which container images are being used

docker-compose images
Posted by Uli Köhler in Container, Docker

Minimal local nginx setup using Docker

If you have not installed Docker, see our guide at How to install docker and docker-compose on Ubuntu in 30 seconds

1. Create your nginx config file (my-nginx.conf). This is a template that reverse proxys TechOverflow:

server {
    listen 80 default_server;
    listen [::]:80 default_server;

    location / {
        proxy_pass https://techoverflow.net;
        proxy_http_version 1.1;
    }
}

3. Start nginx using docker:

docker run -it -p 80:80 --rm -v $(pwd)/my-nginx.conf:/etc/nginx/conf.d/default.conf nginx:latest

4. Go to http://localhost and see the result!

Explanation of the docker command:

  • docker run -it: Create a new docker container and run it in interactive mode (i.e. it will not run in the background, once you kill the command, nginx will exit)
  • -p 80:80: Makes port 80 of the nginx server (the standard HTTP port) available on the host’s port 80. The first 80 is the host port whereas the second port 80 is the container’s port.
  • --rm: Once the container is stopped, delete it!
  • -v $(pwd)/my-nginx.conf:/etc/nginx/conf.d/default.conf: Map my-nginx.conf in the current directory ($(pwd)) to /etc/nginx/conf.d/default.conf on the container.
  • nginx:latest: In the container run the official nginx image from DockerHub in the latest version.

Explanation of the nginx config file:

  • server { ... }: Everything inside this blog will belong together. You can
  • listen 80 default_server; Listen on port 80 (the standard HTTP port) and make this the default server, i.e. respond to any domain name that does not have any other server configured.
  • listen [::]:80 default_server; Same as the previous line, but for IPv6. [::] means: Listen on all IPv6 addresses.
  • location / { ... }: Everything inside this block is valid for any URL starting with / i.e. any URL at all. In clauses like location /app { ... } the content of the clause would be valid for URLs starting with /app only, e.g. http://localhost/app/ or http://localhost/app/dashboard.
  • proxy_pass https://techoverflow.net; Redirect requests to the current location (/) to the server https://techoverflow.net using a reverse proxy.
  • proxy_http_version 1.1; This sets the HTTP version that nginx uses to make the requests to https://techoverflow.net. This is not always neccessary but might increase compatibility.
Posted by Uli Köhler in Docker, nginx

Docker: How to remove all stopped containers

Run this command in your shell:

docker rm $(docker ps -a -q)

 

Posted by Uli Köhler in Container, Docker

Docker: Stop & remove all running containers

To stop & delete all running containers in docker use

docker rm $(docker ps -a -q) --force
Posted by Uli Köhler in Docker

How to create a LXC Ubuntu container in 15 seconds

Run this:

lxc launch ubuntu:22.04 mycontainer

In case you see this error message

Error: Failed container creation: No storage pool found. Please create a new storage pool

see How to fix lxd ‘Failed container creation: No storage pool found. Please create a new storage pool.’

Now you can connect to your container using

lxc exec mycontainer /bin/bash

 

Posted by Uli Köhler in Container, Linux, LXC

Towards a docker-based build of C/C++ applications

Note: Based on this post I have published buildock on GitHub.

Many C/C++ programmers and project managers know the pain of creating a reproducible build environment for all developers: Works for me is a common meme not without a reason.

My approach is to dockerize not neccessarily the application itself but the build system, encapsulating both the specific compiler version and the system around it plus all required system-level libraries in a Docker image.

Take the obligatory Hello World in C++:

// main.cpp
#include <iostream>
using namespace std;

int main() {
    cout << "Hello, World!" << endl;
    return 0;
}

and the corresponding Makefile:

all:
        g++ -o helloworld main.cpp

How can we compile this simple project without installing a compiler and GNU make on the local computer (no cheating by using build servers allowed)?

Actually it’s rather simple:

docker run --user $(id -u):$(id -g) -v $PWD:/app -it ulikoehler/ubuntu-gcc-make make

Breaking it down:

  • docker run: Create a new docker container and run a command in it
  • --user $(id -u):$(id -g): This makes the docker container run with the current user’s ID and the current user’s group – both for preventing the compiler to create output files as root and to safeguard against some IT security risks. Also see How to run docker container as current user & group
  • -v $PWD:/app: Mount the current directory ($PWD) on /app in the container. Since the Dockerfile used to build the container contains the WORKDIR /app directive, whatever command we run in the container will by default be executed in the /app directory – and therefore in the current local directory on the host.
  • -it runs the container in interactive mode, i.e. keypressed are passed down to the command you are running. Also, this means that our command will only finish when the container has finished executing.
  • ulikoehler/ubuntu-gcc-make: This is the image we’re using for that example. It’s nothing more than an ubuntu:18.04 base image with build-essentials and make installed and WORKDIR set to /app
  • make: This is the command we’ll run in the container. You can use any command here, even no command is possible (in which case the container’s default command will be used – in case of ulikoehler/ubuntu-gcc-make that is CMD [ "/usr/bin/make" ])

Here’s the complete Dockerfile used to generate ulikoehler/ubuntu-gcc-make:

FROM ubuntu:18.04
RUN apt update && apt -y install build-essential make && rm -rf /var/lib/apt/lists/*
WORKDIR /app
CMD [ "/usr/bin/make" ]
Posted by Uli Köhler in Build systems, C/C++, Container, Docker

How to have multiple Dockerfiles in one directory

Usually you build a docker image like

docker build -t myimagename .

but using this method you can only have one Dockerfile in each directory.

In case you don’t want to have separate directories for your Dockerfiles you can use the -f argument to docker build:

docker build -f FirstDockerfile .

Note that you still need to add . at the end so docker build knows where to COPY files from if you have COPY or similar statements in your Dockerfile.

Posted by Uli Köhler in Docker