bup

How to backup docker-compose PostgreSQL database using bup

This command will generate a PostgreSQL dump using pg_dump and immediately feed it into bup split (without creating an intermediate file) for backup.

It assumes that .env contains a line

POSTGRES_USER=myuser

so that the sc

Local .bup variant

Set BUP_DIR

export BUP_DIR=/var/bup/my-database.bup
source .env && docker-compose exec -u postgres -T postgres pg_dump -U${POSTGRES_USER} | bup -d $BUP_DIR split -n mydb-pgdump.sql

bup remote variant

export BUP_DIR=/var/bup/my-database.index.bup
export BUP_REMOTE=bup-server:/bup/my-database.bup
source .env && docker-compose exec -u postgres -T postgres pg_dump -U${POSTGRES_USER} | bup -d $BUP_DIR split -r $BUP_REMOTE -n mydb-pgdump.sql

 

Posted by Uli Köhler in bup, Databases

How to setup bup-server on Synology NAS using docker

This guide shows you how to create a bup server. This is based on our previous post How to setup a “bup remote” server in 5 minutes using docker-compose but uses Synology’s built-in Docker GUI instead of docker-compose.

First, create two shared directories bup-backups (which will store the backups itself) and bup-config )which will store the dropbear SSH server configuration, that is SSH host keys and authorized client keys).

Alternatively, you can also use sub-directories of existing shared directories, but I’d like to keep them separate.

Then create a new Docker container by opening Docker -> Container, clicking Create and follow these steps:

Download ulikoehler/bup-server:lastest

Create a new container from the image

Map out port 2022 (bup server SSH port)

You can choose any other port in Local Port but keep the Container Port the same.

Map the volumes

As we said before, any directory will do. Create the sub-directories as needed.

Creating the SSH keys

On your local linux computer, create a SSH key using

ssh-keygen -t ed25519 -f id_bup -N ""

Upload id_bup and id_bup.pub to the bup-config shared folder.

Furthermore, copy id_pub.pub to bup-config/dotssh/authorized_keys.

After that you can startup the container.

Test the login

Use

ssh -i id_bup -p 2022 [email protected][AS IP address]

to try to connect to your NAS.

Troubleshooting

In case connecting via SSH does not work, most likely the issue is with your public/private key and/or your authorized_keys file. Check if it is in the right directory (/home/bup/.ssh/authorized_keys on the container). Also check the logs of the Docker container.

Posted by Uli Köhler in bup, Synology

bup remote server docker-compose config with CIFS-mounted backup store

In our previous post How to setup a “bup remote” server in 5 minutes using docker-compose we outlined how to setup your own bup remote server using docker-composeRead that post before this one!

This post provides an alternate docker-compose.yml config file that mounts a remote CIFS directory as /bup backup directory instead of using a local directory. This is most useful when using a NAS and a separate bup server.

For this example, we’ll mount the CIFS share //10.1.2.3/bup-backups with user cifsuser and password pheT8Eigho.

Note: For performance reasons, the CIFS server (NAS) and the bup server should be locally connected, not via the internet.

# Mount the backup volume using CIFS
# NOTE: We recommend to not use a storage mounted over the internet
# for performance reasons. Instead, deploy a bup remote server locally.
volumes:
  bup-backups:
    driver_opts:
      type: cifs
      o: "username=cifsuser,password=pheT8Eigho,uid=1111,gid=1111"
      device: "//10.1.2.3/bup-backups"

version: "3.8"
services:
  bup-server:
    image: ulikoehler/bup-server:latest
    environment:
      - SSH_PORT=2022
    volumes:
      - ./dotssh:/home/bup/.ssh
      - ./dropbear:/etc/dropbear
      # BUP backup storage: CIFS mounted
      - bup-backups:/bup
    ports:
      - 2022:2022
    restart: unless-stopped

 

Posted by Uli Köhler in bup, Docker, Networking

How to setup a “bup remote” server in 5 minutes using docker-compose

The bup backup system implements remote backup on a server by connecting via SSH to said server, starting a bup process there and then communicating via the SSH tunnel.

In this post, we’ll setup a server for bup remote backup based on our ulikoehler/bup-server image (which contains both bup and dropbear as an SSH server).

1. Initialize the directory structure & create SSH keyset to access the server

I recommend doing this in /opt/bup, but in principle, any directory will do.

mkdir -p dotssh bup
# Generate new elliptic curve public key
ssh-keygen -t ed25519 -f id_bup -N ""
# Add SSH key to list of authorized keys
cat id_bup.pub | sudo tee -a dotssh/authorized_keys
# Fix permissions so that dropbear does not complain
sudo chown -R 1111:1111 bup
sudo chmod 0600 dotssh/authorized_keys
sudo chmod 0700 dotssh

1111 is the user ID of the bup user in the VM.

2. Create docker-compose.yml

Note: This docker-compose.yml uses a local backup directory – you can also mount a CIFS directory from e.g. a NAS device. See bup remote server docker-compose config with CIFS-mounted backup store for more details.

version: "3.8"
services:
  bup-server:
    image: ulikoehler/bup-server:latest
    environment:
      - SSH_PORT=2022
    volumes:
      - ./dotssh:/home/bup/.ssh
      - ./dropbear:/etc/dropbear
      # BUP backup storage:
      - ./bup:/bup
    ports:
      - 2022:2022
    restart: unless-stopped

3. Startup the container

At this point, you can use docker-compose up to startup the service. However, it’s typically easier to just use TechOverflow’s script to generate a systemd script to autostart the service on boot (and start it right now):

curl -fsSL https://techoverflow.net/scripts/create-docker-compose-service.sh | sudo bash /dev/stdin

When you run docker-compose logs -f, you should see a greeting message from dropbear such as

bupremotedocker-bup-remote-1  | [1] Dec 25 14:58:20 Not backgrounding

4. Create a .ssh/config entry on the client

You need to do this for each client.

Copy id_bup (which we generated earlier) to each client into a folder such as ~/.ssh. Where you copy it does not matter, but the user who will be running the backups later will need access to this file. Also, for that user you need to create a .ssh/config entry telling SSH how to access the bup server:

Host BupServer
    HostName 10.1.2.3
    User bup
    Port 2022
    IdentityFile ~/.ssh/id_bup

Set HostName to the IP or domain name of the host running the docker container.
Set User to bup. This is hard-coded in the container.
Set Port to whatever port you mapped out in docker-compose.yml. If the ports: line in docker-compose.ymlis - 1234:2022, the correct value for Port in .ssh/config is 1234.
Set IdentityFile to whereever id_bup is located (see above).

Now you need to connect to the bup server container once for each client. This is both to spot issues with your SSH configuration (such as wrong permissions on the id_bup file) and to save the SSH host key of the container as known key:

ssh BupServer

If this prompts you for a password, something is wrong in your configuration – possibly, your are connecting to the wrong SSH host since the bup server container has password authentication disabled.

5. Connect using bup

Every client will need bup to be installed. See How to install bup on Ubuntu 22.04 and similar posts.

You have to understand that bup will need both a local directory (called the index) and a directory on the bup server called destination directory.  You have to use one index directory and one destination directory per backup project. What you define as backup project is up to you, but I strongly recommend to use one backup project per application you backup, in order to have data locality: Backups from one application belong together.

By convention, the /bup directory on the server (i.e. container) is dedicated for this purpose (and mapped to a directory or volume outside of the container).

On the local host, I recommend using either /var/lib/bup/project.index.bup or ~/bup/project.index.bup and let bup auto-create project-specific directories from there. If you use a special user on the client to do backups, you can also place the indexes. If the index is lost, this is not an issue as long as the backup works (it just will take a few minutes to check all files again). You should not backup the index directory.

There is no requirement for the .bup or .index.bup suffix but if you use it, it will allow you to quickly discern what a directory is and whether it is important or nor.

In order to use bup, you first need to initialize the directories. You can do this multiple times without any issue, so I do it at the start of each of my backup scripts.

bup -d ~/buptest.index.bup init -r BupServer:/bup/buptest.bup

After that, you can start backing up. Generally this is done by first running bup index (this operation is local-only) and then running bup save (which saves the backup on the bup remote server).

bup -d ~/buptest.index.bup index . && bup save -r BupServer:/bup/buptest.bup -9 --strip-path $(pwd) -n mybackup .

Some parameters demand further explanation:

  • -9: Maximum compression. bup is so fast that it hardly makes a difference but it saves a ton of disk space especially for text-like data.
  • --strip-path $(pwd) If you backup a directory /home/uli/Documents/ with a file /home/uli/Documents/Letters/Myletter.txt this makes bup save the backup of said file under the name Letters/Myletter.txt instead of  /home/uli/Documents/Letters/Myletter.txt.
  • -n mybackup. The name of this backup. This allows you to separate different backups in a single repository.

6. Let’s restore!

You might want to say hopefully I’ll never need to restore. WRONG. You need to restore right now, and you need to restore regularly, as a test that if you actually need to recover data by restoring, it will actually work.

In order to do this, we’ll first need to get access to the folder where. This is typically stored on some kind of Linux server anyway, so just install bup there. In our example above, the directory we’ll work with is called buptest.bup.

There are two conventient ways to view bup backups:

  1. Use bup web and open your browser at http://localhost:8080 to view the backup data (including history):
    bup -d buptest.bup web
  2. Use bup fuse to mount the entire tree including history to a directory such as /tmp/buptest:
    mkdir -p  /tmp/buptest && bup -d buptest.bup fuse /tmp/buptest

     

Posted by Uli Köhler in bup, Container, Docker

WordPress backup script using bup (bup remote)

This script backups a WordPress installation (including data,base files & directories, excluding cache) to a bup remote server running on 10.1.2.3. You need to ensure passwordless access to that server.

It is based on automated extraction of database host, username & password, see How to grep for WordPress DB_NAME, DB_USER, DB_PASSWORD and DB_HOST in wp-config.php for more details.

#!/bin/bash
export NAME=$(basename $(pwd))
export BUP_DIR=/var/bup/$NAME.bup
export REMOTE_BUP_DIR=/bup-backups/$NAME.bup
export REMOTE_SERVER=10.1.2.3
export BUP_REMOTE=$REMOTE_SERVER:$REMOTE_BUP_DIR

# Init
bup -d $BUP_DIR init -r $BUP_REMOTE
# Save MariaDB dump (extract MariaDB config from wp-config.php)
DB_NAME=$(grep -oP "define\(['\"]DB_NAME['\"],\s*['\"]\K[^'\"]+(?=[\'\"]\s*\)\s*;)" wp-config.php)
DB_USER=$(grep -oP "define\(['\"]DB_USER['\"],\s*['\"]\K[^'\"]+(?=[\'\"]\s*\)\s*;)" wp-config.php)
DB_PASSWORD=$(grep -oP "define\(['\"]DB_PASSWORD['\"],\s*['\"]\K[^'\"]+(?=[\'\"]\s*\)\s*;)" wp-config.php)
DB_HOST=$(grep -oP "define\(['\"]DB_HOST['\"],\s*['\"]\K[^'\"]+(?=[\'\"]\s*\)\s*;)" wp-config.php)
mysqldump -h$DB_HOST -u$DB_USER -p$DB_PASSWORD $DB_NAME | bup -d $BUP_DIR split -n $NAME-$DB_NAME.sql

# Save wordpress directory
bup -d $BUP_DIR index --exclude wp-content/cache --exclude wp-content/uploads/cache . && bup save -r $BUP_REMOTE -9 --strip-path $(pwd) -n $NAME .

# OPTIONAL: Add par2 information
#   This is only recommended for backup on unreliable storage or for extremely critical backups
#   If you already have bitrot protection (like BTRFS with regular scrubbing), this might be overkill.
# Uncomment this line to enable:
# bup on $REMOTE_SERVER -d $REMOTE_BUP_DIR fsck -g

# OPTIONAL: Cleanup old backups
bup on $REMOTE_SERVER -d $REMOTE_BUP_DIR prune-older --keep-all-for 1m --keep-dailies-for 6m --keep-monthlies-for forever -9 --unsafe

 

Posted by Uli Köhler in bup, Wordpress

How to install bup on Alpine Linux

In order to install the bup backup software on Alpine Linux, you currently have to compile it yourself.

First, install the prerequisites:

apk add bash make g++ python3-dev git automake autoconf par2cmdline py3-pip && pip3 install wheel && pip3 install pyxattr

Now we can clone bup:

git clone -b 0.33 --depth 1 https://github.com/bup/bup

and build:

cd bup && ./configure && make -j4 install PREFIX=/usr

After this, bup should be installed in /usr/bin/bup. The bup clone directory we created in the first step is not needed any more.

Posted by Uli Köhler in Alpine Linux, bup

How to fix bup make AssertionError: assert not date.startswith(b’$Format’)

Problem:

While trying to build bup using make you see the following error message:

set -e; bup_ver=$(./bup version); \
echo "s,%BUP_VERSION%,$bup_ver,g" > Documentation/substvars.tmp; \
echo "s,%BUP_DATE%,$bup_ver,g" >> Documentation/substvars.tmp
Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/uli/dev/deb-buildscripts/bup/lib/bup/main.py", line 181, in <module>
    cmd_module = import_module('bup.cmd.'
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/home/uli/dev/deb-buildscripts/bup/lib/bup/cmd/version.py", line 5, in <module>
    from bup import options, version
  File "/home/uli/dev/deb-buildscripts/bup/lib/bup/version.py", line 20, in <module>
    assert not date.startswith(b'$Format')
AssertionError
make: *** [GNUmakefile:252: Documentation/substvars] Fehler 1

Solution:

This error occurs because bup can’t determine the correct version & date for bup. This happens because you deleted the .git directory which bup needs in order to determine its version.

You can create a clone of bup with the .git directory intact by just cloning, for example, the specific version you want to build:

git clone -b 0.33 --depth 1 https://github.com/bup/bup

and then run

./configure
make -j4

as usual

Posted by Uli Köhler in bup

BUP MySQL backup using mysqldump without intermediary file

BUP is a nice git-based backup tool that is both free and easy to use and saves space by differential backups.

You can also use bup to backup a MySQL database using mysqldump directly instead of first dumping to a .sql file and then backing up the file.

This is possible by piping the mysqldump output directly into bup split:

mysqldump [...]| bup -d $BUP_DIR split -n mysqldump.sql

By using -n mysqldump.sql you are telling bup that the file created by the dumping should be named mysqldump.sql in the backup.

Full example:

export BUP_DIR=/var/lib/bup/mysql.bup
export MARIADB_ROOT_PASSWORD=piahaen9ehilei0Ieneirohthue4Iu

bup -d $BUP_DIR init
mysqldump -uroot -p${MARIADB_ROOT_PASSWORD} --all-databases | bup -d $BUP_DIR split -n mysqldump.sql

 

Posted by Uli Köhler in bup

Local redmine backup using bup (docker-compose compatible)

This script uses bupto backup your docker-compose based redmine installation to a local bup folder e.g. in /var/lib/bup/my-redmine.bup:

#!/bin/bash
# Auto-determine the name from the directory name
# /opt/my-redmine => $NAME=my-redmine => /var/lib/bup/my-redmine.bup
export NAME=$(basename $(pwd))
export BUP_DIR=/var/lib/bup/$NAME.bup
bup_directory() {
        echo "BUPing $1"
        bup -d $BUP_DIR index $1 && bup save -9 --strip-path $(pwd) -n $1 $1
}
# Init
bup -d $BUP_DIR init
# Save MariaDB
source .env && docker-compose exec -T mariadb mysqldump -uroot -p${MARIADB_ROOT_PASSWORD} --all-databases | bup -d $BUP_DIR split -n $NAME-mariadb.sql
# Save directories
bup_directory redmine_data
bup_directory redmine_themes
# Backup self
bup_directory backup.sh
bup_directory docker-compose.yml
# OPTIONAL: Add par2 information
#   This is only recommended for backup on unreliable storage or for extremely critical backups
#   If you already have bitrot protection (like BTRFS with regular scrubbing), this might be overkill.
# Uncomment this line to enable:
# bup fsck -g

# OPTIONAL: Cleanup old backups
bup -d $BUP_DIR prune-older --keep-all-for 1m --keep-dailies-for 6m --keep-monthlies-for forever -9 --unsafe

It will backup:

  • MySQL data from inside redmine using mysqldump
  • The redmine_data folder
  • The redmine_themes folder
  • The backup script backup.sh itself
  • docker-compose.yml

Place it in the same folder where docker-compose.yml is located.

The script is compatible with our previous post How to create a systemd backup timer & service in 10 seconds

Posted by Uli Köhler in bup, Docker