Backing up Docker services
A "just in case" copy is useless if you can't restore from it. Let's look at how to take consistent copies of a Docker service's data, encrypt them and keep them in two places.
Consistency over speed
Simply copying the volume files of a running database gives a smeared snapshot: some pages have already changed, others haven't. The database may reject such a copy. The right way is to take a dump with the database's own tool, and copy file volumes separately.
#!/usr/bin/env bash
set -euo pipefail
DAY=$(date +%F)
TMP=$(mktemp -d)
# consistent database dump straight from the container
docker exec db pg_dump -U postgres app > "$TMP/db.sql"
# application files from the volume
docker run --rm -v app_data:/data -v "$TMP":/out alpine \
tar czf /out/data.tgz -C /data .
# a single encrypted archive
tar cf - -C "$TMP" . | openssl enc -aes-256-cbc -pbkdf2 -salt \
-pass file:/root/.backup-pass -out "/backups/app-$DAY.tar.enc"
rm -rf "$TMP"Two storage locations
A copy next to the service won't help if the whole location goes down. Keep a second copy on another node or with another provider. We send the encrypted archive to a separate server right after it's created and make sure both copies are fresh.
Test recovery
A backup you've never restored is an assumption, not a guarantee. Every so often, bring a copy up on a test node and confirm the service starts. That's the only way to know the plan works before you need it.