How should I backup & restore docker named volumes

Question

I'm a little bit confused with the functionnality of named volumes in a docker compose file specifically when it comes to backup/restore my app.

I'm actually testing this dockercompose file :

      version: '2'
      services:
          django:
              build: 
                  context: "{{ build_dir }}/docker/django"
              depends_on:
                  - db
              environment:
                  [...]
              volumes:
                  - code:/data/code
                  - www:/var/www
                  - conf:/data/conf
              networks:
                  - front
                  - db
              expose:
                  - "8080"
              entrypoint: "/init"
          db:
              build:
                  context: "{{ build_dir }}/docker/postgres" 
              environment:
                  [...]
              volumes:
                  - data:/var/lib/postgresql/data
              networks:
                  - db

      volumes:
          data:
          www:
          code:
          conf:

      networks:
          front:
              external:
                  name: "proxy_nw"

As the documentation said, I tried to use named volume instead of data only container. But how am I suppose to backup my data ?

With a data only container I would have done a docker run --rm --volume-from DOC backup_container save which is really easy.

Now I read in this topic that I should use something like docker run --rm --volume data --volume www --volume code --volume conf backup_container save. This is not so simple because I have many applications with different types and names of volumes so it means that my command to save my data would have to be different for each application. It complicate automation process.

Edit: Actually this syntaxe docker run --volume data --volume www container_image my_command is not correct. It needs the mountpoint inside the container, so it would be docker run --volume data:/somewhere --volume www:/somewhereelse container_image my_command. So it's even more complicated to use with a backup container.

So, what are the best practices in this case ? Should I use just one named volume for all my containers ?

Answer 1

Actually it should be done same way as written in official documentation. Data volume container stores it's data in "virtual root", so you should backup with next command:

docker run --rm \ 
  --volume [DOCKER_COMPOSE_PREFIX]_[VOLUME_NAME]:/[TEMPORARY_DIRECTORY_TO_STORE_VOLUME_DATA] \
  --volume $(pwd):/[TEMPORARY_DIRECTORY_TO_STORE_BACKUP_FILE] \
  ubuntu \
  tar cvf /[TEMPORARY_DIRECTORY_TO_STORE_BACKUP_FILE]/[BACKUP_FILENAME].tar /[TEMPORARY_DIRECTORY_TO_STORE_VOLUME_DATA]

where:

• --rm means that the image created for this run command will be cleaned up
• DOCKER_COMPOSE_PREFIX in default is your project directory name
• VOLUME_NAME is the data-volume container name from compose file
• TEMPORARY_DIRECTORY_TO_STORE_VOLUME_DATA is a directory to mount your volume data
• TEMPORARY_DIRECTORY_TO_STORE_BACKUP_FILE is a directory virtually mapped to your current directory, where the backup will be placed
• BACKUP_FILENAME - a name of backup file (you find it in current directory)
• ubuntu - you may change image type to another container with tar :)

Get data back into the volume(restore):

docker run --rm \ 
  --volume [DOCKER_COMPOSE_PREFIX]_[VOLUME_NAME]:/[TEMPORARY_DIRECTORY_STORING_EXTRACTED_BACKUP] \
  --volume $(pwd):/[TEMPORARY_DIRECTORY_TO_STORE_BACKUP_FILE] \
  ubuntu \
  tar xvf /[TEMPORARY_DIRECTORY_TO_STORE_BACKUP_FILE]/[BACKUP_FILENAME].tar -C /[TEMPORARY_DIRECTORY_STORING_EXTRACTED_BACKUP] --strip 1

where:

• TEMPORARY_DIRECTORY_STORING_EXTRACTED_BACKUP is a directory where the extracted files will be copied to (this is linked with the volume and will therefore write to it)
• -C - tell tar where to extract the contents
• --strip 1 - remove leading path elements (e.g. the parent directory if the backup contents are located in a /temp folder or similar)

Answer 2

Base on this answer. I made an easy tool here： docker_named_volume_backup

first run command docker volume ls to list the named volume you want to backup.

then for backup

#sudo backup_docker_volume.sh <volumn_name> <tar_file>
sudo bash ./backup_docker_volume.sh codimd_database-data backup1.tar

for restore

#sudo restore_docker_volume.sh <volumn_name> <tar_file>
sudo bash ./restore_docker_volume.sh codimd_database-data backup1.tar

Answer 3

Here is an example of how I backup and restore mongo volume data as docker image.

# let a9c25f42ccca be mongo container id    
docker stop a9c25f42ccca

# create a temp container with volume taken from mongo 
# make a local tar archive inside it
docker run --name temp_backup --volumes-from a9c25f42ccca ubuntu tar cvf /mongo_backup.tar /data/db

docker start a9c25f42ccca

# make an image and remove temp container
docker commit temp_backup my_mongo_backup
docker rm temp_backup

# push image with backup to remote registry if needed
docker push my_mongo_backup

And restoring.

#pull image if needed
docker pull my_mongo_backup

docker stop a9c25f42ccca

# run transient container out from image with backup
# take volume from mongo
# clear any existing data
# restore data from tar arhcive
docker run --rm --volumes-from a9c25f42ccca my_mongo_backup bash -c "rm -rf /data/db/* && tar xvf /mongo_backup.tar -C /data --strip 1"

docker start a9c25f42ccca

Answer 4

I finally changed my approach. I parse the volumes of my containers looking for the mountpoints and I backup everything in a separate folder in the tar.

So I'm not using a container to do it but an external script. I don't know if it's a better approach but it works well.

Answer 5

As stated in official documentation there is no out of the box single command to backup/restore volume(s). The one should do it on it's own by mounting a volume to some container and packing the content in some suitable way as given by manuals:

docker run --rm --volumes-from dbstore -v $(pwd):/backup ubuntu tar cvf /backup/backup.tar /dbdata

For plain files stored in a volume this could be enough, but for database it's better to use some dump producing tools. Like mongodump/mongorestore or pg_dump/pg_restore to deal with metadata instead of binary files. As binary files backup has tighter coupling to certain database version.

Some time ago I encountered this issue and developed simple scripts to cover my needs. But with growing of my apps and theirs number I decided to make it common. And recently I open sourced a tool I developed that optionally can upload backups to a Google Drive folder.

pip install cobra-archiver

To backup all the volumes.

cobra backup build

To backup all the volumes but skip-this-one and put the backup in ./backup directory one can use this.

cobra backup build --exclude skip-this-one --backup-dir ./backup

To backup and push to Google Drive.

cobra backup build --push \
    --creds /path/to/google-service-acc-key.json \
    --folder-id google-drive-folder-id \
    --exclude skip-this-volume

Additionally it can backup plain file system directories like this

cobra backup build --dir ./backup/me

To see the cli description on certain command the line is as follows.

$ cobra backup build --help

As well as backing up it allows easy restoration of previously backed up stuff.

cobra backup pull --latest --restore \
    --creds /path/to/google-service-acc-key.json \
    --folder-id google-drive-folder-id

It's also possible to have "dump" database backup automated. To achieve this one can use hooks. Please see the details on github. Real example is available in e2e test sources.