Backup Docker Volumes with Duplicati and Docker Compose
Backing up your Docker persistent volumes is a crucial part of disaster recovery for container filesystem resources that exist on your Docker hosts. This guide looks into installing Duplicati with Docker Compose. We will focus on backing up container data and volume data.
Table of contents
What is Duplicati?
Duplicati is an open-source backup client solution to create backup jobs that perform incremental backups of all the volumes and is compatible with numerous file transfer protocols, including FTP, SSH, WebDAV, and even cloud services. Cloud services include Microsoft OneDrive, Amazon Cloud Drive & S3, Google Drive, Box.com, Mega, hubiC, and others.
Duplicati features
It also has the following features:
- Run encrypted incremental backups
- Backups using protocols like ftp ssh
- Create local backups and store local backups to a directory
- Create and send backup of your Docker volumes to the cloud
- Backup encryption
- Scheduling and retention
Duplicati Docker Compose install
Let’s look at the steps you need to do to install Duplicati on your Docker host. Most likely, you already have Docker installed along with Docker Compose if you want to back up your host’s Docker volumes. However, remember that you will need to have Docker and Docker Compose installed as a prerequisite.
Preparing the Duplicati Docker Environment
Run through the following commands below to install Docker and Docker Compose in Ubuntu Server.
# Add Docker's official GPG key:
sudo apt-get update
sudo apt-get install ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpg
# Add the repository to Apt sources:
echo \
"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
$(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update
# Install the latest version of Docker and Docker Compose
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-pluginSteps for Duplicati Docker Installation
- Docker Compose File Creation: Initiate by creating a docker-compose.yml file. This file is the heart of your Duplicati Docker setup, defining how your Docker container will operate.
- Defining the Duplicati Docker Service: Use the Docker image tag lscr.io linuxserver duplicati:latest for the most updated version. This tag ensures you’re using a multi-arch image, suitable for different systems​. The available image tags also include the “development” image tag that is the canary releases of Duplicati.
- Configuring Environment Variables: Use the appropriate environment variable for Duplicati, such as PUID and PGID, which defines the permissions within the Docker environment.
- Mapping Volumes and Ports: Properly map Docker volumes for storing Duplicati’s configuration and backup files. Additionally, configure the ports to access Duplicati’s user interface.
- Launching Duplicati Docker Container: Execute the docker-compose up command to start your Duplicati service within the Docker container.
As an example, you can use the below Docker Compose code. Note below, under the volumes configuration. This configuration line is where you will setup your volumes for Duplicati, and then the volumes for where you want to store backups and the paths to Docker volumes that you want to back up.
---
version: "2.1"
services:
duplicati:
image: lscr.io/linuxserver/duplicati:latest
container_name: duplicati
environment:
- PUID=1000
- PGID=1000
- TZ=Etc/UTC
- CLI_ARGS= #optional
volumes:
- /path/to/appdata/config:/config
- /path/to/backups:/backups
- /path/to/source:/source
ports:
- 8200:8200
restart: unless-stoppedAfter you create the docker-compose file, you can bring up the containers using Docker compose:
docker-compose up -dDocker CLI installation
You can also use the Docker CLI to install Duplicati instead of Docker Compose. To use the Docker CLI, use the following:
docker run -d \
--name=duplicati \
-e PUID=1000 \
-e PGID=1000 \
-e TZ=Etc/UTC \
-e CLI_ARGS= `#optional` \
-p 8200:8200 \
-v /path/to/appdata/config:/config \
-v /path/to/backups:/backups \
-v /path/to/source:/source \
--restart unless-stopped \
lscr.io/linuxserver/duplicati:latestBrowse to Duplicati web interface
Now that we have the container spun, we can browse out to the port configured for the web interface. In this case, we accepted the default port 8200.
You will be asked if you are using a multi-user system or a single user.
Now, we can view the interface. It is very intuitive and simple.
Let’s see how to add a backup.
Create backup jobs in Duplicati
The process to create a backup job is easy. This is done using the Add Backup wizard in Duplicati.
When you click the Add backup option, it will launch the Add a new backup wizard. Here we will select Configure a new backup.
This is a 5-step process. The first screen has you name the backup job. It defaults to AES-256 encryption built in. You will need to generate or create your own password for the encryption process.
Next, we will select the destination. If you click the drop-down on the Storage type, we get an idea of just how many options are available with Duplicati.
I am choosing the Local folder or drive for the test. The destination I am choosing should look familiar from the Docker Compose file.
There are many advanced options to use as well if you choose.
On step 3, we choose the Source data. Scroll down in the folder browser to source. This is the source folder we configured in our Docker Compose file.
On step 4 choose your Schedule. Below are the defaults when you get to this step. By default, it runs every day.
Finally, on the options screen in step 5, we set the remote volume size and the backup retention. Configure your settings here and click Save.
What is the remote volume size?
Understanding Block Size
- What’s a Block? Think of blocks as small pieces of your files. Duplicati breaks your files into these pieces, or “chunks”, for backup.
- Default Chunk Size: By default, each chunk is 100 kilobytes (kb). If a file is smaller than this, or doesn’t split evenly into 100kb chunks, you’ll end up with a smaller final chunk.
- Important: Once you back up with a certain chunk size, you can’t change it later. Duplicati will stop you from doing this to avoid errors.
Choosing Larger Block Sizes
- Fewer, Bigger Chunks: Larger chunk sizes mean you get fewer but bigger pieces. This can be good if your files are big. But, there’s a catch with smaller sizes; they can slow things down.
- Speed and Storage: Bigger chunks mean less clutter in Duplicati’s memory (like a smaller index in a book), especially if you’re not using a super-fast SSD drive.
- Restoration and Storage Impact: Bigger chunks are handy when restoring files, as they can be put back together faster. However, if you change just a small part of a file, Duplicati has to replace the whole chunk, which could use more storage and internet data.
- Deduplication Efficiency: Bigger chunks make it harder for Duplicati to find duplicate pieces, as each chunk contains more data.
Running a Duplicati backup job
Now that we have our Duplicati backup job created, we can run the job.
The backup job begins to run. In addition to the progress bar, it will tell you how many files and the data size left in the backup job which is a nice touch.
It took my backup job around 6:30 minutes to run.
Restore Docker volumes using Duplicati
Let’s look at the Duplicati restore wizard to restore data that we have captured with the Duplicati backup.
The first screen asks you where you want to restore from. The options are:
- Direct restore from backup files
- Restore from configuration
- Your Backup name
Next, choose the data you want to restore and then click Continue.
Finally, select your Restore options. These include:
- Where do you want to restore the files to?
- How do you want to handle existing files
- Permissions options
Frequently Asked Questions
Duplicati uses encryption, which is like putting a lock on your data when it’s backed up. This encryption happens even before your data leaves your computer, adding a layer of security, especially if you’re using cloud storage like Google Drive or Microsoft OneDrive.
Duplicati is capable of backing up open or in-use files. However, for a more flawless process, especially in complex environments like Docker containers or when dealing with database files, it’s recommended to use specific methods like snapshots or scheduling backups during low-usage periods.
As we saw earlier, Duplicati allows you to set schedules for your backups. You can automate backups to run daily, weekly, or at any custom interval, ensuring regular updates to your backup without manual intervention. This is particularly useful in Docker environments where consistent data backup is crucial.
For large files, Duplicati’s chunk and volume size settings play a significant role. Larger chunk sizes can make the backup and restoration of big files more efficient, as fewer, larger chunks are easier to manage and transfer. However, this might increase the storage space used if only small parts of a file change frequently.
Make sure to set the appropriate environment variables, correctly mapping Docker volumes for data and configuration files, and choosing optimal block and volume sizes based on your file sizes and backup frequency. Also, ensuring regular monitoring and updates of your Duplicati Docker container helps maintain the efficiency of your backup system.
Incremental backups in Duplicati become more efficient with an optimal block size. A smaller block size might increase the frequency of incremental updates, as smaller changes in files are detected and backed up. However, this can also lead to increased storage use. On the other hand, a larger block size might reduce the frequency of backups but can miss smaller file changes.
It supports various file transfer protocols like FTP, SSH, and others. This allows users to choose their preferred method for transferring backup data to different storage solutions, be it local storage, a remote server, or cloud services.
When using Duplicati on non-SSD drives, it’s important to consider the impact of block size on performance. Larger block sizes can lead to faster lookup times, as there are fewer blocks to manage. This can offset traditional hard drives’ slower read/write speeds compared to SSDs.
To back up Docker volumes with Duplicati, you need to map the Docker volumes correctly in your docker-compose.yml file. Make sure the volumes containing your crucial data are correctly referenced in the Duplicati service configuration, allowing Duplicati to access and back up this data.
The volume size in Duplicati impacts how much data is sent in each transfer. Larger volumes mean more data is sent at once, which can be efficient but might use more bandwidth. Smaller volumes reduce bandwidth usage per transfer but might lead to more frequent transfers.
Wrapping up Duplicati Docker installation
Duplicati is a slick way to back up your Docker volumes and use modern storage locations to store your data. The Duplicati solution is easy to stand up, it has a ton of features, it is open-source, and can target cloud storage services. It really is one of the most fully-featured Docker container volume backup solutions you will find, outside of some type of commercial offering. I think it is a great way to bolster normal backup routines of an entire Docker host virtual machine by having backups of your Docker data by Duplicati stored in a different location.
