Containers

Backup Docker Volumes with Duplicati and Docker Compose

Securely backup all your Docker volume data using Duplicati with Docker Compose. Encryption, scheduling, FTP, SSH, WebDAV, OneDrive, Amazon, Google & more!

Backing up your Docker persistent volumes is a crucial part of disaster recovery for container filesystem resources that exist on your Docker hosts. This guide looks into installing Duplicati with Docker Compose. We will focus on backing up container data and volume data.

What is Duplicati?

Duplicati is an open-source backup client solution to create backup jobs that perform incremental backups of all the volumes and is compatible with numerous file transfer protocols, including FTP, SSH, WebDAV, and even cloud services. Cloud services include Microsoft OneDrive, Amazon Cloud Drive & S3, Google Drive, Box.com, Mega, hubiC, and others.

Duplicati features

It also has the following features:

  • Run encrypted incremental backups
  • Backups using protocols like ftp ssh
  • Create local backups and store local backups to a directory
  • Create and send backup of your Docker volumes to the cloud
  • Backup encryption
  • Scheduling and retention

Duplicati Docker Compose install

Let’s look at the steps you need to do to install Duplicati on your Docker host. Most likely, you already have Docker installed along with Docker Compose if you want to back up your host’s Docker volumes. However, remember that you will need to have Docker and Docker Compose installed as a prerequisite.

Preparing the Duplicati Docker Environment

Run through the following commands below to install Docker and Docker Compose in Ubuntu Server.

# Add Docker's official GPG key:
sudo apt-get update
sudo apt-get install ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpg

# Add the repository to Apt sources:
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
  $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update

# Install the latest version of Docker and Docker Compose
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

Steps for Duplicati Docker Installation

  1. Docker Compose File Creation: Initiate by creating a docker-compose.yml file. This file is the heart of your Duplicati Docker setup, defining how your Docker container will operate.
  2. Defining the Duplicati Docker Service: Use the Docker image tag lscr.io linuxserver duplicati:latest for the most updated version. This tag ensures you’re using a multi-arch image, suitable for different systems​. The available image tags also include the “development” image tag that is the canary releases of Duplicati.
  3. Configuring Environment Variables: Use the appropriate environment variable for Duplicati, such as PUID and PGID, which defines the permissions within the Docker environment.
  4. Mapping Volumes and Ports: Properly map Docker volumes for storing Duplicati’s configuration and backup files. Additionally, configure the ports to access Duplicati’s user interface.
  5. Launching Duplicati Docker Container: Execute the docker-compose up command to start your Duplicati service within the Docker container.

As an example, you can use the below Docker Compose code. Note below, under the volumes configuration. This configuration line is where you will setup your volumes for Duplicati, and then the volumes for where you want to store backups and the paths to Docker volumes that you want to back up.

---
version: "2.1"
services:
  duplicati:
    image: lscr.io/linuxserver/duplicati:latest
    container_name: duplicati
    environment:
      - PUID=1000
      - PGID=1000
      - TZ=Etc/UTC
      - CLI_ARGS= #optional
    volumes:
      - /path/to/appdata/config:/config
      - /path/to/backups:/backups
      - /path/to/source:/source
    ports:
      - 8200:8200
    restart: unless-stopped

After you create the docker-compose file, you can bring up the containers using Docker compose:

docker-compose up -d

Docker CLI installation

You can also use the Docker CLI to install Duplicati instead of Docker Compose. To use the Docker CLI, use the following:

docker run -d \
  --name=duplicati \
  -e PUID=1000 \
  -e PGID=1000 \
  -e TZ=Etc/UTC \
  -e CLI_ARGS= `#optional` \
  -p 8200:8200 \
  -v /path/to/appdata/config:/config \
  -v /path/to/backups:/backups \
  -v /path/to/source:/source \
  --restart unless-stopped \
  lscr.io/linuxserver/duplicati:latest

Browse to Duplicati web interface

Now that we have the container spun, we can browse out to the port configured for the web interface. In this case, we accepted the default port 8200.

You will be asked if you are using a multi-user system or a single user.

First run setup
First run setup

Now, we can view the interface. It is very intuitive and simple.

Viewing the simple duplicati interface
Viewing the simple duplicati interface

Let’s see how to add a backup.

Create backup jobs in Duplicati

The process to create a backup job is easy. This is done using the Add Backup wizard in Duplicati.

When you click the Add backup option, it will launch the Add a new backup wizard. Here we will select Configure a new backup.

Add a new backup job
Add a new backup job

This is a 5-step process. The first screen has you name the backup job. It defaults to AES-256 encryption built in. You will need to generate or create your own password for the encryption process.

General backup settings screen
General backup settings screen

Next, we will select the destination. If you click the drop-down on the Storage type, we get an idea of just how many options are available with Duplicati.

Storage types for backup storage in duplicati
Storage types for backup storage in duplicati

I am choosing the Local folder or drive for the test. The destination I am choosing should look familiar from the Docker Compose file.

Configuring destination storage
Configuring destination storage

There are many advanced options to use as well if you choose.

Advanced options
Advanced options

On step 3, we choose the Source data. Scroll down in the folder browser to source. This is the source folder we configured in our Docker Compose file.

Select source data
Select source data

On step 4 choose your Schedule. Below are the defaults when you get to this step. By default, it runs every day.

Setting the schedule for the backups
Setting the schedule for the backups

Finally, on the options screen in step 5, we set the remote volume size and the backup retention. Configure your settings here and click Save.

Backup options
Backup options

What is the remote volume size?

Understanding Block Size

  • What’s a Block? Think of blocks as small pieces of your files. Duplicati breaks your files into these pieces, or “chunks”, for backup.
  • Default Chunk Size: By default, each chunk is 100 kilobytes (kb). If a file is smaller than this, or doesn’t split evenly into 100kb chunks, you’ll end up with a smaller final chunk.
  • Important: Once you back up with a certain chunk size, you can’t change it later. Duplicati will stop you from doing this to avoid errors.

Choosing Larger Block Sizes

  • Fewer, Bigger Chunks: Larger chunk sizes mean you get fewer but bigger pieces. This can be good if your files are big. But, there’s a catch with smaller sizes; they can slow things down.
  • Speed and Storage: Bigger chunks mean less clutter in Duplicati’s memory (like a smaller index in a book), especially if you’re not using a super-fast SSD drive.
  • Restoration and Storage Impact: Bigger chunks are handy when restoring files, as they can be put back together faster. However, if you change just a small part of a file, Duplicati has to replace the whole chunk, which could use more storage and internet data.
  • Deduplication Efficiency: Bigger chunks make it harder for Duplicati to find duplicate pieces, as each chunk contains more data.

Running a Duplicati backup job

Now that we have our Duplicati backup job created, we can run the job.

Run the backup job now
Run the backup job now

The backup job begins to run. In addition to the progress bar, it will tell you how many files and the data size left in the backup job which is a nice touch.

Duplicati backup progress
Duplicati backup progress

It took my backup job around 6:30 minutes to run.

Backup completed successfully
Backup completed successfully

Restore Docker volumes using Duplicati

Let’s look at the Duplicati restore wizard to restore data that we have captured with the Duplicati backup.

The first screen asks you where you want to restore from. The options are:

  • Direct restore from backup files
  • Restore from configuration
  • Your Backup name
Select where you want to restore from
Select where you want to restore from

Next, choose the data you want to restore and then click Continue.

Select which files you want to restore
Select which files you want to restore

Finally, select your Restore options. These include:

  • Where do you want to restore the files to?
  • How do you want to handle existing files
  • Permissions options
Select restore options in duplicati
Select restore options in duplicati

Frequently Asked Questions

How does Duplicati ensure data security during backups?

Duplicati uses encryption, which is like putting a lock on your data when it’s backed up. This encryption happens even before your data leaves your computer, adding a layer of security, especially if you’re using cloud storage like Google Drive or Microsoft OneDrive.

Can Duplicati back up open or in-use files effectively?

Duplicati is capable of backing up open or in-use files. However, for a more flawless process, especially in complex environments like Docker containers or when dealing with database files, it’s recommended to use specific methods like snapshots or scheduling backups during low-usage periods.

Is it possible to automate backup jobs with Duplicati in a Docker environment?

As we saw earlier, Duplicati allows you to set schedules for your backups. You can automate backups to run daily, weekly, or at any custom interval, ensuring regular updates to your backup without manual intervention. This is particularly useful in Docker environments where consistent data backup is crucial.

How does Duplicati handle large files and what’s the impact on backup and restoration?

For large files, Duplicati’s chunk and volume size settings play a significant role. Larger chunk sizes can make the backup and restoration of big files more efficient, as fewer, larger chunks are easier to manage and transfer. However, this might increase the storage space used if only small parts of a file change frequently.

What are some best practices for configuring Duplicati in a Docker Compose setup?

Make sure to set the appropriate environment variables, correctly mapping Docker volumes for data and configuration files, and choosing optimal block and volume sizes based on your file sizes and backup frequency. Also, ensuring regular monitoring and updates of your Duplicati Docker container helps maintain the efficiency of your backup system.

How does changing the block size affect incremental backups in Duplicati?

Incremental backups in Duplicati become more efficient with an optimal block size. A smaller block size might increase the frequency of incremental updates, as smaller changes in files are detected and backed up. However, this can also lead to increased storage use. On the other hand, a larger block size might reduce the frequency of backups but can miss smaller file changes.

Can Duplicati work with various file transfer protocols for backups?

It supports various file transfer protocols like FTP, SSH, and others. This allows users to choose their preferred method for transferring backup data to different storage solutions, be it local storage, a remote server, or cloud services.

Are there any considerations for using Duplicati on non-SSD (spinning) disks?

When using Duplicati on non-SSD drives, it’s important to consider the impact of block size on performance. Larger block sizes can lead to faster lookup times, as there are fewer blocks to manage. This can offset traditional hard drives’ slower read/write speeds compared to SSDs.

In Docker Compose, how does one configure Duplicati to back up Docker volumes specifically?

To back up Docker volumes with Duplicati, you need to map the Docker volumes correctly in your docker-compose.yml file. Make sure the volumes containing your crucial data are correctly referenced in the Duplicati service configuration, allowing Duplicati to access and back up this data.

What’s the impact of volume size settings on network bandwidth during backups?

The volume size in Duplicati impacts how much data is sent in each transfer. Larger volumes mean more data is sent at once, which can be efficient but might use more bandwidth. Smaller volumes reduce bandwidth usage per transfer but might lead to more frequent transfers.

Wrapping up Duplicati Docker installation

Duplicati is a slick way to back up your Docker volumes and use modern storage locations to store your data. The Duplicati solution is easy to stand up, it has a ton of features, it is open-source, and can target cloud storage services. It really is one of the most fully-featured Docker container volume backup solutions you will find, outside of some type of commercial offering. I think it is a great way to bolster normal backup routines of an entire Docker host virtual machine by having backups of your Docker data by Duplicati stored in a different location.

Subscribe to VirtualizationHowto via Email 🔔

Enter your email address to subscribe to this blog and receive notifications of new posts by email.

Brandon Lee

Brandon Lee is the Senior Writer, Engineer and owner at Virtualizationhowto.com and has over two decades of experience in Information Technology. Having worked for numerous Fortune 500 companies as well as in various industries, Brandon has extensive experience in various IT segments and is a strong advocate for open source technologies. Brandon holds many industry certifications, loves the outdoors and spending time with family.

Related Articles

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.