Setting up Duplicacy on UnRAID with Backblaze B2

  • 2019-11-10 (Sun)
  • 6-minute read
!
Warning: This post is over 365 days old. The information may be out of date.

In this post we’re going to look at Duplicacy, a tool that allows you to backup large amounts of data to any storage provider.

So what is Duplicacy?

Duplicacy is a backup/restore tool for servers. They have a free CLI version and a paid GUI version, but you probably only need the CLI tool unless you’re extremely scared of the terminal.

Big warning before we begin

  • Duplicacy expects sole access to the backup destination. That means, if a backup is running and you issue a prune command on another server, it may result in a loss of ALL of your data! Always make sure that only one instance of Duplicacy is operating at a given moment.
  • RAID is not a backup. If someone tells you they have a bulletproof backup solution with RAID only, you have the right to laugh in their face.
  • If you can’t restore it, you don’t have it. If you have everything stored on Amazon Glacier and your house burns down, you’ll probably spend all your money on a new house, not for downloading your music stash from Glacier. Always make sure the backup is in an easily-accessible medium and make sure to test your backup so that you don’t have trouble restoring it later.

Onto the guide!

Steps to install

Now, UnRAID boots off of USB drives, which means that if you just blindly copy your Duplicacy executable to a system folder, the changes will be lost on next boot. So we need to install Duplicacy on a persistent partition, and then copy it across.

Navigate to your USB drive on your UnRAID server:

cd /boot/config/plugins

Download the Duplicacy executable:

wget -O duplicacy https://github.com/gilbertchen/duplicacy/releases/download/v2.2.3/duplicacy_linux_x64_2.2.3

Now it’s very certain that there is a new version by the time you are reading this article. Simply go to the releases tab on GitHub, right click on duplicacy_linux_x64_xxx where xxx is the version number, and select “Copy link location” (may vary depending on browser). Then paste it after wget -O duplicacy.

Then, make the necessary changes in your go file to copy over the executable on every boot. Type the following two lines at the end of /boot/config/go:

# Copy duplicacy
ln -s /boot/config/plugins/duplicacy /usr/local/bin/

This should symlink the executable to an executable location every time UnRAID starts up.

You’re done! Reboot the machine (or run ln -s /boot/config/plugins/duplicacy /usr/local/bin/) and make sure you can run duplicacy -h in your terminal.

Setting up a Duplicacy “repository”

You need to get Duplicacy to init a new “repository”. This means you’re marking what you want to back up. Duplicacy will create a new hidden folder called .duplicacy in your current directory.

duplicacy init repositoryname b2://servername-duplicacy -e

So what does this mean? Let’s go over this:

  • Let’s say we’re currently in /mnt/user, so we’re backing up everything underneath it. For example, Duplicacy will back up /mnt/user/Movies, /mnt/user/Music, and so on.
  • init tells Duplicacy we want to set up a repository here.
  • repositoryname is just a name you will give to the repository. Set it to whatever.
  • b2:// is the protocol (Backblaze B2). You can use other protocls if you use different providers, such as sftp:// and so on.
  • servername-duplicacy is the bucket name you have in B2. You should already have this created before running the command!
  • -e tells Duplicacy to encrypt the backups.

You can actually set up duplicacy in the root folder of your shares, /mnt/user/*. This way, duplicacy stores all the configuration and other details in /mnt/user/.duplicacy/*.

Stop Duplicacy asking for keys

The next step to automation is to get Duplicacy to stop asking for your application ID and password all the time. Otherwise, you will need to manage them yourself every time you back up, which will not work in an automated environment.

Caution: If you go through with these commands, then your application keys and encryption password will be stored as plain-text inside your .duplicacy folder. Anyone with access to this folder will be able to see your keys. Only issue the following commands if you are certain that only you will have access to the server.

Run the following commands:

duplicacy set -storage b2://servername-duplicacy -key b2_id -value b2_id_here
duplicacy set -storage b2://servername-duplicacy -key b2_key -value b2_key_here
duplicacy set -storage b2://servername-duplicacy -key password -value encryption_key_here

Excluding folders and files

You may want to customize the filters file in .duplicacy/ before running the backup. Personally, I have mine set to exclude Docker volume folders, VM folders, and the Docker image (since they can always be recreated with no loss).

For the full documentation, refer to the official documentation since I don’t know regular expressions yet.

Here is my UnRAID filters file if you are interested:

-appdata/
-backup/
-VM/
-Downloads/
-ISO/
-docker.img
-libvrt.img

Automating backups

You’re almost there! We need to automate backups with User Scripts, since we all know you won’t manually remote every day to issue the command.

Download User Scripts (plenty of guides online) on your UnRAID server. You can use Community Applications for it. Once you’re there, create a script with the following:

#!/bin/bash

# https://stackoverflow.com/a/185473/1388019
lockfile="/tmp/duplicacy.lock"

if [ -e ${lockfile} ] && kill -0 `cat ${lockfile}`; then
    echo "duplicacy already running"
    exit
fi

# make sure the lockfile is removed when we exit and then claim it
trap "rm -f ${lockfile}; exit" INT TERM EXIT
echo $$ > ${lockfile}

# run the backup with default settings
cd /mnt/user
/usr/local/bin/duplicacy -log backup -threads 2 -stats

# clean up lockfile
rm -f ${lockfile}

Let’s go over this line by line. On the first couple of lines, from the line that starts with lockfile to echo $$, we set up a lockfile. In case User Scripts mess up and run the script twice (again, two instances of Duplicacy running can seriously corrupt the backup), the lockfile prevents the script from running.

Then the two lines after that start the backup, with 2 threads. Adjust to match your CPU. -stats just shows you what it’s backing up at a given moment.

Save it with an appropriate name and description, set a schedule, and let it run in the background! The backup is now completely automated!

Deleting old backups

By the way, Duplicacy will not automatically delete old backups. To do that, you must prune your backup set.

Add the following two lines in the original script, before the “clean up lockfile” section:

# Delete any backups older than 120 days. Adjust if required
duplicacy prune -exhaustive -exclusive -keep 0:120

This deletes any previous data backups older than 120 days. Adjust the date if necessary.

How do I restore?

I’m not sure because my server hasn’t crashed yet. (knock on wood)

If it ever does I will update this section. But it should just be a matter of running something like duplicacy restore in the root directory after restoring your configuration files.

To uninstall

  1. Delete the executable over at /boot/config/plugins/duplicacy.
  2. Edit the go file over at /boot/config/go. Remove the two lines that we added above.
  3. Delete configuration files if required. Go to your source directory (for me it’s /mnt/user) and then delete the .duplicacy directory.
  4. Remove the User Scripts.

Once that’s done, reboot your UnRAID server.

duplicacyunraidbackblaze-b2backup

comments