Append-only Restic backups on a Hetzner Storage Box

This quick guide is written after hours of trying to get a basic restic backup repository setup on a Hetzner Storage Box. This took so long because Hetzner documentation on the Storage Box is lacking and restic’s rclone backend is complex.

Solution

On the Storage Box, setup an authorized_keys file as such:

command="rclone serve restic --stdio path/to/repo" <public key>

On a client, run restic commands as such:

restic -o rclone.program="ssh -p 23 u###@u###.your-storagebox.de -i /path/to/private/key" [command]

The rclone.program command can be simplified to simply ssh <host> if you configure an SSH config for whichever user you’ll be running this script as.

Background

Before creating backups, restic requires the creation of a repository through some backend. The simplest way to create a repository on a Storage Box would be to use the SFTP backend, but I would prefer using an append-only repository so that if my machines are compromised, I do not lose all backups alongside the original data from an attacker running a forget or prune command. So, I must use the REST Server backend or rclone backend instead.

Side tangent, I think append-only backups are an important feature that some people forget to think about when it comes to backing up data. If your backup scheme involves clients sending data to a server, have you considered what happens if they become compromised and send commands to delete data? You may lose your original copy of the data and any backups.

Of course, if you’re using a scheme where a server pulls data you are safe, or if your server keeps snapshots you’re also safe. Many cloud (backup) storage providers don’t support the former, and I prefer not to use the latter because it is vendor-specific and requires setup outside of the backup tool itself. I prefer a simple storage provider which gives me some amount of storage, as well as the ability to run restic (historically Borg) in server mode.

To be clear, append-only mode isn’t foolproof, for the reasons covered in the append-only repository documentation. This page and the rest of restic’s documentation on security considerations and threat modelling is excellent, by the way.

Back to setting up the backend for my restic repository, let’s take a look at the Hetzner documentation. Well, there’s no mention of restic, but there is a mention of Borg’s append-only mode telling people to look at its documentation. I happen to know Borg’s append-only mode operates in a similar way of running a backend on the server, so let’s keep looking. Connecting to the Storage Box through SSH gives some clues:

/home > help
# --- snip ---
| Available as server side backend:                                           |
|   borg                                                                      |
|   rsync                                                                     |
|   scp                                                                       |
|   sftp                                                                      |
|   rclone serve restic --stdio                                               |
|                                                                             |
| Please note that this is only a restricted shell which do not               |
| support shell features like redirects or pipes.                             |
|                                                                             |
| You can find more information in our Docs:                                  |
|   https://docs.hetzner.com/robot/storage-box/                               |
+-----------------------------------------------------------------------------+

That last backend looks like the one we need, with an added --append-only flag and path. Let’s try it with the command= forced command syntax of the authorized_keys file. How you would know that this is the right way to do so is beyond me, especially because trying to run the command in the restricted interactive shell fails with “Command not found.”

command="rclone serve restic --stdio <repository>" <public key>

Now for the client side: the documentation is mostly centered around running rclone, a program which manages files on cloud storage providers, locally which will then connect to some backend (e.g. SFTP) as such:

$ restic -r rclone:<repository> [command]

Trying to setup rclone locally proved unsuccessful, because it was the wrong way to go about it. Attempting to use the REST backend (which is what rclone serve actually serves) won’t work either, because of the --stdio flag which serves the REST backend over stdin/out instead of an HTTP port. And no, you cannot remove this flag and listen on a port becuase Hetzner doesn’t allow listening on arbitrary ports. Instead, we need to tell restic to use the rclone instance that will run on the other end of the SSH connection, using the remarkably simple to remember command:

$ restic -o rclone.program="ssh <host>" -r rclone:<repository> [command]

The best part about this is that it makes rclone itself no longer necessary on client machines, so I only need to install restic.

As I write this, I will mention that this invocation is actually mentioned at the very bottom of the restic rclone backend documentation. I ended up figuring this out through a blog post by Simon Ruderich. Funnily enough, that blog post is also linked to on the restic documentation. Guess I should have read through things more carefully being trying to set this up.