Keeping a backup somewhere other than your server

This picks up from a server you can already reach over SSH.

You need an off-site backup because the copy that lives on the same server fails with the server. Ransomware, a bad delete, hardware loss or a datacentre incident can take your on-server backups with it. The fix is to keep an encrypted copy somewhere else, automate it, make it hard to tamper with, and test restores regularly.

CISA recommends maintaining offline, encrypted backups and testing them. NIST requires off-site storage and testing at an alternate site. The well known 3-2-1 rule that CISA endorses is a good baseline: three copies of your data, on two different media, with one off-site.

Before you start

• Decide where your off-site copy will live. Common choices:
  • Object storage that supports immutability, for example enabling Amazon S3 Object Lock on a bucket.
  • A second server you control, reached over SSH.
  • A dedicated restic server using rest-server in append-only mode.
• Pick a tool. Restic encrypts by default and is designed for secure, verifiable backups. Borg supports authenticated encryption and can enforce append-only behaviour when serving over SSH.
• Plan how you will test restores. A backup you have not restored is a hypothesis. Both restic and Borg have built-in check and restore commands.
• Know what not to do:
  • Do not keep all copies on the same server or in always-online storage that ransomware can encrypt at the same time.
  • Do not copy database data directories while the database is running. Use proper dump or physical backup tooling.
  • Do not run rsync with --delete until you have verified source, destination and trailing slashes. A mistake there can erase data in seconds.
  • Do not store the only copy of your backup password or key on the server you are backing up. If you lose the server, you lose access.
• Have root access or sudo on your VPS. Our Linux VPS use AlmaLinux, Ubuntu or Debian. This guide covers AlmaLinux 9 and Ubuntu 24.04.

Step 1: Choose your off-site target and backup tool

Decide on a primary path, then you can add a second copy later for 3-2-1.

• If you prefer an encrypted backup that integrates well with object storage, choose restic. It encrypts all repository data and supports init, backup, check, restore and mount workflows. You can pair it with immutable object storage or run a rest-server in append-only mode.
• If you want SSH-to-SSH with server-enforced append-only, choose Borg. You can restrict an SSH key so the server only runs borg serve in append-only mode for a specific repository.

Step 2: Install the backup tools

Install restic and Borg from the distribution packages.

AlmaLinux 9: this enables EPEL, then installs both tools.

# Enable the Extra Packages for Enterprise Linux repository
sudo dnf install -y epel-release

# Install restic and borgbackup
sudo dnf install -y restic borgbackup

Ubuntu 24.04: this refreshes package lists and installs both tools.

# Update package lists
sudo apt update

# Install restic and borgbackup
sudo apt install -y restic borgbackup

Step 3: Prepare safe credentials and a restic repository

This creates a password file for restic and initialises a repository at your remote location. Keep a copy of the password somewhere off the server too.

# Create a password file readable only by root (restic uses this to decrypt the repo)
sudo install -m 600 -o root -g root /dev/null /root/.restic-pass
echo 'choose-a-strong-unique-password' | sudo tee /root/.restic-pass > /dev/null

# Example: initialise a repository over SFTP/SSH on a remote server you control
# Replace user, host and path with your details
export RESTIC_PASSWORD_FILE=/root/.restic-pass
restic -r sftp:[email protected]:/backups/host1 init

If you plan to use object storage with immutability, create a bucket and enable Object Lock in your provider, then point restic at that repository. If you run your own rest-server, enable its append-only mode so clients cannot delete or rewrite history.

Step 4: Run your first restic backup

This backs up important directories and avoids pseudo-filesystems that do not belong in a backup. Adjust the include and exclude list to suit what you host.

# Create an exclude file so you do not back up transient or pseudo-filesystems
sudo tee /root/.restic-excludes > /dev/null <<'EOF'
/dev
/proc
/sys
/tmp
/run
/mnt
/media
/lost+found
/var/cache
EOF

# Run the initial backup. Add or remove paths to match your server.
export RESTIC_PASSWORD_FILE=/root/.restic-pass
restic -r sftp:[email protected]:/backups/host1 backup \
  /etc /home /var/www \
  --exclude-file /root/.restic-excludes

# List snapshots to confirm it worked
restic -r sftp:[email protected]:/backups/host1 snapshots

Restic encrypts data by default. The repository password is required to access anything inside the backup.

Step 5: Automate backups with a systemd timer

This creates a script, a service and a timer so your server backs up on a schedule and catches up if it was down. The timer uses Persistent=true to run a missed job on boot.

# Write a backup script that runs restic backup, retention and an integrity check
sudo tee /usr/local/sbin/backup-restic.sh > /dev/null <<'EOF'
#!/usr/bin/env bash
set -euo pipefail

export RESTIC_PASSWORD_FILE=/root/.restic-pass
export RESTIC_REPOSITORY="sftp:[email protected]:/backups/host1"

TIMESTAMP=$(date -Is)
echo "[$TIMESTAMP] Starting restic backup"

restic backup /etc /home /var/www --exclude-file /root/.restic-excludes

# Retention: keep daily for a week, weekly for 4 weeks, monthly for 6 months
restic forget --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --prune

# Spot-check integrity by reading a subset of data
restic check --read-data-subset=1%

echo "[$TIMESTAMP] Backup run complete"
EOF

# Make it executable and locked down
sudo chmod 700 /usr/local/sbin/backup-restic.sh
sudo chown root:root /usr/local/sbin/backup-restic.sh

# Create a systemd service that runs the script
sudo tee /etc/systemd/system/backup-restic.service > /dev/null <<'EOF'
[Unit]
Description=Restic backup

[Service]
Type=oneshot
ExecStart=/usr/local/sbin/backup-restic.sh
Nice=10
IOSchedulingClass=best-effort
IOSchedulingPriority=7
EOF

# Create a timer to run daily at 02:15, and catch up after downtime
sudo tee /etc/systemd/system/backup-restic.timer > /dev/null <<'EOF'
[Unit]
Description=Run restic backup daily

[Timer]
OnCalendar=*-*-* 02:15:00
Persistent=true

[Install]
WantedBy=timers.target
EOF

# Load the new units and start the timer
sudo systemctl daemon-reload
sudo systemctl enable --now backup-restic.timer

# Show the timer to confirm when it will next run
systemctl list-timers --all | grep backup-restic

Environment for systemd services is not your interactive shell. Keeping repository and password handling inside the script avoids surprises for non-interactive runs.

Step 6: Make the remote copy resistant to tampering and deletion

Backups that the attacker can change or delete are a weak defence. Aim for immutability or append-only behaviour.

• Object storage: enable a bucket-level immutability feature such as S3 Object Lock. Set a retention that matches your risk appetite so objects cannot be removed or overwritten during that window.
• rest-server: run it in append-only mode, which accepts new data but refuses history rewrites or deletes.
• Borg over SSH: restrict the SSH key on the backup server to an append-only borg serve for a single repository. This example creates a dedicated repo and forces that behaviour.

# On the backup server, create a repository directory and user
sudo useradd --system --create-home --shell /usr/sbin/nologin backup
sudo mkdir -p /backups/host1
sudo chown backup:backup /backups/host1
sudo chmod 700 /backups/host1

# On the client, create an SSH key dedicated to backups
ssh-keygen -t ed25519 -f ~/.ssh/borg-backup -N ''

# On the backup server, restrict that key to append-only borg serve for this repo
# Append a line like this to ~backup/.ssh/authorized_keys (replace the key)
command="borg serve --append-only --restrict-to-repository /backups/host1",restrict ssh-ed25519 AAAAC3... client@host

# On the client, initialise and back up with Borg using that key
export BORG_RSH="ssh -i ~/.ssh/borg-backup -oIdentitiesOnly=yes"
export BORG_REPO="[email protected]:/backups/host1"

borg init --encryption=repokey-blake2 "$BORG_REPO"
borg create --stats --verbose "$BORG_REPO"::"$(hostname)-{now:%Y-%m-%d_%H%M}" /etc /home /var/www \
  --exclude /dev --exclude /proc --exclude /sys --exclude /tmp --exclude /run --exclude /mnt --exclude /media --exclude /lost+found

# Prune old Borg archives conservatively
borg prune -v --list "$BORG_REPO" --keep-daily 7 --keep-weekly 4 --keep-monthly 6

Restricting the key forces the server side to accept only new archives into that repository and disallows deletions. This reduces the blast radius if the client is compromised.

Step 7: Include databases safely in your backup

Databases need a dump or a physical backup that is consistent. Copying raw data files while the service runs is not safe.

• MySQL or MariaDB with InnoDB: use mysqldump with a single transaction for a consistent logical backup.

# Dump all databases in a consistent snapshot and compress the result
sudo mysqldump --single-transaction --routines --events --all-databases | gzip > /var/backups/mysql-$(date +%F).sql.gz

# Add /var/backups/mysql-*.sql.gz to your restic or Borg backup include list

• PostgreSQL:
  • For logical backups: use pg_dump or pg_dumpall.
  • For physical online backups with WAL: use pg_basebackup as part of a proper setup.

# Logical dump of all PostgreSQL databases
sudo -u postgres pg_dumpall > /var/backups/pg-$(date +%F).sql

# Or prepare a physical backup (requires configuration and appropriate privileges)
# sudo -u postgres pg_basebackup -D /var/backups/pgbase -Ft -z -P

Schedule your database dumps before the filesystem backup runs, so the dump files are included in the off-site copy.

Step 8: Keep live filesystem backups consistent with LVM snapshots (optional)

If your data sits on LVM, take a snapshot, back up from it, then drop the snapshot. This avoids partial changes during the run. Monitor the snapshot’s copy-on-write space. If it fills, the snapshot becomes invalid.

# Identify your LV, then create a read-only snapshot with enough COW space
sudo lvcreate -L 5G -s -n rootsnap /dev/vg0/root

# Mount the snapshot read-only
sudo mkdir -p /mnt/rootsnap
sudo mount -o ro /dev/vg0/rootsnap /mnt/rootsnap

# Back up from the snapshot mount instead of the live filesystem
restic -r sftp:[email protected]:/backups/host1 backup /mnt/rootsnap/etc /mnt/rootsnap/home /mnt/rootsnap/var/www \
  --exclude-file /root/.restic-excludes

# Unmount and remove the snapshot
sudo umount /mnt/rootsnap
sudo lvremove -y /dev/vg0/rootsnap

# While the snapshot exists, monitor its COW usage. If Data% hits 100, it is invalid.
sudo lvs -a -o +data_percent

Step 9: Prune old backups safely

Removing references is not enough. You must also free space. Restic uses forget to set what to keep and prunes when asked. Borg prunes archives and can do a dry-run first.

# Restic: keep recent points and prune unreferenced data
restic -r sftp:[email protected]:/backups/host1 forget \
  --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --prune

# Borg: preview what would be deleted, then prune for real
borg prune --dry-run -v --list "$BORG_REPO" --keep-daily 7 --keep-weekly 4 --keep-monthly 6
borg prune -v --list "$BORG_REPO" --keep-daily 7 --keep-weekly 4 --keep-monthly 6

Be conservative. A too-aggressive retention policy can remove the only copy of something you still need.

Step 10: Test that you can restore

Checking a repository is good. Doing a test restore is better. CISA and NIST both call for regular restore testing.

# Restic: verify integrity and perform a test restore to a temporary path
restic -r sftp:[email protected]:/backups/host1 check --read-data-subset=1%
sudo mkdir -p /tmp/restore-test
restic -r sftp:[email protected]:/backups/host1 restore latest --target /tmp/restore-test
ls -al /tmp/restore-test

# Optionally mount to browse snapshots
# restic -r sftp:[email protected]:/backups/host1 mount /mnt/restic

# Borg: check the repository and simulate an extract
borg check "$BORG_REPO"
borg list "$BORG_REPO"
borg extract --dry-run "$BORG_REPO"::"$(borg list --last 1 --format '{archive}{NL}' "$BORG_REPO" | tail -n1)" etc/hosts

Schedule these tests on a cadence and keep notes of how long they take and what you restored. That becomes your playbook during a real incident.

Step 11: If you prefer rsync, use it with care

Rsync can copy a whole server to a second server, but it is easy to make a destructive mistake. Do not start with --delete. Understand that a trailing slash changes what gets copied. Preserve permissions, ACLs, xattrs and hard links. Exclude pseudo-filesystems.

# Dry-run a whole-server copy over SSH, excluding ephemeral paths
rsync -aAXHvn --numeric-ids \
  --exclude={/dev/*,/proc/*,/sys/*,/tmp/*,/run/*,/mnt/*,/media/*,/lost+found} \
  -e "ssh -i ~/.ssh/rsync-backup -oIdentitiesOnly=yes" \
  /  [email protected]:/backups/rsync-host1/

# Once you have verified the direction and exclusions, remove -n to run for real
# Only add --delete once you are sure you want the destination to mirror the source

What next

If you need help choosing between restic, Borg, object storage or an SSH target, please open a support ticket and we will look at your workload with you.

If you are planning a second copy for 3-2-1, consider placing it in a different site. For example, pair a UK London VPS with a US VPS so a regional issue does not affect both. You can see our locations on our Linux VPS page.

Carry on with monitoring and recovery rehearsal. Add alerts for failed timers, and run a full restoration exercise on a spare server. Browse more topics in our VPS guides.