Server Backup Strategy: 3-2-1 Rule, Scripts, Restore Tests

A solid server backup strategy follows the 3-2-1 rule: keep three copies of your data, on two different types of storage, with one copy offsite, and prove regularly that you can restore from it. A backup you have never restored is only a hope, so this guide covers the rule, the difference between snapshots and backups, working scripts for MySQL and files, and a restore test you can automate.

The 3-2-1 Rule in Practice

The rule is simple, but each number protects against a different failure:

• 3 copies: the live data plus two backups. One bad job or one corrupted file should never leave you with nothing.
• 2 media or systems: for example local disk on the server plus a separate backup server or object storage. This protects against a failing disk, a broken filesystem, or a hypervisor-level problem.
• 1 offsite: a copy in a different facility and on a different network and account. This protects against provider-level outages, ransomware that reaches everything mounted on the host, and mistakes like deleting the wrong VM.

A practical layout for a single web or game server:

The local copy is convenient but does not count as protection against host loss. Treat it as a staging area for the remote copy.

Snapshot vs Backup: What Is the Difference?

Snapshots and backups are often confused, and the confusion causes real data loss.

A snapshot is a point-in-time marker of a disk or VM, usually stored on the same storage as the original. It is fast and ideal for a quick rollback before an upgrade. A backup is an independent copy of the data, stored somewhere else, that you can restore even if the original system is gone.

Snapshots also have caveats. A snapshot of a running database is only crash-consistent unless the application is quiesced, so a MySQL server may need crash recovery when you restore it. Long-lived snapshots on VMware-based platforms can also slow down disk I/O as the delta grows. Use them for short windows, and rely on application-aware backups for the long term.

What to Back Up

Before writing scripts, list what you would need to rebuild the server from scratch:

• Databases (MySQL/MariaDB, PostgreSQL)
• Application data: /var/www, uploads, game world files
• Configuration: /etc, systemd units, cron jobs, firewall rules
• TLS keys and certificates if they are not easily reissued
• A list of installed packages (dpkg --get-selections or rpm -qa)

You do not need to back up the OS itself if you can reprovision it quickly and reapply configuration. Many operators back up /etc and data, and keep a short rebuild document.

MySQL Backup Script

Use mysqldump with --single-transaction for InnoDB tables. It takes a consistent snapshot of the data without locking tables. Note that it does not give consistency for MyISAM tables; convert them to InnoDB if you can.

First, store credentials in a root-only file so passwords never appear in process lists:

# /root/.my.cnf  (chmod 600)
[client]
user=backup
password=CHANGE_ME

Create a limited user:

CREATE USER 'backup'@'localhost' IDENTIFIED BY 'CHANGE_ME';
GRANT SELECT, SHOW VIEW, TRIGGER, EVENT, LOCK TABLES, PROCESS ON *.* TO 'backup'@'localhost';

Then the script:

#!/usr/bin/env bash
# /usr/local/sbin/mysql-backup.sh
set -euo pipefail
umask 077

BACKUP_DIR=/var/backups/mysql
KEEP_DAYS=7
STAMP=$(date +%F_%H%M)
CNF=/root/.my.cnf

mkdir -p "$BACKUP_DIR"

for db in $(mysql --defaults-extra-file="$CNF" -N -e "SHOW DATABASES" \
  | grep -Ev '^(information_schema|performance_schema|sys|mysql)$'); do
  mysqldump --defaults-extra-file="$CNF" \
    --single-transaction --quick --routines --triggers --events \
    "$db" | gzip -9 > "$BACKUP_DIR/${db}_${STAMP}.sql.gz"
done

find "$BACKUP_DIR" -type f -name '*.sql.gz' -mtime +"$KEEP_DAYS" -delete

pipefail matters here: without it, a failed mysqldump piped into gzip would still exit successfully and leave you with an empty-looking but valid archive.

File and Config Backup with Restic

For files, a deduplicating tool such as restic (or BorgBackup) is better than plain tar or rsync because it keeps multiple versions, encrypts data client-side, and handles retention. Restic can write straight to a remote server over SFTP.

On the production server, create a dedicated SSH key and initialize the repository on the backup host (example address from the documentation range):

ssh-keygen -t ed25519 -f /root/.ssh/backup_key -N ""
export RESTIC_REPOSITORY=sftp:backup@192.0.2.50:/srv/restic/web01
export RESTIC_PASSWORD_FILE=/root/.restic-pass
restic init

Store the repository password somewhere outside the server as well. If the server dies and the password was only on it, the backup is useless.

The backup script:

#!/usr/bin/env bash
# /usr/local/sbin/files-backup.sh
set -euo pipefail

export RESTIC_REPOSITORY=sftp:backup@192.0.2.50:/srv/restic/web01
export RESTIC_PASSWORD_FILE=/root/.restic-pass

/usr/local/sbin/mysql-backup.sh

restic backup /etc /var/www /var/backups/mysql /home \
  --exclude-caches --exclude '*.log'

restic forget --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --prune
restic check --read-data-subset=5%

Schedule it with cron or a systemd timer:

# /etc/cron.d/backup
30 3 * * * root /usr/local/sbin/files-backup.sh >> /var/log/backup.log 2>&1

Remote Backup Done Right

The offsite copy is where most strategies quietly fail. Check these points:

• Different failure domain: use a server in another facility, on another network, ideally under a separate account. A second VDS or dedicated server in a different location than production works well as a backup target.
• Restricted access: the production server should be able to add backups but not erase history. Use a dedicated SSH user, restrict it in authorized_keys with restrict and a forced command, or use an append-only mode such as rest-server --append-only. If an attacker gets root on production, they should not be able to wipe the backups.
• Encrypted in transit and at rest: SFTP covers transit, and restic encrypts at rest.
• Bandwidth awareness: run the first full backup off-peak. Later runs transfer only changed blocks.
• Pull instead of push (optional): for higher isolation, have the backup server pull data, so production holds no credentials for the backup host.

Test Your Restores

Run a restore test on a schedule, not only after an incident. A minimal automated check:

#!/usr/bin/env bash
set -euo pipefail
export RESTIC_REPOSITORY=sftp:backup@192.0.2.50:/srv/restic/web01
export RESTIC_PASSWORD_FILE=/root/.restic-pass

TMP=$(mktemp -d)
restic restore latest --target "$TMP" --include /var/backups/mysql

LATEST=$(ls -t "$TMP"/var/backups/mysql/*.sql.gz | head -n1)
gunzip -t "$LATEST"

mysql -e "DROP DATABASE IF EXISTS restore_test; CREATE DATABASE restore_test;"
gunzip -c "$LATEST" | mysql restore_test
mysql -N -e "SELECT COUNT(*) FROM information_schema.tables WHERE table_schema='restore_test';"

rm -rf "$TMP"

Better still, restore to a separate scratch VM and start the actual application against it. Once or twice a year, do a full rebuild drill: provision a clean server, restore, and time it. That time is your real recovery time, and it is usually longer than people assume.

Common Mistakes and Fixes

• Backups stored only on the same server. Fix: add the remote copy and keep the local one as a cache.
• No monitoring. Cron fails silently. Fix: send a success ping to a monitoring endpoint (such as a Healthchecks-style URL) at the end of the script, and alert when it is missing.
• Inconsistent database copies. Copying /var/lib/mysql while MySQL runs yields a corrupt copy. Fix: use mysqldump, or a physical tool such as Percona XtraBackup or mariabackup.
• Lost encryption keys or passwords. Fix: store them in a password manager or offline.
• Retention too short. Silent corruption or a deleted record may go unnoticed for weeks. Fix: keep weekly and monthly versions.
• World-readable dumps. Fix: umask 077 and restrictive directory permissions.
• Never restored. Fix: schedule the restore test above and review its output.

Putting It Together

A reasonable baseline for most servers: a nightly database dump plus a file backup to a remote repository, a snapshot before every upgrade or risky change, retention of daily, weekly, and monthly versions, alerting on missed runs, and a restore test every month. Host the backup target on a separate machine in a different location from production. Backups protect data, but not uptime, so combine them with DDoS protection and firewall rules to reduce the chance you need to restore in the first place.