m4nifest0-tech

Search the site

Storage and backup

3-2-1 backup with automated restore verification

Backup chain with Proxmox Backup Server, a NAS, and an off-site copy, paired with automated test restores that confirm it can be relied on.
Completed

Problems faced

  • Backups were being created but never tested: the only check was a successful job result.
  • All copies lived on the same storage system, exposed to the same failures.
  • There was no off-site copy and no separation between production credentials and backup credentials.
  • There were no recovery objectives defined per service class and no written restore procedure.

Implemented solution

The chain follows the 3-2-1 rule. Production data is backed up by scheduled jobs to a local Proxmox Backup Server, replicated to a NAS with an independent storage system, and synchronized to an off-site destination. Verify jobs, prune, and garbage collection are scheduled on the backup server. A script periodically restores a sample machine in an isolated environment, checks that it boots, and logs the outcome, flagging errors. Recovery objectives per service class drive backup frequency and retention.

Results

  • The test restore of a sample machine is automatic and its outcome is logged, not taken for granted.
  • Copies live on distinct systems and in distinct locations, reducing the risk of losing everything to a single event.
  • The restore procedure is written down and tested at regular intervals, not just documented.
  • The off-site copy is operational and included in the same periodic checks.

Technical description

A backup is only useful if you can restore it. This case study builds a backup chain following the 3-2-1 rule and pairs it with an automated check that actually restores the data, instead of relying on job results.

Architecture

The 3-2-1 rule requires at least three copies of the data, on two different media, one of them off-site. In addition to the production data, the chain has three backup copies. The cluster's virtual machines and containers are backed up by scheduled jobs to a local Proxmox Backup Server (first copy). A second copy lives on a NAS, with an independent storage system. The third copy is sent to an off-site destination, with client-side encryption before upload. The first copy, the second copy, and the off-site copy are all operational and are covered by the same restore checks.

Components and choices

  • Proxmox Backup Server provides incremental, deduplicated backups, configurable retention, and data integrity checks.
  • Prune and garbage collection enforce retention and reclaim space; verify jobs periodically re-read the data to catch silent corruption.
  • NAS on ZFS as the second copy, with periodic snapshots and scrubs.
  • Separate credentials: the user the nodes use to push backups has only the permissions it needs and is not the same as the backup server's administrator.
  • Objectives per service class: each group of workloads has a recovery point objective (how much data can be lost) and a recovery time objective (how long I can wait), from which frequency and retention follow.

Automated restore verification

Warning

The verify job checks that the data is intact, not that the machine boots. You need both checks.

A periodic script restores a sample machine to a temporary ID, disconnects the network to avoid conflicts, starts it, and waits for the guest agent to respond. Then it logs the outcome, flags errors, and removes the test machine.

Test restore (simplified outline)
qmrestore pbs-locale:backup/vm/105/<data-ora-snapshot> 9100 --storage local-zfs --unique 1
qm set 9100 --delete net0
qm start 9100
qm agent 9100 ping
qm stop 9100 && qm destroy 9100 --purge

Checking that the machine boots is the minimum level: for critical services I add application-level checks, such as a service responding or reading back test data. The outcome of every test is logged and notified; a failed restore is treated as an incident to investigate, not as a simple warning. The sample machine changes on a rotating basis, so over time every type of workload gets tested.

Possible next steps

Rotating the sample across all service classes, test restores of individual files, an immutable or offline copy against ransomware (the 3-2-1-1-0 extension), and sending the verification results to the monitoring system. The basic method is illustrated in 3-2-1 backup strategy with Proxmox Backup Server and a NAS.