Backup restore testing: proving your business software can recover

Most small businesses with a custom app can tell you that backups exist. A nightly job runs, a file lands in cloud storage, the dashboard is green. Far fewer can tell you how long it would take to get the app working again from those files, or whether anyone has ever tried. That gap only shows up on the worst day, when a bad migration wipes a table or ransomware encrypts the server.

This post is about closing that gap: deciding what "recovered" means for your system, picking a backup method that fits, and running restore tests that prove it.

Backups are a means; restores are the point#

Google's Site Reliability Engineering book has a chapter on data integrity that puts it bluntly: "No one really wants to make backups; what people really want are restores." The same chapter warns that a recovery process "may be in a latent broken state, which you aren't aware of until you attempt to recover data", and that "you only know that you can recover your recent state if you actually do so."

Latent breakage is common in small systems because nothing exercises the restore path. A few typical ways it happens:

• The backup job writes a zero-byte file after a password rotation and nobody looks at the size.
• The dump covers the database, but uploaded files (invoices, photos, signed PDFs) live on a disk that isn't backed up.
• The backup is fine, but the only person who knows the restore commands has left.
• The restore works but takes nine hours, and the business assumed one.

None of these show up in a "backup succeeded" notification. They show up when you restore.

Set two numbers before you pick a tool#

Disaster recovery planning has two standard targets, and NIST's contingency planning guide (SP 800-34 Rev. 1) defines both. The recovery point objective (RPO) is "the point in time to which data must be recovered after an outage". In plain terms, how much recent work you can afford to lose. The recovery time objective (RTO) is how long the system can be in recovery "before negatively impacting the organization's mission or mission/business processes". How long you can be down.

Set these with the people who use the system, not just whoever runs it. Ask the operations manager what happens if the job tracker loses everything since last night. If the answer is "we'd re-enter twenty jobs from email, annoying but fine", a nightly backup gives you an RPO of up to 24 hours and that's acceptable. If the answer is "we'd lose a day of card payments with no other record", nightly isn't good enough.

Writing the numbers down matters because they turn a vague worry into something you can test. "Restore must complete within four hours and lose no more than one hour of data" is a pass/fail check. "We have backups" isn't.

Choose a backup method that can meet the targets#

The method has to match the RPO. Using PostgreSQL as the example, the official docs say there are three fundamentally different approaches: SQL dump, file system level backup and continuous archiving.

A SQL dump with pg_dump is the simplest and suits a lot of internal tools. According to the PostgreSQL documentation on SQL dumps, a dump "represents a snapshot of the database at the time pg_dump began running", and it doesn't block other operations while it works, apart from those needing an exclusive lock such as most forms of ALTER TABLE. The same page notes that dump output can generally be loaded into newer PostgreSQL versions, while file-level backups and continuous archiving are "extremely server-version-specific". That portability is handy when your restore target is a different, newer server.

The catch is the RPO. A dump every night means you can lose up to a day. If that's too much, continuous archiving (shipping the write-ahead log so you can replay to a point in time) is the usual next step, or a managed database's point-in-time restore feature. The extra moving parts are worth it only if your RPO needs them.

Whatever the database, list everything the app needs to come back: the database, uploaded files, configuration and secrets (stored safely, not in the backup bucket in plain text), and the code or container image for the exact version in production. The #StopRansomware Guide from CISA and partner agencies makes the same point for infrastructure. It recommends keeping "golden images" of critical systems and offline copies of infrastructure-as-code templates, so you can redeploy as well as reload data.

Replication is not a backup#

A replica or a synced folder feels like a backup because the data exists in two places. It isn't one. The Google SRE chapter explains that systems which sync copies automatically can push "a corrupt database row or errant delete" to every copy before anyone notices, and it states that "replication and redundancy are not recoverability."

The same applies to file sync tools. If someone deletes a folder, or malware encrypts it, the sync faithfully copies that change. You need copies from earlier points in time that a change to the live system can't touch.

Protect backups from the thing you're backing up against#

The CISA guide recommends maintaining "offline, encrypted backups of critical data" and testing them regularly.

Australia's Essential Eight takes a similar line on access. The regular backups requirements, as published in ASD's Blueprint for Secure Cloud, say that even at the lowest maturity level unprivileged accounts can't access other accounts' backups and are prevented from modifying or deleting backups. At higher levels that restriction extends to privileged accounts other than backup administrators, and at Maturity Level Three backup administrators themselves can't modify or delete backups during the retention period.

For a small custom app, a practical version looks like this. The server that runs the app can write new backups but has no permission to delete or overwrite old ones. Retention is enforced by the storage (object lock or a bucket retention policy) rather than by a script on the app server. At least one copy sits in a separate account that the app's credentials can't reach.

Run a restore test, and time it#

The Essential Eight requirement includes disaster recovery exercises that test "the restoration of data, applications and settings" from backups to a common point in time. That's a good description of what a test should cover: the whole app, not just a file you checked was non-empty.

A restore test for a small system can be done in an afternoon:

1. Pick a backup at random, not the newest one, so you also test retention.
2. Restore it into a clean environment with no access to production: a fresh database server, a fresh copy of the file storage and the app version recorded with the backup.
3. Start a timer when you begin and stop it when a user could log in and do real work. Compare that with your RTO.
4. Check the data, not just the row counts. Open the most recent record and compare its timestamp with your RPO. Open a few uploaded files. Run a report someone actually uses and see if the totals look right.
5. Write down every step you had to work out or look up, and fix the runbook.

For PostgreSQL dumps, watch for one detail in step 2. The SQL dump docs point out that psql keeps going after an SQL error by default, which can leave you with a partially restored database that looks fine at a glance. Running it with ON_ERROR_STOP=on, or restoring in a single transaction, makes failures obvious instead of silent.

The SRE chapter suggests asking, of every test: "Are your backups valid and complete, or are they empty?" and "Does the recovery process complete in reasonable wall time?" It also advises automating these tests where possible and running them continuously. For a small business, continuous might mean a scheduled job that restores last night's dump into a throwaway database each week and alerts if it fails or a sanity query returns nothing. That catches the zero-byte backup within days instead of during an outage.

What a rehearsed restore looks like#

The SRE chapter describes a Gmail incident from February 2011 in which the service "lost a significant amount of user data" and Google restored it using GTape, its global backup system for Gmail. It was the first time that system was used at large scale on live customer data. The restore had been planned and rehearsed beforehand, and the chapter reports that more than 99% of the data was recovered before the estimated completion time.

Your system is much smaller, but the lesson scales down. The restore went well because people had already done it. When we build internal tools at TheAILAB, the restore steps sit in the same repository as the code, and the first restore test happens before the app goes live, not after something breaks.

If you do one thing this month, put a restore test in the calendar with a named owner and a stopwatch. Restore last Tuesday's backup into a clean environment, time it, and compare the result with the RPO and RTO you agreed with the business. Whatever goes wrong in that hour is the cheapest version of that failure you'll ever get.

Sources

1. Chapter 26 - Data Integrity: What You Read Is What You Wrote — Google, Site Reliability Engineering (book) (checked 2026-10-23)
2. Recovery Point Objective (glossary, from NIST SP 800-34 Rev. 1) — NIST Computer Security Resource Center (checked 2026-10-23)
3. Recovery Time Objective (glossary, from NIST SP 800-34 Rev. 1) — NIST Computer Security Resource Center (checked 2026-10-23)
4. Regular backups — Australian Signals Directorate, Blueprint for Secure Cloud (checked 2026-10-23)
5. #StopRansomware Guide — CISA, MS-ISAC, NSA and FBI (September 2023) (checked 2026-10-23)
6. Chapter 26. Backup and Restore — PostgreSQL Global Development Group (PostgreSQL 16 documentation) (checked 2026-10-23)
7. 26.1. SQL Dump — PostgreSQL Global Development Group (PostgreSQL 16 documentation) (checked 2026-10-23)