Almost every company has backups. Almost none has a recovery rehearsal with a date on the calendar. The difference between the two only shows up the day someone needs a file back — and by then it isn't an exercise anymore.

The reason it keeps getting postponed isn't laziness. It's that the backup reports success every single day, and the restore asks for nothing. The job finishes, the dashboard turns green, the report lands in your inbox. The whole system was built to tell you the write succeeded. Nobody built anything to tell you the read will.

What "backup completed" actually proves

It proves the bytes left the source and arrived at the destination. That's all.

It doesn't prove the index mapping file to object is consistent. It doesn't prove the encryption key sitting in the vault is the right key. It doesn't prove the account that will perform the recovery has read permission on the bucket. It doesn't prove anyone knows the order in which services must come back up. And it doesn't prove the thing that hurts most: how long it takes.

Each of these is an independent failure mode. A green backup and an impossible restore coexist without contradiction.

The four things you test

Integrity. Is the file that comes back identical to the one that went up? The honest check is comparing the checksum of the original against the checksum of the restored copy, not opening the file and deciding it looks right. For a database, integrity means restoring the dump and running the DBMS's own consistency check. A dump that downloads whole and won't import is an intact, useless file.

Time. Measured with a clock, from the request until the data is usable. That number is your real RTO. The RTO in the document is an intention. The gap between the two is usually wide, and it's widest when the storage sits in an archive class (Glacier Flexible Retrieval, Deep Archive, Azure Archive): those classes add a rehydration step before the download, with its own duration, which varies by the tier you pick. Infrequent access classes and Glacier Instant Retrieval skip rehydration — the object comes out through direct access. AWS's documentation on restoring archived objects and the Azure Blob rehydration overview describe the tiers available on each platform. If your backup lives in an archive class, that time isn't a footnote: it's the bulk of the clock. We wrote about this behavior in Deep Archive in practice.

Permissions. The test has to run under the identity that will exist on the bad day. It's tempting to recover using the admin credential sitting on your laptop — and that's exactly the one that may not exist during the incident, because the incident might be the loss of that laptop, a forced credential rotation, or a compromised account. Test with the recovery account, from a host that isn't the backup server.

Dependency order. Systems don't come back file by file. They come back in layers: network, secrets, database, queue, application, DNS. Restoring the application before the database gives you an application that starts, answers the healthcheck, and serves errors. The order has to be written down, and the rehearsal is what reveals the forgotten item — almost always a secret, a certificate, or an environment variable that never made it into the backup scope.

An exercise that fits in a morning

Don't try to rehearse the full disaster on the first attempt. A rehearsal that's too big is a rehearsal that never happens. Start with a small, closed scope.

Pick the target the day before. A real directory, a small database, a VM volume. Something that matters, but whose unavailability stops nobody.
Write the hypothesis first. "I expect to recover 8 GB in under two hours, using the restore-ops account, without needing anyone from the infrastructure team." A written hypothesis turns the rehearsal into a measurement. Without one, the result is always "it worked."
Restore to an isolated destination. Separate bucket, separate host, separate database. Never on top of production. A rehearsal restore that overwrites good data is an incident created by the exercise itself.
Time three milestones. Request issued; data available for download; data usable by the application. The three numbers tell different stories, and the third is the only one the business understands.
Verify with checksums. sha256sum at source and destination, compared automatically. Sampling works for large volumes, as long as the sample is random and not the first files in the listing.
Write down what was missing. Every first rehearsal finds something absent. That's the main output of the exercise, not a failure of it.
Close with a record. Date, scope, timings, who ran it, what broke, what changed in the runbook. One page. Without it, the next rehearsal starts from zero.

Rehearsing costs money, and the cost has a known shape

This is the real objection, and it deserves an honest answer. Recovery isn't free. Billing usually has four components, and all the major providers use some combination of them:

Requests. Charged per operation. Recovering a million small files generates a million operations, and that bill sometimes exceeds the month's storage cost.
Data egress. Moving data out of the cloud generally carries a per-gigabyte price at most providers. Restoring to a host inside the same region is usually much cheaper than restoring to the office.
Archived data retrieval. In archive tiers, on top of storage you pay for the volume retrieved, at a different price per urgency tier.
Minimum retention and early deletion fees. Cold classes charge for a minimum period. Deleting the rehearsal copy before that window can trigger a charge for the remainder.

I'm not putting numbers here because they change and vary by region. Open your provider's current price table, for your region and your class, and build the rehearsal's bill before running it. Two decisions cut that cost without faking the test: restore a sample instead of the whole set, and restore into the same region. A sample measures integrity and permissions well enough for the rehearsal. What it doesn't measure is total time — for that, once a year, the big rehearsal is worth paying for.

A cadence that works

Monthly, small, automated: a rotating target, restored by script, checksum compared, result logged. Quarterly, medium: an entire service, with the dependency order exercised and different on-call people running it. Annually, large: the total-loss scenario, timed end to end.

The rotation criterion is simple: rehearse first what you're most afraid of losing and least used to touching.

What this article does not prove

I'm offering no measurement here, and that is deliberate: your clock depends on volume, storage class, region, and destination disk. Someone else's test number is not a promise about your case — measure your own.

There are situations where the advice above is bad. If your backup sits in a cold class with a long minimum retention and the budget is tight, a monthly rehearsal of the whole set is waste: use a sample and save the full test for once a year. If the dataset contains sensitive personal data, restoring into a test environment may violate your own policy — in that case the rehearsal requires anonymization or an environment with the same controls as production, and that changes the effort involved. And if your operation is a single server run by one person, the three-cadence ritual is bureaucracy: a well-documented semiannual rehearsal is enough.

This text also doesn't cover recovery of distributed systems with replicated state, where the dependency order is far more complex than the list above suggests. And it doesn't address who holds the key, which is a separate problem and just as serious — that's in who can open your backup.

What's left is the part that holds for everyone: an untested restore is a hypothesis. The hypothesis costs one morning to become a fact.

NubliVault is Nublify's backup product: it packs millions of small files and stores them in a cloud different from the source. How it works · Security