First in a six-part series on restore testing. The client described here is a composite, drawn from a few engagements that shared the same shape. The details are real. The organisation isn’t.
Every backup article ever written ends the same way. Build it properly, secure it properly, and test your restores. I’ve written that line on this site five times in the last four months. Not once have I said what it means. So this series is me making good on it, and week one is the awkward bit: working out what a restore test is actually supposed to prove, and to whom.
Because most organisations are running a test that answers a question nobody asked.
The two hour RTO that took eight
The client is a business administration firm. Around 500 staff across several offices and countries. Roughly 100 VMs, a line-of-business case management platform sitting on SQL, a pile of file shares that nobody has ever successfully rationalised, and M365 on top of the lot.
A good shop. A competent, but very busy team. Backups had been ticking over for years and restores were tested monthly; a VM here, a mailbox there, a folder off a file share. Twelve months of green.
Their head of IT inherited a DR process but had no confidence in it, so I was brought in to test it, find it’s faults and fix it. Nothing was broken. The backups were fine. The restore worked exactly as designed. They just had no idea what their recovery time actually was, because they’d never measured the thing that mattered.
Three audiences, three different tests
Here’s the bit that clarified it for me when I stopped and thought about it properly. A restore test is being asked to do three separate jobs, and most organisations only ever do the first.
For yourself, it’s operational: Does the data come back, and does the thing work afterwards. This is the test everyone runs, and it’s the easiest to pass because you get to choose the scope.
For your insurer, it’s evidentiary: Underwriters have moved to evidence-based assessment and the forensic review at claims time compares what you attested to against what actually existed. I went through this in detail a couple of months back. A test you ran but didn’t document is, for these purposes, a test you didn’t run.
For your regulator, it’s a capability: This is the one that catches people. DORA expects resilience testing on a continuous basis. NIS2’s Article 21 measures cover business continuity as an ongoing obligation, not an event. “We restored a VM last March” is a thing that happened. It is not a demonstration that you can do it again next March with different people on shift.
A test can satisfy the first two and still fail the third. That’s not a compliance technicality – it’s the difference between having done a restore once and being able to recover.
The scoping ladder
Four rungs. Worth being honest with yourself about which one you’re actually on.
- File or item. Someone deleted a folder. You get it back.
- Single VM or single mailbox. One workload, restored, powers on.
- A service. The application and everything it needs to be genuinely usable by a person with a job to do.
- The estate, or a defensible subset of it, under conditions that resemble a bad day.
The firm above was testing at rung two, monthly, diligently. The C-suite cares about rungs three & four. Those are not the same exercise and the gap between them is where their six extra hours went.
This is rarely dishonesty. It’s that nobody ever defined what was being asked, so everyone answered the version of the question they were already able to answer.
Related mess worth sorting out now: “restore test”, “DR test” and “failover test” get used interchangeably and mean different things. A restore test brings data back from a backup. A failover test moves a running service to a secondary location, usually from replication, and often never touches a backup at all. A DR test is whatever the person saying it wants it to mean. I’ll be specific throughout this series, and if you take one thing from this post, be specific in your own documentation too. Auditors and underwriters read the word, not the intent.
What “success” means
File presence isn’t success. Service functionality is.
The questions that decide whether a restore is actually usable:
- Does the application start, and does it start without somebody remembering an undocumented step?
- Can a user authenticate into it?
- Does it talk to the things it needs to talk to?
- Is the data consistent to a point in time that somebody can name?
That second one is where two hours went, and it’s where a great many recoveries go. Identity is the dependency that breaks the most restores and gets tested the least, so it’s getting its own post next week rather than a bullet here. The wider dependency chain is week three.
The things nobody tests
Short list, deliberately uncomfortable.
Certificates. DNS. Licensing servers that phone home and won’t activate on restored hardware. Service accounts whose passwords live in the head of a contractor who finished up in 2023. The runbook written by someone who’s since left, describing a version of the environment that no longer exists.
And the one that gets least attention: whether the person who ran the test is the person who’d actually be available at three in the morning on a Sunday. A recovery capability that depends on one individual isn’t a capability. It’s a bloke.
Write it down while you’re doing it
Reconstructing a test eighteen months later for a claim does not work. A forensic team knows what a retrofitted document looks like – the dates are too tidy and the failures are all missing.
Minimum record, captured as you go:
- Date, and who ran it
- Scope, stated in the terms above so it can’t be misread later
- What was restored, from what, to where
- Elapsed time per phase, not just a total
- What failed
- What was changed as a result
- When it’s being repeated
Those last two are what turn an event into a capability. A test with no remediation trail reads as theatre, and reads that way to exactly the people you don’t want thinking it. If you haven’t found fault, you haven’t tested hard enough.
I’m not going to build the full evidence pack here. That’s week six, and it’s a proper piece of work in itself. For now: record it as it happens or you don’t have it.
The point isn’t the test
If a restore test never fails anything, it’s scoped too small. That’s the whole argument. The value isn’t the green tick at the end, it’s the failures and the list of tasks you’ve got to work through to fix them before the next test.
The firm in this piece came out of that exercise with a recovery time roughly 300% longer than the one they committed to the board, and a fairly urgent conversation to have about it. Uncomfortable. Considerably less uncomfortable than the alternative version, where they find out during an incident with a regulator already asking questions.
Next week: identity. Why the restore that comes back fastest is usually the one nobody can log into, and what forest recovery looks like when you’ve never had to do it.