The Clock Starts Before You Do: Measuring Recovery Time Honestly

Fifth in a six part series on restore testing. Same composite client throughout: a business administration firm, around 500 staff, two offices.

They ran it again about three months later.

By then the DSRM password was where it should be, the issuing CA was in scope, the revocation problem was understood, and the bubble had no route out of it. All the things the first test had found, fixed. The technical recovery came in at ninety minutes, start to finish, against six hours forty the first time round.

Everyone was delighted, and they were right to be. It’s a genuine improvement and it cost them almost nothing but attention.

Then somebody asked what the number would be if it happened at three in the morning on a bank holiday Sunday, and the room went a bit quiet.

Because the honest answer was still north of five hours, and the difference had nothing whatsoever to do with backup.

the clock the business is using

Here’s the disconnect at the centre of this whole subject.

Backup platforms measure from the moment a restore job starts to the moment it reports complete. That’s the number on the report and it’s the number that gets quoted. For this client the first time round, it was forty minutes.

The people who suffer the outage are measuring from when the thing stopped working to when they could do their job again. Nobody in that group cares when the restore job started. They weren’t consulted about when the restore job started.

So there are two clocks and they share almost no overlap. The gap between them is detection, escalation, somebody deciding to invoke, people being found and got online, and at the far end validation and telling users they can go back in. On a good day in office hours that gap is small enough to ignore. At half two on a Sunday morning it’s most of your recovery time.

If your documented RTO is measured on the first clock and your contracts are written against the second, you have a gap you haven’t priced.

phases, and why the total is close to useless

Six hours forty tells you nothing you can act on. It’s a single number describing a process with about seven distinct stages, and until you break it apart you can’t tell which stage to spend money on.

The phases I’d separate, at minimum:

Detection. Outage begins to somebody knowing about it.

Decision. Somebody knowing to somebody with the authority saying yes, invoke.

Mobilisation. People, access, bridge open, hands on keyboards.

Target readiness. Infrastructure available to restore onto, which is zero if it’s warm and can be substantial if it isn’t.

Data movement. The bit the backup software measures.

Service bring up. Everything from parts two and three. Identity, dependencies, the application actually functioning.

Validation and release. Somebody competent confirming it works, and users being told.

Now look at what that did for this client. Of the first test’s six hours forty, data movement was forty minutes. Everything else was bring up. Which means every euro they could have spent on faster restore performance was aimed at ten percent of the problem.

I’ve watched organisations buy flash repositories to fix recovery times that were dominated by a missing password and a certificate timeout. It’s not that the faster tier is a bad purchase. It’s that nobody measured before buying, so nobody knew it would move the number by about six percent.

Break the total apart and the right investment is usually obvious and usually cheap. That’s the whole reason for doing it this way.

your test is the best day you will ever have

Worth saying plainly, because test numbers get quoted as if they were operational ones.

A scheduled test has the right people, awake, expecting it, in daylight, with a known scope, working on one service, in an environment that was stood up in advance by somebody who knew what they were doing. Every one of those is a favourable condition that won’t hold during an actual incident.

I’m not going to hand you a multiplier, because I don’t believe in them and the honest range is too wide to be useful. What I’d do instead is walk the phase list above and ask, for each one, what changes at three in the morning. Detection probably gets slower unless your monitoring is genuinely good. Decision gets a lot slower if the person who can authorise isn’t answering the phone. Mobilisation depends entirely on whether your on call arrangement is real or nominal.

Bring up, oddly, barely changes, as long as the runbook is good. That’s the bit you’ve already fixed.

For this client the gap between the ninety minute technical number and the five hour honest number was almost entirely detection and decision. They’d fixed everything in their control as an IT function and the remaining bottleneck belonged to somebody else. Which is a useful thing to be able to prove, not least because it moves the conversation to where it can actually be solved.

the bit where estate numbers fall apart

One service tested cleanly does not let you calculate the estate.

Restore three services at once and they contend for the same repository read throughput, the same proxies, the same network, and the same two engineers. Restore points that come back in forty minutes on their own do not come back in forty minutes each when there are nine of them queued.

And the sequencing from part two makes it worse in a way that’s easy to miss. Identity has to be up before anything else is usable, so the dependency graph is partly serial no matter how much parallel capacity you have. Your estate recovery time is the critical path through that graph, plus contention, not the longest individual restore.

If you’ve never tested more than one service at a time, you don’t have an estate number. You have a service number and an assumption.

while you’re in there, check your actual RPO

Slightly off topic but it comes up every single time and it’s usually wrong.

Your stated RPO is derived from your job schedule. Your actual RPO is derived from when jobs finish. A nightly job that kicks off at ten in the evening and takes until four in the morning for the big database means that at three am your most recent usable point is the previous night’s completion, not last night’s start.

Look at the completion timestamps for your important workloads across a month rather than the schedule, and work out the real worst case. For plenty of estates it’s a good deal further back than anyone has told the business.

how to record it so the numbers survive contact

Record wall clock timestamps at each phase boundary, not durations. Durations get computed afterwards and can’t be misremembered or rounded in anyone’s favour.

Give that job to somebody whose only job it is. The person doing the work cannot also do the timing, because the moments you most need captured are exactly the moments they’re busiest. It’s a dull hour for somebody and it’s the difference between a test that produced evidence and a test that produced a feeling.

Write down the conditions alongside the numbers. Time of day, who was present, what was already warm, what was in scope, what help was given that wouldn’t be available at three in the morning. A number without its conditions is not evidence of anything, and this becomes important next week.

I’ll admit to getting this wrong myself, years ago. I handed a client a single figure off a test report and let them put it in a document, and if anyone had pushed on it I couldn’t have told you where the time went or what the number assumed. It was accurate. It just wasn’t defensible, and those are different things.

the number you publish

Two numbers, and keep them apart.

The measured one, which is what the test produced under the conditions you recorded. And the committed one, which is what goes in the service schedule and needs headroom over the measured one, because the measured one was your best day.

Committing to a number tighter than you’ve demonstrated is how you end up with service credits and an uncomfortable meeting. Committing to one with sensible headroom means a bad day is still a day you meet your obligations on. Nobody has ever been criticised for recovering faster than promised.

This client had been quoting four hours against a demonstrated six hours forty. After the second test they moved to eight, with the phase breakdown attached to justify it. Their clients accepted it without much argument, largely because they could show their working.

this month

Take your next restore test and add one person holding a notepad with the seven phases written on it and a column for wall clock time.

That’s it. That’s the whole change.

Then look at where the time actually went and ask whether your current spending plans point at the biggest number. They very often don’t.

the useful bit

The ninety minute figure was real and the improvement was real, and it still wasn’t the number that mattered. What made it useful was knowing it was ninety minutes of a five hour picture, and being able to say precisely where the other three and a half went.

Next week, the last one. Turning all of this into the document that satisfies an underwriter, a DORA supervisor and a NIS2 competent authority, without writing it three times.