Your Infrastructure Has an RPO Too

Cohesity and HashiCorp announced something last week that’s more interesting than the press release makes it sound.

Cohesity’s cloud rebuild service has been validated against Terraform. It reads your infrastructure code, works out which cloud resources to protect and in what order to bring them back, then backs up the code and the application data on their own schedules. When you recover, it recreates the environment from the stored code and restores the data separately.

The part worth stopping on is small. The product shows you the timing gap between the infrastructure backup and the data backup.

That’s an odd thing to build a feature around until you sit with it. Somebody worked out that these two things are protected on different cycles, that nobody knows how far apart they are, and that the gap matters enough to put on screen. They’re right, and most organisations running infrastructure as code have never measured it.

The Claim Worth Testing

The pitch for infrastructure as code has always carried a disaster recovery argument in the side pocket. If the whole environment is defined in a repository, you can rebuild it anywhere. New region, new account, new provider. The environment becomes reproducible, and reproducible sounds a lot like recoverable.

It’s a good argument and it’s half right, which is the dangerous proportion.

Code describes the shape of an environment. It doesn’t hold its state. Nearly everything that makes a recovery difficult lives in the gap between those two, and that gap goes unexamined because the repository looks so complete when you scroll through it.

What the Code Doesn’t Hold

Start with the obvious one. Your code will stand up an empty database very reliably and holds nothing about what was in it. The data restore is a separate job, on a separate schedule, from a separate system. That’s the gap Cohesity put on screen, and in most estates it’s hours at best.

Secrets aren’t in there either, and quite right too. Nobody commits credentials or certificates to a repository. So a rebuild from code gives you an environment with no secrets in it, and those live in a vault which is itself infrastructure that needs recovering before anything depending on it. The sequencing problem from part three of the restore testing series turns up here wearing a different coat.

Then there’s the one that catches people who have actually tried this. Rebuilt resources come back with new addresses. Anything still holding the old one keeps pointing at the old one, and that list is longer than anyone expects: DNS, access policies written by another team, a partner’s allowlist, an application’s config, your monitoring, the backup job itself. The code is internally consistent. The world around it isn’t.

And then drift, which is the quiet one. Every environment collects changes made at two in the morning during an incident by somebody who meant to go back and codify it later. A firewall rule opened for a vendor. A setting adjusted to stop something falling over. None of it reaches the repository. Rebuild from code and you get the environment as the code believes it to be, which isn’t the environment that was running, and the difference only shows up when whatever depended on that change stops working.

The Map Was Stored in the Estate

Here’s the part that gets skipped in nearly every conversation I have about this.

Infrastructure code needs a record of what it already manages, and that record lives somewhere in your cloud estate. So ask where. If it sits in the same account or region you’re recovering, then it’s inside the blast radius of the thing you’re recovering from. The map to the estate was kept in the estate.

The same question applies to a few other bits that look like tooling and are really dependencies. The identity your pipeline signs in with. The system that runs the deployment. The vault. Each has to be standing before it can help rebuild anything, and some of them are defined in the code they’d be used to deploy.

It’s the chicken and egg problem the on-premises world sorted out decades ago by keeping the recovery toolchain outside the production domain. Cloud estates have quietly reintroduced it. Worth putting on an actual page: what has to be alive before your first deployment can run at all, and is any of it in the same place as production.

The Number Nobody Has

Everybody can tell you their data RPO. It falls out of the job schedule, give or take the completion time problem I’ve gone on about before.

Almost nobody can tell you their infrastructure RPO. The question is simple enough. As of this morning, how far is the running environment from the code that’s meant to describe it? Not when was the last commit. When did the code last match reality.

In a mature platform team with policy gates and no console access, that might be minutes. In a typical estate where a handful of people have admin and the repository is maintained with good intentions, it’s longer, and the parts that have drifted are disproportionately the parts somebody changed in a hurry because something was broken. Which is to say, the parts that matter.

That’s what Cohesity’s feature is pointing at, even if it’s only showing the gap between two backup schedules rather than the gap between code and reality. The schedule gap is measurable and they’ve measured it. The drift gap is the bigger one and it’s still yours.

This Month

Find out where the record of your managed infrastructure is kept, and whether it sits inside the environment it describes. If it does, that’s half a day of work to fix and it’s the highest value thing on this page.

Then run a comparison between the code and what’s actually running, and read what comes back. Not to apply it, just to see the size of the gap. If nobody has done it before, set aside more time than you think for reading the output.

Write down what has to be alive before your pipeline can deploy anything at all, and check how many of those things are defined in the repository they’d be deploying.

And if you’re buying or selling cloud rebuild as a capability, ask the question the Cohesity feature implies. What’s the gap between the infrastructure recovery point and the data recovery point, and has anyone looked at it since the thing was set up.

The Useful Bit

Infrastructure as code made environments reproducible, and reproducible has been quietly standing in for recoverable ever since.

They’re not the same. A repository gives you the shape of an environment, which is a great deal more than most estates had ten years ago, and nothing at all about the data in it, the secrets it needs, the addresses other systems are holding, or the change somebody made at two in the morning and meant to write up later.

The useful move isn’t to distrust the code. It’s to know the size of the gap, because at the minute it’s the one recovery number most teams have never put a figure on.