Let's say the asteroid hits.

Not metaphorically. The building containing the PostgreSQL primary is gone, along with the local cluster, its storage and everything else that normally makes the site authoritative.

Fortunately, there is a remote replica somewhere else.

Now we have to decide when that replica becomes primary.

At first this sounds like an ordinary failover problem. The primary stops responding, monitoring notices, some interval passes and another system takes over. That's how a great deal of highly available infrastructure is designed, and there are good ways to do it. Quorum, fencing and consensus mechanisms exist precisely because distributed systems have spent decades dealing with machines that disappear without explaining why.

The problem in this particular architecture is that absence doesn't tell us what happened.

A destroyed building is absent. So is a building with a severed fiber connection. A failed router can make a healthy database absent from the perspective of another site. So can a configuration error, a power outage or a network partition.

Those failures may look similar from far away while requiring very different responses.

If the original primary still exists and can accept writes, promoting another copy without first establishing authority creates the possibility that two databases will develop different histories. Transactions accepted on one side may never have existed on the other.

That's the failure I'm more interested in avoiding.

PostgreSQL doesn't decide the larger policy for us. It gives us the mechanisms from which replication and promotion can be built, while the surrounding architecture has to determine when promotion is appropriate.

There are environments where automating that decision makes perfect sense. If the system has reliable fencing, enough independent participants to establish quorum and a well-defined procedure that prevents the old primary from continuing to act as primary, automated failover can provide availability without casually accepting split-brain.

That's not the architecture I'm describing here.

I'm starting from a different tradeoff: a locally authoritative system with a geographically distant copy whose primary purpose is to survive the loss of the local site.

Under those conditions, I don't particularly care whether the remote copy becomes writable thirty seconds after the local site disappears.

I care whether it becomes writable correctly.

So if the local site suddenly stops communicating, the remote replica does something remarkably undramatic.

It waits.

Alerts can fire immediately. Monitoring can collect whatever evidence remains available. Recovery tooling can prepare everything necessary for promotion. What doesn't happen automatically is the transfer of authority itself.

Someone first determines what actually happened.

Perhaps the fiber was cut and the local site is fine. In that case there may be nothing to promote.

Perhaps the building really is a crater.

Now the decision is different. We inspect the remote replica, determine its recovery point, understand what transactions may have been lost and decide whether we're willing to make that history authoritative.

Then we promote it.

Everything after that decision can be automated. Scripts can perform the promotion, change service discovery, update routing and bring applications back against the new primary. There's no virtue in making a person manually type twenty commands just because a person made the decision.

The distinction is between automating the procedure and delegating the authority to begin it.

For this system, I want the procedure automated and the exceptional transfer of authority witnessed.

That's partly because I'm optimizing for something different from continuous availability.

If this were a service where several seconds of downtime carried enormous consequences, I'd make different tradeoffs. I'd probably build enough independent infrastructure that the system could establish authority automatically without relying on one site or one human.

A regional organization can tolerate a different failure mode.

If an asteroid has just removed the building, being offline for another ten minutes while someone establishes what survived is not the most important problem we're having.

Preserving a coherent history is.

That realization also changes how I think about remote infrastructure.

For years, conversations about cloud computing were often framed around migration. Something ran locally, and moving it to somebody else's infrastructure was modernization.

That never interested me very much.

We already have machines. We know where they are. We can operate them. For workloads that make sense locally, replacing working local infrastructure with rented remote infrastructure merely because one arrangement is called "cloud" doesn't solve a problem I have.

Geographic separation does.

A second site can't be destroyed by the same fire, flood, electrical failure or unusually determined asteroid. Remote infrastructure therefore offers something our local cluster can't manufacture for itself no matter how many redundant machines we put in the room: distance.

That's a capability worth buying.

It doesn't follow that the distant site needs to become the normal center of the architecture.

The relationship can be narrower. Local infrastructure can remain authoritative during ordinary operation while selected data is replicated elsewhere. Archives can exist in more than one physical place. Services that benefit from geographic distribution can use it. A remote environment can also contain enough of the declared configuration to reconstruct what disappeared.

This is less dramatic than either "everything should be local" or "move everything to the cloud."

It's choosing dependencies according to what they're good at.

The same principle affects how the local sites are organized. If I want another site to reproduce a service after losing this one, the important knowledge can't exist only in my head or in a machine I configured six years ago and have been afraid to touch ever since.

Configuration has to become reproducible.

This is where tools such as GitOps become useful. The repository doesn't make the infrastructure resilient by itself, and it certainly isn't a constitution. It does something more practical: it gives multiple environments a common description of what we're trying to run.

DNS solves another ordinary problem. Applications are easier to move when they know services by stable names rather than by assumptions about which particular machine currently provides them.

Neither idea is profound.

Together they remove a surprising amount of unnecessary attachment between identity and location.

A database service can have a stable name even though the machine providing it changes. A replacement site can converge toward a known configuration rather than being reconstructed from somebody's memory. A remote replica can remain a replica until circumstances require it to become something else.

That begins to look somewhat like federation, although I use the word more cautiously now.

The sites don't need to be sovereign little houses in a village. They need clear responsibilities, enough independence to survive the failures they're expected to survive, and explicit rules about which decisions can be made locally and which require coordination.

Some failures should stay local.

Some state should exist in several places.

Some transitions can be automatic.

Others may deserve a human decision.

The interesting part is identifying which is which before anything fails.

This also changes what redundancy means.

Ten servers in one building can protect against several kinds of failure. They can survive a dead power supply, failed disk or individual machine. They can't provide meaningful geographic redundancy because they're all standing under the same roof.

A remote copy addresses a different boundary.

Neither makes the other obsolete.

The local cluster handles the failures it's well positioned to handle. The distant site handles failures that require distance. The architecture becomes stronger because the two aren't pretending to solve the same problem.

Then one day the asteroid arrives.

The monitoring system loses the local site and raises every alarm it has. The remote database is still there, containing whatever state replication delivered before communication stopped.

It doesn't know there's a crater.

It doesn't need to.

Someone establishes that the original authority isn't coming back, examines what survived and deliberately initiates recovery. The mechanical parts happen quickly because they were prepared long before the emergency.

There is still loss.

Any asynchronous replication scheme can leave a gap between the last transaction committed locally and the last one safely received elsewhere. If we require zero data loss across arbitrary site destruction, the architecture becomes different again, with different costs during normal operation.

That's another decision we should make before the asteroid arrives.

Resilience doesn't mean arranging things so that catastrophe has no consequences. Usually that's impossible.

It means deciding which consequences we can tolerate, preserving enough information and capability to recover from them, and being explicit about where authority lives while we do.

For this system, the cloud doesn't need to be the landlord or the guest.

It's another place where something useful can survive.

And if the day comes when that surviving copy has to become primary, I want us to know that we're moving authority rather than merely reacting to silence.