From Local-First Infrastructure to Hybrid Kubernetes

Once the local cluster can survive the failures we've actually designed it to survive, the next question isn't how to move it into the cloud.

It's what the cloud could do for us that another machine in the building can't.

That's a narrower question, and I think it produces a better architecture.

Ten servers in one building can provide considerable redundancy. A workload can move when a machine fails. Traffic can enter through more than one node. Storage can survive a disk or server disappearing. Maintenance can happen without shutting everything down.

Then the building loses power.

Or the network connection disappears.

Or something happens that makes the entire site unavailable.

Adding an eleventh server to the same room doesn't address that failure.

Now another location becomes interesting.

It could be another office, a colocated server or infrastructure rented from a cloud provider. The important distinction initially isn't who owns the machine. It's that the machine doesn't share all of the same failure boundaries as the local cluster.

For this architecture, I'll call that remote environment the cloud.

I don't want to stretch the existing Kubernetes cluster into it.

A Kubernetes control plane works best when the machines participating in it have the network characteristics the control plane expects. Extending one cluster across geographically separated networks introduces latency, partitions and dependencies that aren't necessary for what I'm trying to accomplish.

I'd rather have two clusters.

The local cluster remains local.

The remote cluster remains remote.

They can run many of the same applications and use much of the same configuration, but neither needs the other in order to operate normally.

That immediately changes what hybrid means.

There are no cloud worker nodes waiting for the local Kubernetes scheduler to discover them. The scheduler in the local cluster knows only about local nodes. The scheduler in the remote cluster knows only about remote nodes.

If work moves between those environments, something above the individual schedulers has to cause that to happen.

Sometimes that means the same application is already deployed in both places.

Sometimes a deployment process increases replicas remotely.

Sometimes a queue allows workers in either environment to consume work.

Sometimes nothing moves at all until a human decides that a sufficiently large failure has occurred.

Those are different mechanisms because they're solving different problems.

Before implementing any of them, I'd make the remote cluster boring in exactly the same way we made the local cluster boring.

Install it.

Inspect what appeared.

Deploy something disposable.

Send traffic through it.

Reboot things.

Delete things.

Verify certificate issuance and renewal.

If Git is going to describe the desired application state, prove that a clean cluster can recover enough of that state from the repository to become useful again.

The cloud shouldn't receive an exemption from the tests we applied locally merely because someone else owns the hardware underneath it.

Then I would decide what belongs there.

A stateless web application is relatively easy.

If the application can run in either environment without depending on local state that isn't available remotely, we can deploy it to both clusters. The remote copy might normally receive no public traffic at all. It might receive a small amount of traffic continuously so we know it actually works. Or it might be activated only during maintenance or an incident.

Those are operational choices rather than properties of Kubernetes.

Traffic steering happens outside the clusters.

DNS can participate in that decision. So can an upstream proxy or another edge-routing mechanism. The appropriate choice depends on how quickly traffic needs to move, how health is determined, what dependencies remain behind each endpoint and how much additional infrastructure we're willing to operate.

I wouldn't begin with an arbitrary percentage such as eighty percent local and twenty percent remote.

I'd begin with the purpose of the remote copy.

If its job is disaster recovery, zero percent of ordinary traffic may be reasonable.

If its job is also to prove continuously that the remote application works, sending some ordinary traffic there may provide useful evidence.

If its job is genuine overflow during bursts, traffic may shift only after some measured condition is reached.

Those architectures shouldn't be collapsed into one percentage.

Background work gives us another option.

Suppose the local system performs image processing, document conversion or some other task that doesn't need to finish during the HTTP request that created it.

Put the work behind a queue.

Local workers can process the ordinary workload. If demand grows beyond what we want the local machines to handle, remote workers can be started and given access to an appropriate queue.

Now the cloud really can absorb some volatility without becoming the normal home of the application.

Even here, I'd want to know where the queue lives and what happens when the local site disappears.

If remote workers depend on a queue that exists only inside the failed building, we've created cloud workers that become useless during exactly the event we expected them to help with.

Every hybrid feature creates another path worth drawing.

State is where I would slow down considerably.

Running the same container in two clusters is easy compared with deciding which copy of changing data is authoritative.

PostgreSQL makes the problem obvious.

If the local database is primary and a remote copy receives changes asynchronously, the remote environment can provide a valuable recovery path. It does not provide a guarantee that every acknowledged local transaction has already reached the remote site.

There is a gap.

Usually the gap is small.

During the failure that matters most, the size of that gap suddenly matters a great deal.

So before calling the remote database redundant, I want an explicit recovery point objective.

How much recently committed data could we tolerate losing if the entire local site disappeared at this moment?

Seconds?

Minutes?

None?

The last answer changes the architecture considerably.

For a regional publishing system, accepting a small amount of possible data loss during destruction of the primary site may be entirely reasonable. The cost and complexity required to guarantee something stronger may solve a problem whose consequences don't justify it.

Whatever answer we choose should be deliberate.

Then comes promotion.

I don't necessarily want the disappearance of the local site to cause the remote PostgreSQL instance to promote itself immediately.

Silence is ambiguous.

The building may be gone.

The fiber connection may have failed.

The remote environment may simply be unable to reach a perfectly healthy local database that is still accepting writes.

If both sides begin accepting independent writes, recovery becomes a different problem.

For this architecture, catastrophic site failover can remain an explicit decision.

Confirm what happened.

Determine how current the remote copy is.

Record the recovery point.

Promote it.

Change whatever routing or configuration is necessary to make the remote environment authoritative.

The steps after the decision can be automated heavily.

The authority to begin them doesn't have to be.

That distinction won't be appropriate for every system. Some applications require automated regional failover and are designed specifically to make it safe.

This one doesn't necessarily need that.

Files require their own answer.

Some application files may belong in replicated object storage. Others can be copied asynchronously off-site. Some generated artifacts don't need replication at all because they can be recreated from authoritative data.

Again, I wouldn't begin with a product.

I'd begin by classifying the state.

What can't be recreated?

What can be recreated but would be expensive?

What exists only to make something faster?

What needs to be available immediately after site loss?

What can wait until tomorrow?

Only then do MinIO, Ceph, a cloud object store or a simple off-site copy process become answers to actual questions.

This is also where backups need to remain separate from replication.

A remote replica protects against some hardware and site failures.

It can also reproduce a bad deletion, corrupted application state or operator mistake remarkably efficiently.

Backups preserve the ability to return to an earlier state.

I want both questions answered.

Can we continue from another place?

Can we return to an earlier time?

They aren't interchangeable.

Configuration has a similar problem, although Git makes part of it easier.

If application manifests and related configuration are versioned, both clusters can derive much of their intended state from the same repository. Tools such as Argo CD or Flux can automate reconciliation if operating them solves enough problems to justify another controller.

I wouldn't install GitOps merely because two clusters exist.

I'd install it when manually keeping those clusters aligned becomes more dangerous than operating the GitOps machinery.

Even then, I expect differences.

The remote environment may use different storage.

It may normally run fewer replicas.

Some services may not exist there at all.

A disaster-recovery cluster shouldn't have to impersonate the local cluster perfectly in order to be useful.

The objective is to preserve the parts required for the recovery we've chosen.

This is where the phrase cloud migration starts to feel misleading to me.

Nothing important has necessarily migrated.

The local cluster still handles ordinary work. Local hardware still provides the steady-state capacity. The remote environment exists because geography gives it something the local cluster can't manufacture internally.

Distance.

That distance is useful.

It is also expensive in ways that aren't always financial.

Remote state arrives later. Network dependencies multiply. Recovery procedures become more complicated. Someone else's infrastructure enters the failure model. Services available in one environment may not exist in the other.

So I want to spend that complexity where distance actually buys something.

Off-site backups clearly benefit.

Disaster recovery clearly benefits.

Some burst workloads may benefit.

Public applications that need geographic redundancy may benefit.

Other workloads may gain almost nothing.

Hybrid architecture doesn't require every service to participate equally.

By the time the remote cluster becomes useful, I should be able to describe its job without saying simply that we "use the cloud."

Maybe it can serve the public website if the local site disappears.

Maybe it holds a recoverable PostgreSQL copy with an understood replication delay.

Maybe it stores backups that survive loss of the building.

Maybe workers can be added there when a queue grows beyond what the local machines should handle.

Each of those is a specific capability attached to a specific failure or capacity boundary.

That's enough.

The cloud doesn't need to become an extension of the local cluster.

It doesn't need to become headquarters either.

It's another place.

The useful question is what becomes possible because that place fails differently from ours.