Replicating the Front Door and the Basement

Once a Kubernetes cluster can lose a worker node without turning the event into an emergency, it's tempting to call the cluster redundant.

Some of it may be.

That's a useful distinction to make before adding anything else.

Kubernetes can reschedule a workload after a node disappears, but that tells us only that another node is capable of running the workload. It doesn't necessarily tell us that users can still reach it or that the data it needs will be available when it arrives somewhere else.

Those are separate problems.

I like dealing with them separately because otherwise the word redundancy starts covering so many things that it stops telling us much.

First, consider ingress.

Suppose an ingress controller is running on one node. Kubernetes may be perfectly capable of recreating that controller elsewhere if the node fails, but traffic still needs some route to the replacement.

Running multiple ingress-controller replicas is a useful first step because it removes one process from the failure path. If possible, I want those replicas placed on different nodes so the loss of one machine doesn't remove every ingress instance at once.

Then I test it.

Send traffic through the cluster and remove one of the ingress pods.

If nothing interesting happens, remove the node hosting one of them.

Now we find out whether the rest of the network architecture agrees with our definition of redundant.

It may not.

Perhaps incoming traffic still depends on an address associated with the failed node. Perhaps an upstream router has nowhere else to send it. Perhaps Kubernetes has several healthy ingress instances that the outside world has no way to reach.

That's not a contradiction. We've simply reached the next failure boundary.

Now we need some mechanism outside the ingress pods themselves that can direct traffic toward a healthy node.

On a small on-premises network, that might be a virtual IP capable of moving between machines. Another environment might use a pair of load balancers, router integration, BGP announcements or some other mechanism appropriate to the network already in place.

The particular product matters less to me initially than being able to draw the path.

A request arrives at the public address.

Something determines where that request goes.

The selected node accepts it.

The ingress controller routes it toward the application.

Then I should be able to point at each step and ask what happens when it disappears.

This is where diagrams become considerably more useful than adjectives.

Instead of saying ingress is highly available, draw the request path and start crossing things out.

One ingress pod disappears.

One worker disappears.

A switch disappears.

The machine holding a virtual address disappears.

The Internet connection disappears.

At some point traffic stops.

That's fine.

The objective isn't to construct an infrastructure diagram on which every conceivable X can be crossed out without consequence. It's to know which failures we've designed around and which ones we haven't.

Storage introduces a different set of questions.

Early in a Kubernetes build, node-local storage can actually be useful because it makes state difficult to ignore. A pod running on one node writes to a disk on that node. Move the pod somewhere else and the relationship becomes obvious.

The new pod doesn't automatically bring the old disk with it.

Before solving that problem, I want to know whether it needs solving.

Some data is disposable. Caches can often be recreated. Temporary processing files may not matter after the job finishes. Container images can be pulled again. Derived artifacts may be cheaper to regenerate than to replicate continuously.

Other state needs a durability plan.

That doesn't necessarily mean every important workload should receive replicated Kubernetes storage. A database may implement replication at the database layer. User uploads might belong in object storage. Some datasets may be adequately protected by local persistence plus backups, while others need to remain available through a node failure with almost no interruption.

The application determines the requirement.

Only then should the storage architecture answer it.

If we decide to provide distributed block or filesystem storage from the local cluster, replication introduces another set of failure boundaries to understand.

How many copies exist?

Where can those copies be placed?

How does the storage system decide that a copy is current?

Does it require quorum?

What happens when a node disappears rather than merely shutting down cleanly?

What happens when it returns?

Those questions matter more than the reassuring fact that the interface says replicated.

Two copies, for example, sound considerably safer than one. Depending on the storage system, however, losing connectivity between those copies can create a decision problem rather than merely a durability problem. A system designed around quorum may deliberately require three members or some other arrangement so that a network partition doesn't leave two sides independently believing they should continue.

This is why I wouldn't begin by selecting a replication factor from a tutorial.

I'd begin with the failure we're trying to survive and work backward.

Lose one disk.

Lose one node.

Lose connectivity between nodes.

Restart the failed member.

Fill a disk.

Let a replica fall behind.

Then observe what the particular storage system actually does.

Storage tends to make this process more interesting because failure isn't always binary. A process is relatively easy to describe as running or not running. Storage can be available and slow, available and rebuilding, healthy from one node and unreachable from another, or apparently functional while redundancy has quietly degraded.

That makes monitoring important, but it also makes testing important.

A dashboard telling me that three replicas exist isn't the same as demonstrating that the workload remains usable after one of them disappears.

This is where ingress and storage begin to look less like two versions of the same redundancy problem.

With ingress, we're primarily asking whether another usable path exists when part of the traffic path fails.

With storage, we're asking several questions at once: whether the data survives, whether it remains accessible, and whether the system can determine which state should be used after failures and recovery.

That doesn't make storage more truthful or ingress less important.

It means they have different failure semantics.

The distinction becomes particularly important once we add a database.

If a web request fails because ingress is unavailable, the user can often try again.

If a database acknowledges a transaction and later loses it, retrying isn't necessarily equivalent. If two database instances independently accept conflicting writes, simply reconnecting them doesn't tell us which history should survive.

That's why I want the storage behavior understood before asking PostgreSQL to depend on it.

Not because the cluster has to complete some philosophical progression from traffic to memory, but because each layer introduces assumptions the next layer will inherit.

Ingress assumes something about the network.

Persistent volumes assume something about storage.

PostgreSQL will eventually assume something about those volumes.

The application will assume something about PostgreSQL.

Testing the layers incrementally makes it easier to discover which assumption was wrong.

There's another reason to move slowly here.

Redundancy costs something.

A second ingress replica consumes very little, but additional load balancers, switches, storage replicas and network paths accumulate hardware, configuration and operational complexity. Every redundant component can itself become something that needs upgrading, monitoring and understanding.

At some point the infrastructure designed to eliminate single points of failure can acquire enough moving parts to create new failure modes.

That doesn't argue against redundancy.

It argues for knowing what each additional copy is buying.

A small regional operation might reasonably tolerate several minutes of website downtime while refusing to tolerate the loss of subscriber records. Another application might require continuous public availability while being able to regenerate almost all of its internal data.

Those systems shouldn't have identical architectures merely because both run Kubernetes.

The requirements come first.

By the time ingress and storage have both survived the disturbances we've designed for, I do notice a change in how I think about the cluster.

I'm less interested in which machine normally runs something.

That doesn't mean machines have disappeared.

They become most visible precisely when a failure crosses a boundary we haven't made redundant.

What changes is that ordinary operation depends less on a particular machine continuing to exist.

A workload can move.

A traffic path can change.

Some forms of state can remain available while the hardware underneath them changes.

Now conversations about off-site recovery become easier because we've developed a better vocabulary for what we're trying to protect.

A remote copy doesn't simply make the cluster "more redundant." It addresses a different boundary.

Ten machines in one building can protect against quite a few machine failures.

They can't protect the building.

That is where another site becomes interesting.

It might be a second office. It might be colocated hardware. It might be a cloud provider. The implementation can wait until we've decided what has to survive that larger failure and how quickly we need it back.

The same method still works.

Draw the path.

Identify the state.

Choose the failure.

Remove something.

See what remains.

Then decide whether what remains is enough.

That's what I mean when I say I want the cluster to become boring.

Boring isn't a property the architecture eventually acquires.

It's what happens when we've seen a failure before and know what the system does next.