A single-node Kubernetes cluster can teach you quite a lot.

You can learn how workloads are deployed, how services find each other, how ingress works, where configuration lives and what happens during an ordinary restart. You can build something useful enough that Kubernetes stops feeling experimental.

Then you add another machine and discover how much of what you learned depended on there being only one.

That's why I like adding the second node relatively early.

I'm not doing it primarily for capacity. If the first machine isn't overloaded, another sixteen gigabytes of memory may accomplish nothing immediately. The value is that two machines give the cluster somewhere else for work to exist, which makes a whole category of assumptions visible for the first time.

Before adding it, I want the first node reasonably understood.

In this example I'm using k3s on Ubuntu. The existing server boots cleanly, its services come back without intervention, certificates renew, ingress works and I can explain enough of the installation that rebuilding it wouldn't require archaeology.

Then I prepare another machine.

I generally prefer similar hardware and the same operating system because variation creates additional questions, and right now I'm interested in Kubernetes rather than whether two different Linux configurations behave differently under load. The machines don't have to be identical, but predictable addressing, synchronized time, compatible networking and a common approach to updates remove variables that aren't particularly interesting yet.

There will be plenty of opportunities to introduce differences later.

Joining the machine to k3s is mechanically simple. The existing cluster provides the information and credentials necessary for another node to join, and the new machine is configured to contact the existing server.

A short time later, kubectl get nodes shows two nodes.

That is not redundancy.

It is two nodes.

The distinction is worth preserving because Kubernetes doesn't automatically make every dependency redundant merely because another machine has appeared.

If the second machine joined as an agent, for example, the original k3s server may still contain control-plane responsibilities that the agent can't replace. An ingress controller may still have only one replica. Persistent data may still exist on storage associated with one machine. Applications may have only one running instance.

We've increased the number of places where some workloads could run.

Now we need to discover what actually can.

I wouldn't start by deploying a pile of new applications. I'd give the second node something deliberately unimportant to do.

Run a disposable workload there. Delete its pod. Watch Kubernetes recreate it. Drain the node and see what moves. Bring it back and watch what returns.

Then inspect what didn't move.

This is where the second node starts paying for itself.

A workload may be constrained to the original machine because of an unnoticed node selector. A volume may be local. An application may have a dependency reachable only through the first node. A service that looked like part of the cluster may turn out to be something running beside Kubernetes on one particular server.

None of these discoveries means the design is wrong.

Some components should be tied to particular machines.

The important part is knowing which ones are.

Then I start removing machines.

Initially I do it politely.

Drain the second node and verify that workloads eligible to run elsewhere actually do. Reboot it. Let it rejoin. Watch the events Kubernetes produces and make sure I understand what happened.

After that, I become less polite.

Turn it off.

Disconnect its network.

Don't give Kubernetes the graceful sequence it received during the previous test.

Real failures don't usually begin with kubectl drain.

The result may look almost identical from the application's perspective, or it may reveal assumptions the graceful test never exercised. Kubernetes has to recognize that the node is unavailable. Workloads may remain associated with it for some period before replacements appear elsewhere. Anything depending on local state may behave differently from a stateless service that can simply be recreated.

Those differences are what I'm trying to learn.

Then comes the more interesting experiment.

What happens if the first machine disappears?

If that machine is still the only k3s server and the second node is merely an agent, this isn't symmetrical with losing the agent.

Some workloads already running on the agent may continue doing useful work, depending on their dependencies and the surrounding network. What the cluster can schedule, reconcile or change without its control plane is another question.

That's not Kubernetes failing its redundancy test.

We haven't built a redundant control plane.

The test reveals the distinction.

This is one reason I don't like saying that a cluster is simply "highly available" or "redundant." Those descriptions compress too many separate properties into one reassuring adjective.

I would rather make a small table, even if it's only in my head.

Can I lose a workload process?

Can I lose this node?

Can I lose the node running ingress?

Can I lose a control-plane node?

Can I lose a disk?

Can I lose the switch?

Can I lose the building?

The answers don't need to be yes.

They need to be known.

With only two machines, many of them will still be no.

That's useful.

We've found the boundaries of the architecture while we can still draw most of it on a sheet of paper.

The second node also teaches something about scheduling that wasn't obvious when every workload had exactly one possible destination.

Kubernetes doesn't generally move a healthy workload merely because a new node has joined and looks lonely. Existing pods can remain where they are until some scheduling event gives Kubernetes a reason to place a new pod.

That means adding capacity and redistributing existing work are different operations.

Again, I like the explicitness.

If I want replicas spread across nodes, I should express that intention through the workload and scheduling configuration rather than assuming Kubernetes shares my idea of what a balanced cluster looks like.

The same applies to maintenance.

Once workloads can run on more than one machine, draining a node becomes interesting. I can deliberately remove it from service, perform operating-system or hardware work, return it to the cluster and then do the same thing somewhere else.

Now maintenance can begin moving through the infrastructure rather than requiring the entire infrastructure to stop.

That is a meaningful improvement.

It still isn't proof that we're ready for twenty nodes.

A larger cluster introduces problems two lightly loaded machines may never expose. The datastore and control plane need an architecture appropriate to their size and availability requirements. Networking carries more traffic. Storage gets more complicated. Scheduling decisions have more possible destinations. Upgrades affect more machines. Monitoring has more events to summarize, and hardware failures become ordinary enough that operational procedures matter more.

Twenty nodes aren't merely the second node repeated nineteen times.

Still, the second node changes the kinds of questions we're able to ask.

With one machine, a surprising amount of Kubernetes can remain conceptual. We know workloads are schedulable, but there is nowhere else for them to go. We know nodes are replaceable, but we've never asked the cluster to operate while one is absent.

The second machine lets us stop imagining those properties and start testing them.

That prepares us for the next steps.

If losing the node running ingress makes the applications unreachable, we know what to work on next.

If state can't follow a workload to another machine, we've found another boundary.

If losing the only control-plane server leaves us unable to manage the cluster, we can decide whether our availability requirements justify changing that topology.

Each failure tells us something different.

That's the pattern I want to preserve as the cluster grows: add one capability, disturb it deliberately, understand the result, and only then build something that depends on it.

Eventually there may be ten machines.

Perhaps twenty.

By then I don't expect individual machines to feel unimportant. A failed power supply is still a failed power supply, and somebody still has to replace it.

What I want is for the failure of an ordinary machine to have a consequence we've already encountered.

That's the change the second node begins.

The cluster doesn't become infrastructure because it has more than one machine.

It starts becoming infrastructure when losing one of them becomes something we've practiced.