The easiest way to make Kubernetes difficult is to begin with the architecture you eventually want.

Multiple nodes, redundant storage, automated failover, GitOps, remote disaster recovery and hybrid infrastructure are all reasonable things to think about eventually. They also introduce enough interacting parts that a first installation can fail in several places at once.

I'd rather start with one machine.

Not because one machine is a good production architecture for everything, but because it gives us a small enough environment to understand what Kubernetes is actually doing before we ask it to coordinate anything else.

Assume a fresh Ubuntu 24.04 LTS installation on a physical server or virtual machine we control. It has a predictable IP address, working DNS, SSH access and Internet connectivity for retrieving packages and container images.

Before installing Kubernetes, I want the operating system itself uneventful.

Update it. Make sure DNS resolution works. Confirm that the clock is synchronized. Decide how security updates will be handled. Verify that the machine returns cleanly after a reboot.

If the Kubernetes distribution or configuration we're using has particular requirements around swap, networking, firewall rules or kernel settings, deal with those explicitly rather than applying a collection of commands copied from an unrelated installation guide.

The objective isn't to make Ubuntu special.

It's to establish a known starting point.

For this build, I'm using k3s.

There are many legitimate ways to run Kubernetes, and k3s makes a set of decisions for us that would otherwise have to be made individually. It packages the Kubernetes components into a comparatively compact distribution, provides an installation and service-management path, and includes defaults intended to make a small cluster usable without assembling every component ourselves.

Those defaults matter because installing k3s isn't the same thing as installing a completely empty upstream Kubernetes environment.

That's worth understanding before changing them.

A normal k3s installation gives us a server running the Kubernetes control plane and datastore along with the components necessary for the node to run workloads. Depending on the version and installation options, k3s also includes several packaged components and defaults, including networking and an ingress controller.

So after installation, my first question isn't what else I can install.

It's what just appeared.

Check the service.

Run kubectl get nodes.

Look at the pods running across the namespaces.

Find the kubeconfig.

See which components k3s installed and which ones are merely Kubernetes concepts we've read about but aren't actually present in this cluster.

Then reboot the machine.

This is probably the least impressive Kubernetes test we'll ever perform, which is one reason I like doing it immediately.

When the server returns, does k3s start?

Does the node become Ready again?

Do the system workloads recover?

Can kubectl talk to the cluster?

If something doesn't return, now is a very good time to understand why. There is one machine, almost no application state and relatively little configuration to investigate.

We're buying cheap failures while they're still available.

Once the basic cluster survives a reboot, I want to put one disposable application on it.

Nothing important.

Run a tiny web service whose only job is to return a recognizable response. At first we can reach it from inside the cluster and verify that the service abstraction points toward the expected pod.

Then we can work outward.

This is where ingress enters the picture, although with k3s we first need to inspect the ingress capability already installed.

A default k3s installation commonly includes Traefik. If that meets the requirements, there's no particular reason to replace it merely because another ingress controller is more familiar.

If we've deliberately decided to use something else, such as an NGINX-based ingress controller, I'd disable the packaged component according to the k3s configuration and install the replacement intentionally.

The important thing isn't whether the logo says Traefik or NGINX.

It's whether we understand the traffic path.

A request reaches the machine.

The ingress implementation receives it.

An ingress rule associates the hostname or path with a Kubernetes Service.

That Service directs traffic toward the appropriate application pod.

With one tiny application, we can test the entire path without wondering whether a failure belongs to Kubernetes, the application or twelve other services we installed yesterday.

Use the hostname.

Send a request.

Change the ingress rule and see what changes.

Delete the application pod and watch another one appear.

Restart the ingress component and observe what happens.

We're beginning to accumulate expectations about the cluster by disturbing it deliberately.

TLS adds another component to understand.

For a public service, I want certificate issuance and renewal automated because a certificate that depends on someone remembering a calendar date is eventually going to become an expired certificate.

In Kubernetes, cert-manager is one common way to handle that responsibility. It can represent issuers and certificates through Kubernetes resources and automate ACME workflows with a certificate authority such as Let's Encrypt.

That doesn't make cert-manager mandatory.

The ingress or surrounding network architecture may provide another certificate-management mechanism. A small environment may already have TLS termination somewhere else. Different ingress implementations have different capabilities and operational models.

What matters at this stage is knowing where certificate responsibility actually lives.

If we choose cert-manager, install it, configure the appropriate issuer and request a certificate for the test application.

Then inspect what happened.

Where is the certificate represented?

Where is the private key stored?

How does the ingress implementation obtain it?

What will cause renewal?

What happens if the application pod disappears?

What happens if the ingress pod disappears?

Those questions turn "automatic TLS" from a feature on an architecture diagram into behavior we've actually observed.

Then reboot the machine again.

I like rebooting early systems more often than feels necessary because restarts expose an entire category of accidental state.

Something we started manually doesn't return.

A mount we thought was permanent isn't.

A firewall rule existed only because we typed it at a shell six hours ago.

A configuration file wasn't where we thought it was.

With only one node, the blast radius is the whole cluster, but the cluster doesn't contain anything important yet.

That's the bargain.

We aren't trying to prove availability.

We're trying to prove recoverability.

At this point I'd also begin recording the decisions required to reproduce the machine.

That doesn't mean we need a complete GitOps environment on day one. A small repository containing configuration, manifests and notes about the host can be enough to distinguish deliberate state from accumulated experimentation.

The test is straightforward.

If the machine died and another empty Ubuntu server appeared on the table, what information would we need to reconstruct what we have?

Anything whose answer is "I think I remember what I did" deserves some attention.

That includes node labels.

If this machine represents locally owned infrastructure and that distinction will eventually matter to scheduling, we can express it explicitly through labels. The exact vocabulary is ours to design, but the labels should describe properties we'll actually use rather than a hypothetical architecture that may never exist.

Later, if another class of node joins the cluster, workloads can make scheduling decisions based on those properties.

For now, the label is mostly documentation the scheduler can eventually act upon.

Then I stop.

That's an underrated part of infrastructure work.

There are dozens of things we could install next: dashboards, storage systems, GitOps controllers, service meshes, monitoring stacks and increasingly elaborate ways to manage the relatively tiny web service we've just created.

None of them answers the question this first build is supposed to answer.

Can I operate one Kubernetes node and understand what happens to it?

I want to know how k3s starts. I want to know what it installed. I want a workload to survive the ordinary recreation of its pod. I want external traffic to reach it through a path I can explain. I want TLS renewal assigned to a mechanism I understand.

And I want to reboot the machine without wondering whether it will come back.

That still isn't resilient infrastructure.

The machine is an obvious single point of failure. So is anything stored only on its disks. If its network connection disappears, the applications disappear from the outside world with it.

We haven't solved those problems.

We've made them easy to see.

That's enough for the first machine.

The next useful thing we can do isn't make it more sophisticated.

It's give it something else to fail beside.