Where Memory Lives: Introducing PostgreSQL to a Kubernetes Cluster

Adding PostgreSQL changes what a Kubernetes cluster is being asked to do.

Until this point in the build, most of the workloads have been relatively forgiving. A web service disappears, Kubernetes starts another one. An ingress pod moves to another node. A container is replaced from an image that already contains everything necessary to recreate it.

That doesn't make stateless infrastructure trivial, but it gives us a useful escape hatch: if a process disappears, we can usually make another one.

A database introduces a different requirement.

If PostgreSQL disappears and comes back, creating another PostgreSQL process isn't enough. The new process has to find the same database, containing the transactions that were committed before the interruption. We aren't merely asking Kubernetes to recreate a service anymore. We're asking the surrounding storage and recovery architecture to preserve state across the recreation.

That's a useful boundary to cross deliberately.

The architecture I'm describing remains intentionally modest. There is one authoritative PostgreSQL service on the local infrastructure, running against persistent storage that survives the database pod. Remote copies and backups exist for disaster recovery, but we're not trying to make one database simultaneously writable across several geographic locations.

That decision eliminates some interesting capabilities and a considerable amount of complexity.

It also gives us a very simple question to test first.

What happens if we delete the database pod?

Kubernetes will notice that the expected PostgreSQL instance is gone and create a replacement according to whatever controller or PostgreSQL operator we're using. The persistent volume is made available to the replacement instance, PostgreSQL starts, performs whatever crash recovery is necessary and resumes from the state recorded on disk.

If everything underneath it works as intended, the application eventually reconnects and the database looks remarkably uneventful.

That's exactly what we want.

The interesting part isn't that PostgreSQL restarted. It's that the rows written before the restart are still there.

That distinction sounds obvious until you start testing it.

A stateless application can hide quite a few misunderstandings because recreation is part of its normal behavior. Delete it, replace it and move on. Persistent state makes assumptions accumulate across layers. PostgreSQL depends on the volume. The volume depends on the storage system. The storage system depends on its own replication and failure behavior. The application depends on the database returning with the state it expects.

Deleting the pod gives that whole chain a small disturbance and lets us see what actually survives.

So I would test it early, before the database contains anything difficult to replace.

Create some data.

Delete the pod.

Wait for PostgreSQL to return.

Read the data.

Then make the test less polite.

Restart the node hosting it. Interrupt storage under controlled conditions. Verify that the database behaves the way the design says it should rather than the way we remember configuring it six months ago.

This is where PostgreSQL becomes useful as more than another workload. It forces several layers of the cluster to demonstrate their behavior together.

It also helps separate failure cases that are easy to blur when we use the word redundancy for all of them.

A PostgreSQL process can fail while its data remains intact.

A Kubernetes node can fail while another node and the storage system remain available.

The local storage system can suffer failures that its own replication is designed to tolerate.

Then there is a different category of event in which the entire local site disappears.

Those aren't the same problem, so I don't want one failover mechanism pretending they are.

Within the local environment, automated recovery can make perfect sense. A properly designed PostgreSQL cluster can elect or promote another database instance when the conditions for safe failover have been established. Kubernetes operators can automate a great deal of that work, including health checks, replication management, backups and recovery procedures.

A geographically remote disaster-recovery copy has a different job in this design.

Its existence protects against a failure boundary the local cluster can't cross: loss of the site itself.

That remote copy can receive PostgreSQL's transaction stream, archived WAL, backups or some combination appropriate to the recovery objectives we've chosen. What matters is knowing what guarantees each method actually provides.

Asynchronous streaming replication, for example, allows the local primary to commit without waiting for a distant replica to confirm every transaction. That's attractive when we don't want ordinary database latency tied to a remote network connection.

The tradeoff appears during disaster recovery.

If the local site disappears while some committed transactions haven't yet reached the remote replica, those transactions disappear with it.

No amount of orchestration can recover data that never arrived.

So before building the remote side, we need to decide how much loss we're prepared to accept and how quickly we need to recover. Those are the recovery point and recovery time questions hiding underneath the implementation.

The answers don't have to be zero.

In fact, demanding zero data loss and nearly instantaneous recovery can radically change the architecture and its cost. We may introduce synchronous remote replication, additional voting members, more sites, more complicated fencing or managed services specifically designed to provide those guarantees.

Perhaps the application justifies that.

Perhaps it doesn't.

For a regional publishing system, losing the last few seconds of database activity during the physical destruction of the primary site may be an acceptable tradeoff if avoiding synchronous cross-site replication keeps ordinary operation simpler and more reliable.

That's a decision rather than a deficiency.

Once we've made it, the implementation can match it.

We can continuously archive WAL to remote storage. We can maintain a warm replica elsewhere. We can take independent backups that aren't dependent on the replication topology at all. Ideally we do more than one of these, because replication and backup solve different problems.

Replication faithfully reproduces changes.

That includes changes we regret.

Delete something on the primary and a healthy replica may very efficiently reproduce the deletion. Corruption, operator mistakes and other unwanted state can propagate too. A backup gives us a way to return to an earlier point rather than merely maintaining another copy of the present.

That's why I wouldn't describe the cloud copy simply as a backup.

It may contain replicas, backups and archived transaction logs, each serving a different recovery path.

Then we test those too.

A backup we have never restored is evidence that a backup process ran. It isn't yet evidence that we can recover the database.

So periodically we should create a new PostgreSQL instance from the remote material and see whether it actually works. Measure how long recovery takes. Determine how current the recovered state is. Make sure the credentials, encryption keys, DNS records and configuration needed during recovery don't exist only inside the building we're imagining has just disappeared.

This is where disaster recovery becomes much less glamorous and considerably more useful.

We're no longer saying, "The database is replicated."

We're asking, "Replicated where, how far behind, and what can we actually do with it?"

The same discipline applies to credentials and configuration inside Kubernetes. Secrets shouldn't be baked into container images or committed casually alongside ordinary configuration. The database should have a stable service identity so applications aren't tied to whichever node happens to be running it. Persistent storage should have failure behavior we understand rather than merely a reassuring label in a dashboard.

None of these choices makes PostgreSQL particularly cloud-native.

That's not the objective.

The objective is to make the database's dependencies visible enough that we can reason about what happens when each one disappears.

This is also why I like introducing state gradually into a cluster rather than starting with the most sophisticated architecture we can imagine.

First prove that a pod can disappear without losing the database.

Then prove that a node can disappear.

Then test the storage failures the local design claims to tolerate.

Then prove that a backup can become a working database somewhere else.

Only after those things work does it become useful to argue about faster failover, additional replicas or more elaborate recovery automation.

Each test answers a different question.

Eventually the cluster reaches a point where we know more than whether PostgreSQL is running. We know what happens when its process dies, what happens when its machine dies, what survives outside the building and approximately how much history could disappear if the entire site vanished between two replication events.

That's a much more useful kind of confidence than a green status indicator.

Stateless workloads taught the cluster how to recreate things.

PostgreSQL asks whether it can preserve something while everything around that something changes.

Before we ask it to do anything clever, I'd make sure it can do that.