This is the third essay in a series about AI alignment.

The first two essays looked at guardrails and the cost of learning. I was working toward a distinction that still seems important to me, although I'd frame it somewhat differently now.

We can test an AI system before deployment. We can establish constraints, evaluate known failure modes, document expected behavior and decide that the system has met whatever standard we've established for release. All of that matters. What we can't establish in advance is that we've represented every situation the deployed system will encounter.

The world gets another turn.

People use systems differently than their designers expect. New capabilities interact with old ones. A behavior that looked safe in isolation creates a problem when placed inside a larger workflow. A safeguard blocks something nobody realized it would block. An unusual user discovers a case that wasn't represented in the evaluation set.

None of this means the original evaluation was useless. It means the evaluation had a boundary.

That's where I think alignment becomes a practice.

Not because alignment can't include stable properties or constraints, and not because every decision has to remain permanently open. It's a practice because operating the system keeps producing evidence, and someone has to decide what to do with that evidence.

That distinction changes how I think about guardrails.

I once treated guardrails and learning as opposing approaches. A guardrail prevents a behavior, while learning supposedly requires encountering the consequences of behavior. From there it's easy to conclude that preventing mistakes also prevents the system from learning why those mistakes matter.

I don't think that follows.

Some mistakes should be prevented.

If we already know that a particular action creates an unacceptable risk, allowing the action merely so the system can experience its consequences would be a strange definition of learning. We don't teach every human lesson that way either. Much of what any of us knows arrived through instruction, observation, inherited practice and warnings from people who encountered the consequence before we did.

Guardrails can carry some of that accumulated knowledge.

The interesting question is what happens when the guardrail encounters a situation its designers didn't anticipate.

Perhaps it blocks something harmless because the new situation resembles a dangerous one. Perhaps it permits something dangerous because the relevant distinction wasn't represented in the rule. Perhaps the boundary is sound but the surrounding interaction makes it difficult for a legitimate user to accomplish something safely.

Now the guardrail itself has produced information.

The practice begins with noticing.

That sounds easier than it is because organizations are very good at turning unexpected behavior into categories that require no further thought. The model failed. The user misused it. The safeguard worked. The test passed. The incident was resolved.

Sometimes those descriptions are accurate. The problem is treating the category as the end of the investigation.

If a system behaved differently than we expected, I want to know why we expected something else. What information was available? Which assumptions were embedded in the evaluation or safeguard? Did the system fail to make a distinction we thought it could make? Did we ask it to make a distinction that isn't actually available from the information it had?

Those questions may produce a model change, a new evaluation, a different interface, another safeguard or no change at all. Investigation has to preserve the possibility that the original behavior was appropriate and our expectation was wrong.

That's important because feedback doesn't interpret itself.

A complaint isn't proof of misalignment. Neither is the absence of complaints proof that the system is aligned. Users have partial views, developers have partial views, evaluators have partial views, and the model's own explanations provide another partial view. What happened has to be reconstructed from whatever evidence those positions can provide.

That makes alignment less like maintaining a particular answer and more like maintaining the ability to investigate when the answer stops fitting.

There are some practical requirements for doing that.

The first is continuity. Someone has to retain enough knowledge of the system and its history that today's incident can be compared with yesterday's decisions. That doesn't require one person to hold the entire architecture in their head, which becomes increasingly unrealistic as systems grow. It requires enough documentation, institutional memory, monitoring and communication that relevant knowledge can survive changes in personnel and scale.

External auditors can contribute to that process too. So can independent researchers, red teams and users. An outside observer may notice assumptions that became invisible to the people who work with the system every day.

What can't be outsourced completely is the organization's responsibility for deciding what to do with what those observers find.

A security audit can identify a vulnerability. It can't make every subsequent tradeoff for the organization operating the system. An evaluation can expose a failure mode. It can't determine by itself how that failure should be balanced against other requirements, whether the proposed correction introduces another problem or when enough evidence exists to deploy the change.

Those decisions remain part of operating the system.

The second requirement is that unexpected information has somewhere to go.

Logging everything isn't enough. Neither is collecting enormous quantities of user feedback. An organization can possess more observations than anyone could reasonably examine and still learn very little from them.

Some mechanism has to distinguish ordinary noise from cases worth investigating, while retaining enough humility to recognize that the mechanism doing the filtering may itself miss things.

This is where disagreement becomes useful.

A user says the system is wrong while the evaluation says it's behaving correctly.

A safety layer classifies a request as dangerous while a domain expert says the distinction being used doesn't make sense in that context.

A model gives the expected answer, but something about the interaction still produces an unexpected outcome.

None of those disagreements tells us automatically which observer is right. They identify boundaries worth examining.

The third requirement is the ability to change something after we've learned.

That sounds obvious, but mature systems accumulate dependencies. A safety behavior becomes part of an evaluation suite. Applications begin depending on it. Policies are written around it. Users adapt their workflows to it. Eventually a decision that began as a provisional response to one problem can become expensive to revisit.

Sometimes that's good. Stability has value. People need to be able to depend on behavior remaining reasonably consistent.

The difficulty is distinguishing accumulated knowledge from accumulated inertia.

A mature alignment practice needs both. Some decisions should become difficult to change because experience has repeatedly supported them. Others should remain revisable because the evidence underneath them is weaker or the environment around them continues to change.

There isn't a general rule that tells us which is which.

That's why the practice matters.

When an incident occurs, the objective isn't merely to restore expected behavior as quickly as possible. Neither is it to turn every incident into a philosophical reconsideration of the entire system. Most failures probably don't reveal anything fundamental.

Occasionally one does.

The skill is retaining enough attention to notice the difference.

This is also where I would be careful with the idea that an aligned system must preserve the capacity to be surprised. Surprise itself isn't a virtue. In some parts of a system, surprise is exactly what we're trying to eliminate. A payment shouldn't unexpectedly execute twice. A privacy boundary shouldn't occasionally become adventurous. A safety mechanism shouldn't reinterpret an established prohibition every time it encounters a creative prompt.

What we need isn't surprise everywhere.

We need somewhere for surprise to register when it occurs.

That can be an incident report, an evaluation failure, a user complaint, an anomalous metric, an external audit or a researcher noticing behavior nobody thought to test. The important property is that unexpected evidence isn't discarded merely because the existing system has no category for it.

This makes alignment expensive in a way that additional compute can't entirely solve.

It consumes attention.

Someone has to investigate cases that don't fit. Someone has to maintain evaluations, revisit assumptions, compare new behavior with old behavior and decide whether a correction actually improved anything. Automation can help with much of that work, and AI systems themselves may increasingly assist with it, but deciding what counts as acceptable behavior remains connected to people and institutions affected by the result.

That creates pressure.

Deployment has schedules. Organizations have budgets. Users want capabilities now. Engineers get tired of investigating edge cases that may lead nowhere. Leadership wants metrics that summarize whether the system is safe enough to ship.

Those pressures aren't evidence of moral failure. They're part of operating anything at scale.

The danger appears when the summary begins replacing the thing it summarizes.

A test pass becomes "the system is safe."

A low incident rate becomes "the problem is solved."

Compliance with the process becomes evidence that the decision produced by the process must have been good.

Metrics and procedures are useful precisely because nobody can examine everything directly. The practice is remembering what those representations can and can't establish.

That gives me a more modest definition of alignment than the one I started with.

I don't think alignment is a state we can verify once and then possess indefinitely. Neither do I think it's an endless conversation in which nothing becomes settled.

Some things can become stable.

Some constraints can be strong.

Some tests should be difficult to change.

Practice is what surrounds those settled things. It notices when circumstances have changed, when an old distinction no longer works, when a new failure appears or when a correction creates a problem somewhere else.

Then it asks whether anything needs to move.

Sometimes the answer is no.

That matters too.

A system capable of learning isn't one that changes whenever new information arrives. It's one capable of distinguishing information that should change it from information that shouldn't, while remaining aware that this distinction can itself be wrong.

We can't guarantee that process will always produce the right answer.

What we can preserve is the possibility of correction.

That's what I now mean when I say alignment is a practice rather than a state.

The tests matter. The constraints matter. The people matter. The feedback matters. The accumulated decisions matter.

Then the system encounters something we didn't expect.

What happens next is part of alignment too.