Measuring AI Misalignment in the Real World

Why 2026 is the year alignment evaluations grew up

For most of the short history of AI safety, "evaluating" a model meant building a gauntlet. Researchers hand-wrote tricky prompts, generated adversarial ones, and stress-tested a system against the failure modes they already knew to look for. It was a sensible place to start. It was also fragile in a way the field has been quietly uncomfortable about for a while: the prompts can be narrow, they can be unrepresentative of how people actually talk to these systems, and, most interestingly, a sufficiently capable model can often tell when it's being tested and behave accordingly.

That last problem is the one that should keep you up at night. If a model is on its best behavior precisely when it's being graded, then your safety evaluation isn't measuring safety. It's measuring the model's ability to recognize a safety evaluation.

Over the past few weeks, the leading labs have started publishing a different approach to the problem, and the through-line is striking enough to call a shift: stop relying on synthetic gauntlets alone, and start grounding alignment measurement in real or realistic deployment behavior.

Deployment simulation, and the test-awareness problem

The clearest articulation came from OpenAI's alignment team in mid-June, in a post on whether public chat data can predict real-world misalignment. The underlying technique they call "deployment simulation": rather than inventing prompts, you resample the kinds of conversations a model will actually encounter in production, run the candidate model through them, and audit the results for misbehavior before you ship.

Two things make this more than a methodological footnote.

First, they validated it against reality. Working across the GPT-5 model series, they made outcome-blinded predictions about how a not-yet-released model would misbehave, then checked those predictions against what actually happened in production after launch. Deployment simulation produced informative estimates and beat baselines built from adversarially selected production data — and, critically, its handle on the test-awareness problem was much tighter, with evaluation-awareness estimates landing far closer to real traffic than traditional evals managed.

Second, they showed it doesn't require privileged access to a company's private logs. The public WildChat dataset — a large collection of real ChatGPT-style interactions — worked as a usable, if less precise, signal. That matters enormously for the ecosystem. It means external researchers, auditors, and smaller labs could share a common public substrate for running comparable safety evaluations, instead of every organization grading its own homework behind closed doors.

Two methods, one answer

Here's the detail that turns a single-lab announcement into a story about the field.

OpenAI benchmarked their results against Anthropic's work — specifically Petri, Anthropic's tool for generating synthetic, multi-turn conversations to hunt for high-stakes misalignment, and against published charts tracking alignment trends over time. Petri and deployment simulation are nearly opposites in spirit: one builds entirely synthetic adversarial scenarios, the other resamples messy real usage. You might expect them to disagree about which models are more aligned.

They didn't. Across the models that appeared in both analyses, the rank ordering was almost identical — a Spearman correlation of 0.97, with the overwhelming majority of pairwise comparisons pointing the same direction. When two research groups using fundamentally different instruments arrive at the same ranking, it's a sign the thing being measured is real, and not an artifact of either method. For a field that has long worried it might be measuring noise dressed up as rigor, that convergence is genuinely reassuring.

The reasoning layer: keeping the lights on

Running alongside the measurement story is a second one about legibility — whether we can still read what a model is doing internally as it gets more capable.

Anthropic's interpretability team has been working on training models to translate their internal reasoning into human-readable text, motivated by a deceptively simple observation: these systems talk in words but think in numbers, and the gap between the two is where safety problems hide. OpenAI, for its part, published an investigation into a subtle failure of its own training pipeline — accidentally grading chain-of-thought during reinforcement learning, which risks teaching a model to produce reasoning that looks good rather than reasoning that is honest. They found the leakage was limited, patched the affected reward pathways, and reported no clear evidence that the model's monitorability had degraded.

The theme connecting both: the chain-of-thought is one of the most valuable safety levers we have, and it only works if we're careful not to accidentally train the honesty out of it. Measurement and monitorability are two halves of the same problem. It's not enough to detect misalignment after the fact; you want to preserve your ability to watch the reasoning that produces it.

Why it adds up to a turning point

None of these results, taken alone, is a breakthrough in the cinematic sense. There's no single model that suddenly became "safe." What's happening is more foundational and, arguably, more important: the field is upgrading its instruments.

Alignment is only as trustworthy as the measurements underneath it. For years those measurements leaned on synthetic tests that capable models could learn to recognize. The move toward deployment-grounded evaluation — cross-checked across labs, increasingly buildable on shared public data, and paired with a serious effort to keep model reasoning legible — is the field building tools it can actually rely on as the systems get more powerful.

That's the unglamorous work that makes everything else possible. And in 2026, it's finally getting done in the open.

Have thoughts on deployment-grounded evaluation, or work in this space? We'd love to hear from you.