AI said it was done. Was it?

An agent finishes a task and reports success. What has actually been established?

That it produced code. Possibly that the code compiles. Sometimes that a test suite passed and a deployment went out. Every one of those is a real signal, and none of them is the thing anyone actually cares about, which is whether a person using the software can now do what they could not do before.

Three questions that get collapsed into one

• Did the model produce plausible code? Answered by the model itself, which is the problem.
• Did the system accept it? Answered by the compiler, the test suite, and the deploy pipeline.
• Did the outcome happen? Answered by nothing in most pipelines.

The first two are cheap and automated, so they get measured. The third is expensive and manual, so it gets assumed. And because the first two pass loudly and the third fails quietly, the assumption usually holds long enough to ship.

The shape of the failure

Consider a checkout that grants a subscription. The payment integration is the interesting part, so it is the part that gets tested. The entitlement is one line at the end, so it gets assumed. Ship it with that line missing and observe what happens:

• No page errors. Every response is a 200.
• No console exception, no failed request, nothing in the error tracker.
• The unit test on the payment function passes, because that function does exactly what it says.
• A screenshot test passes, because every page renders correctly.
• The success page says the subscription is active, because that page was written by the same person who forgot the line.

Clicking through it as a human feels correct, because the page that confirms success is not the page that would reveal the failure. You have to keep going, to a different screen, and check whether something is true that nobody told you to check.

That is not an exotic bug. It is the most ordinary bug there is, and the entire industry's verification stack looks straight past it.

Why this gets worse, not better

The instinct is that better models fix this. They will certainly write that missing line more often. But the failure is not fundamentally about code quality, it is about who is allowed to declare completion.

A system reporting on its own work has an unavoidable blind spot: it can only check the things it thought of. The line was missing because the author did not think of it. Asking that same author whether they thought of everything is not a check.

As agents write more and humans review less, the number of completion claims grows and the number of people personally confirming them does not. The ratio is the problem, and it moves in one direction.

The agent finished is a different statement from the outcome happened. Most software cannot currently tell the two apart.

What would actually settle it

Something outside the system that produced the work, exercising the real deployment the way a customer would, and reporting what it observed rather than what was intended. Not a claim about the code. Evidence about the result.

That is a narrow thing to build and a hard thing to fake, which is why it is worth building.