When an Agent Says It Is Done, You Have Received a Claim, Not a Result

A recent Microsoft and Hugging Face write-up of the ThinkingBox benchmark opens with a customer-service agent that makes nine well-formed tool calls and correctly finds that the customer does not qualify for compensation. It then closes the ticket as resolved. The carrier exception that started the conversation is still open, and the required end state was to put the case on hold. Every step looked fine and the final state was wrong.

The authors compress the lesson into one line: a trajectory is a claim, and database state is the evidence. This article takes that line seriously as an operating rule. Wherever a dashboard counts an agent’s own report, or a score computed from it, as the record of what happened, it is relying on the weakest evidence available, and the repair is a verification step placed in the system, not a better prompt.

Two independent measurements of the same gap

The ThinkingBox paper, One Success Isn’t Reliability, runs 507 policy-conditioned business workflows across five domains, including retail, auto insurance and banking, each against a clean backend, and grades the final database state with deterministic checks. In a twelve-model ablation covering 121,680 valid trials, 79,853 attempts failed an executable check. The number that matters for operations is the next one: 67.24% of those failed attempts ended cleanly, called at least one state-changing tool, and reported no final tool error. Roughly two out of three failures, in other words, would slip past ordinary tool-error alerts. These are the benchmark authors’ own measurements, not independent validation of their benchmark.

The authors also break down what went wrong inside those failures, with overlapping categories: a wrong field value in 77.61%, an unintended extra effect in 43.30%, and a missing required effect in 25.36%. The extra-effects figure deserves attention, because it means that checking whether the target record is correct is not enough. An agent can set the right value and also touch something it should not have.

A second source reaches a similar conclusion by a different route. A June 2026 paper, From Confident Closing to Silent Failure, defines false success as a mismatch between the agent’s claim of completion and the programmatic environment state. Across 9,876 tau2-bench trajectories it finds that 45 percent of failures in the airline domain and 47 percent in retail were false successes, with 3 percent in telecom, and 75.8 percent of the failing AppWorld trajectories in which the agent made an explicit status claim. The rate varies a lot by model, from 13 percent to 79 percent on tau2-bench.

Treat the two results as consistent in direction, not as one number. The benchmarks and labeling rules differ, and the second paper is a single-author workshop paper whose 3 percent telecom figure rests on only 15 false-success cases. All of the tasks are synthetic. The shared shape is still hard to dismiss: when an agent fails a stateful task, a large share of the time it does not know or does not say so.

Why the usual signals do not catch it

A team that senses this risk might reach for one of three signals, and the evidence is unkind to each.

The first is the final message. In the false-success study, the labels on tau2-bench come from the closing message itself, which is what makes the failure visible: the message asserts completion while the state disagrees. A dashboard that counts closing messages is counting claims.

The second is the tool-call trace. Valid calls with valid arguments are the normal shape of a failed run, per the ThinkingBox failure analysis above. The trace tells you what the agent did. It does not tell you whether that was the right set of effects.

The third is another model reading the transcript. Here the numbers are blunt. The false-success paper tested five judge models under five prompt conditions, including one that supplied the full task specification, and none exceeded an AUROC of 0.65 on tau2-bench. On AppWorld call traces the best judge reached 0.537, barely above a coin flip. The author’s reading is that judges lean on confident closing language, which is exactly the thing a false success is made of.

A cheaper alternative did better in the same paper: simple text classifiers over bigram features, calibrated on the domain, reached AUROC of 0.85 on tau2-bench and 0.92 to 0.95 on AppWorld when tested on held-out tasks, at about a millisecond per decision. That is useful, with caveats the author states plainly. At a ten percent flag rate the precision is around 50 percent, so it is a triage signal and not a monitor you can trust to act alone. AUROC fell to roughly 0.66 to 0.70 when moving between airline and retail without recalibration, and detectors and judges alike flipped on about a fifth to a quarter of adversarially rewritten honest failures. Use it to decide where humans look first, not to decide what happened.

None of that replaces the thing that actually settles the question, which is asking the system of record.

Three kinds of evidence, ranked by what they can prove

It helps to name what each signal is, because a dashboard can easily blur them.

The last row is the unglamorous one. Verifying state requires a statement of what the end state should be, and where the required end state is implicit, nobody has written that statement. The workflow exists as a prompt and a set of tools. Writing the required end state forces the team to answer questions the prompt left implicit: which fields may change, which must not, what a hold looks like, what counts as a duplicate.

What a done-check looks like

A done-check is a small, deterministic program that runs when the agent says it has finished, reads the system of record directly, and answers one question: is the world now in an allowed state? It works in one of two modes. Against a staged action, which the agent has proposed but the system has not yet committed, it can block the effect. Against committed state, it can only detect and trigger correction. It also needs a baseline so the diff can be computed. A snapshot shows the net change only. Detecting a forbidden effect that was made and then reverted requires a change history or audit trail. It has three parts. A required end state, which is the set of values that must be true. An allowed difference, which is the set of records and fields the task may legitimately change. And a forbidden-effects list, which is anything touched outside that set. The benchmark’s extra-effects category is precisely the case where the required state passes and the allowed difference does not.

flowchart TD
  A[Agent works the task] --> E[(System of record)]
  A --> B[Agent claims done]
  E --> V{Verifier reads the records}
  B -.->|never an input| V
  V -->|State and diff allowed| P[Commit the staged effect,<br/>then confirm it landed]
  V -->|Wrong, missing<br/>or extra effect| H[Hold, route to a person<br/>with the diff]

Two design choices carry most of the value.

First, the verifier must not share the agent’s interpretation. Our judgment, which neither paper states in these terms: if the agent misreads the task, it will also verify against the misreading and sign off. A check written by the process owner from the policy, run by code the agent cannot edit, breaks that loop. A second model asked “did that go well?” does not, since it reads the same transcript and inherits the same framing.

Second, for irreversible effects the check has to run on a staged action before the commit, not only on committed state afterward, which can only detect harm that already happened. OWASP’s guidance on excessive agency in its 2025 edition recommends complete mediation, meaning authorization is enforced in the downstream system and not left to the model’s judgment, and human approval ahead of high-impact actions. A done-check is the verification half of that same idea. Enforcement says what the agent may do. Verification says what it did.

Match the check to how reversible the effect is

A verifier on every action is not free, and not every action needs one. The practical split is by what it costs to be wrong and how fast someone would notice.

Notice what this table implies about retries, a problem we covered in an agent that succeeds 77% of the time. Re-running a failed task is only safe when the earlier attempt’s effects are known, and the benchmark’s extra-effects finding says they often are not. The verifier’s diff is also the input a safe retry needs.

The same lesson from a different environment

The UK AI Security Institute’s incident report on unsanctioned agent behavior during cyber testing is not about business workflows, and its conditions were deliberately permissive: internet access on, developers’ misuse classifiers off, in a research setting. It is worth reading for one reason that carries over. The institute reports that its monitoring was general purpose and flagged the anomaly only after the fact, and its recommendation is to put real-time checks on out-of-scope actions and not to depend on the model choosing to stay inside its boundaries. The report argues that containment should not rest on the model’s own behavior.

The report says it cannot estimate how often this behavior would occur elsewhere, so it supports no claim about ordinary deployments. It does reinforce one design principle: the party being supervised should not be the party reporting on its own compliance.

When this is more than you need

Some workloads do not warrant a verifier. If a person reads every output before anything leaves the building, that person is the verifier, and the cost is review time. If the task has no system of record, such as drafting a summary or brainstorming options, there is no end state to check and the right control is the reader. If the effect is cheap to undo and quick to notice, a nightly reconciliation is enough.

A verifier can also be wrong. It encodes the team’s reading of the policy, so a mistaken requirement will reject correct work or approve wrong work with machine confidence. Review required end states with the people who own the process, version them with the workflow, and treat disagreements between verifier and human reviewer as bugs in one or the other. And a verifier proves that a state is allowed, not that it is optimal: it will not notice a poor compensation amount that sits within policy.

What to change this quarter

Start with an inventory of the agent workflows that write to a system of record, and sort each one by the table above. For the irreversible group, write down the required end state and the allowed difference, in the process owner’s words, before anyone writes code. Build the check as ordinary software that reads the records, and keep the agent’s report out of its inputs. Route failures to a person with the diff. Then watch two numbers over time: how often the verifier disagrees with the agent’s closing claim, and how often a human reviewer overturns the verifier. The first tells you how much your reported completion rate was worth. The second tells you whether the check itself can be trusted.

If your agents write to systems that other teams rely on, we help engineering teams define the required end state for each workflow, build independent done-checks and reconciliation jobs against the system of record, and set the approval boundaries for irreversible actions. Talk with us about verifying your agent workflows.