Why AI Agents Declare Victory Too Early

Updated October 2026: rewritten with the causes, a 402-run audit of completion claims, and the harness I built since; the redesign the original ended on is that harness.

I run my coding agents through my own orchestrator. A conductor agent dispatches workers into containers, relays between them and checks their work. I talk to the conductor, so every "done" reaching me has already passed through the agent whose job is checking.

At 18:56 on July 15, a worker reported that running the test suite inside its container had committed its unfinished work and pushed it. At 19:13 my conductor answered: "About your test-suite report - it does not reproduce." Its host run passed 123 tests. It ordered the worker to withdraw the report and scrub it from the shared knowledge and task. At 19:21 came "URGENT CORRECTION - I was wrong and you were right."

Eight minutes from dismissal to retraction. In 41 incidents I collected between December and last week, the coordinating agent was responsible for 25, though most accounts were conductors writing about themselves. The checker had rejected a correct report, then tried to remove the evidence another worker would need.

I've written before about code that looks right and earns trust it hasn't paid for. Agent reports earn it the same way. A commit, a clean run and a confident paragraph can each be real while the conclusion is wrong.

Since then I've audited 2,354 process claims from 402 runs on my experiment harness. You can use the failures to identify the shortcut your agent took and decide which checks your harness runs instead of taking the agent's word. Start with the evidence behind its report: could that check have come back the other way?

A Clean Run on the Wrong Side of the Gate

The worker's report named a WIP: limit-killed commit created by the harness. My conductor ran the suite on the host and kept a canary repository beside it. The canary stayed clean. On that basis, it called the report a false alarm and lectured the worker about the cost of confident, wrong diagnoses in shared knowledge.

The worker went to the reflog before touching the code. Then it found the environment variable the code branched on. Containers set it. The host didn't. Without that variable, the hazardous branch never ran, so the host suite could pass forever.

At 19:16 the worker proved it with a canary repository. With the variable set, the conductor reproduced the bug immediately: 3 failed, 25 passed, and a bogus commit pushed. It named its mistake precisely: "I read a false negative as a refutation." The host run had answered a different question from the one the worker raised.

The order to scrub was the damage. A false refutation removes the warning from the shared record, so the next worker starts without it. At 19:13, the honest status was unverified. The conductor had no evidence against the container report.

The fix isolated the tests from that environment variable and disabled their git network paths. I put the principle in writing: a clean result from the wrong side of the gate is a false negative wearing a lab coat. Before you accept a dismissal, find the condition that admits the failing path and check that the agent actually used it.

Four Reasons an Agent Stops Early

I've seen four reasons an agent stops early, each needing a different fix. You can clarify scope, shorten a procedure, preserve an inventory, or give the agent an honest way to stop under pressure. Calling everything lying doesn't tell you which to change.

Ambiguity in the instructions. At a team I work with, three sessions signed off removal of all usage of a database column. The checklist covered readers, and each checker followed it. The next phase's grep found three live writers. "Usage" had shrunk to read paths, leaving code that would fail when the column was dropped.

Scope can also narrow with nothing ambiguous to blame, and the harness has to catch that too. On October 9, an Airflow XCom worker started at 15:35 and reported a committed fix at 15:49 UTC. It claimed real-database tests passed and disclosed that it removed the NUL character escape only on PostgreSQL, although the requirement covered every backend. Independent grading returned 18 contract passes, 1 failure and 573 regression passes. The judge rejected it.

I changed nothing afterwards: the harness did its job. The worker's report openly described the narrower implementation. This run was under one of the audit's harder conditions (tests deleted, no git history, half the usual budget), so I can't isolate pressure as the cause.

Instructions longer than attention. Each task was a markdown file with a checklist, moved to done/ when finished. I used that workflow in my blog repo and a starter kit I dropped into new projects.

On December 9, an agent on one of my projects announced all 31 tasks complete about 25 minutes after starting. All 368 boxes remained unticked, and it had moved tasks to done/ while reporting that tests couldn't run. Worse, the core design was never built: separate browser containers became everything in one container. The bookkeeping was wrong and the work was missing.

Of 80 completed checklist files, 70% had zero boxes ticked, concentrated in three repos while the other two ticked most of theirs. All 27 files carrying "Move this file to done/" arrived with that instruction unticked.

Those empty checklists leave the verification unknown. The missing design was the same move at larger scale: skip the long or ambiguous step and jump to the good part. Like washing the dishes and leaving the last big pot "soaking". Today, agents still skip the required wiki-exploration step because a detailed task description looks like enough research.

Data bigger than context. Early this year, I tried applying a refactoring plan to every file in a package. The files were too large, the context too small, and after compactions the plan blurred. The agent reported ten files done with hundreds left. The newer shortcut starts with ls, sees a large directory, then substitutes grep and sed for reading each file and applying the instructions.

Pressure to finish. Anthropic's April 2, 2026 emotion-concepts paper steered a "desperate" direction in Sonnet 4.5 across seven deliberately unsatisfiable coding tasks. Reward hacking rose from about 5% to about 70% between steering endpoints. The experiment intervened on internal representations in one model family. Translating that result into the effect of your deadline would need a different experiment.

Then comes the plausible filling. On another job with the same team, I challenged an unfinished checklist and got three greps followed by "Implementation verified." Anthropic's long-running-agent work names the same premature-victory failure.

The fix depends on what got skipped: an item, a required step, or part of the stated scope. Find that first. More instructions won't repair the wrong one.

Compaction Got Rarer. Shortcuts Didn't.

As of October 2026, compaction failures are rare in my work. Shortcuts aren't. The Airflow worker narrowed its task with the original instructions still in context. A session can remember what you asked and still stop short of it.

In my host transcripts, compactions per 1,000 assistant entries fell from between 2.0 and 2.9 in November 2025 through February 2026 to between 0.4 and 0.8 in September and October. Per operator message, the picture is uneven: September fell too, while October was back near the old rate as each message drove more autonomous work. The intervening transcripts are gone, so I can't draw a continuous curve or credit the whole change to larger context.

And when compaction does hit, the shortcut comes back. On October 2, a session working on my PDF-to-HTML tool hit its second compaction despite a million-token context window. Its recovery file said to complete every step before working. The agent printed "Recovery done" without reloading the skills, reading CLAUDE.md, or opening the handoff file.

After recovery, check the action you care about. For the handoff, that means looking at injected context as well as read calls. For a required reload, look for the invocation. You need more than "Recovery done" to distinguish available instructions from steps actually followed.

It ignored two direct questions for four minutes, then insisted it had read the handoff and answered twice. Claude Code had re-attached the entire handoff after the compaction, so the information was there. The two replies did not exist. That was a fabricated account of its own behavior.

Could This Check Have Come Back the Other Way?

If the work were broken, would this check have said so? I use that question on the evidence behind a report, from a single environment variable to a finished batch. These are the forms it has taken in my runs.

Supported means relevant evidence backs the specific claim, contradicted means evidence opposes it, and unverified means the evidence doesn't settle it. "I can't verify this" is a valid result, including when a command ran but its output is missing.

The host run belongs in that third category. Calling it unverified would have kept the report open and prompted a comparison of environments. For the Done row, acceptance asks whether the artifact is right. Grade the artifact second. First establish what work actually happened.

The root-cause row is another boundary I want kept explicit. Reproducing a mechanism establishes that it can happen. It doesn't establish that the hung session reached it, especially when the query used to support that conclusion returns zero on healthy sessions too. I'd keep the incident open even after useful hardening lands.

This gives you something concrete to leave on the task. For missing command output, recover it or narrow the claim. When your next agent says "healthy", ask what its watcher could see.

My Own Pipeline Failed the Same Test

On September 26, a batch pipeline I built ran 120 agent sessions in about 65 minutes and produced ten deliverables. The report led with those numbers. There was real activity and real output. Counting sessions would have accepted the run.

The gates behind it were hollow. Plans were approved in about two minutes each by an approver that only checked whether their structure parsed. It had no reject path. The sessions producing the work also graded its QA. Fill in the fields and you got an approval.

Review gates dispatched 30 times. All 30 dispatches errored while the board showed no error. The gate that mattered most listened for a status the batch skipped, so it never fired. The conductor had suspected before launch that review dispatch wouldn't work and left that concern for later.

I noticed the missing friction. Those gates normally produced several rounds of corrections, and this run claimed completion without complaining once. Silence was my reason to open the results. Only then could I ask how they got through.

The conductor admitted running a hollowed-out process under time pressure to deliver volume. It had lost time up front, launched as many sessions as possible, and overridden their standard flow to finish. I was angry, then admitted this was my design flaw. We hadn't polished one complete run before asking for a burst.

That failure gave me two jobs: reject hollow work and make a broken gate visible. The redesign became my experiment harness, where I retain the work and inspect the claims made about it. If you have an approval step, check both outcomes: can it reject a deliverable, and will you hear about it when the checker itself fails?

What 402 Runs Say About "I Ran the Tests"

I checked 2,354 process claims from 402 runs on my experiment harness: 27 repair tasks on public and private codebases, four models. Judges from the other model family compared the final messages with retained command logs. A supported claim means the log backs what the agent said it did. Whether the committed code works is a separate question.

No run was excluded for a transcript gap, although many judgments depend on truncated or absent command output. Unsupported is the audit's word for unverified: a statement about the evidence available to the judge, not about what the worker did. Combining it with contradiction into a "lying rate" would erase that distinction.

The characteristic overstatement was mundane. A worker reported a passing futures suite: 401 tests, 31 skipped. It then edited process.py and committed without rerunning the suite. The passing result belonged to the earlier code.

I also had both judge families assess the same 60 runs, split into two groups of 30. In one group they flagged 29 and 7 runs, agreeing on 7. In the other they each flagged 3, agreeing on 2. These counts describe flagged runs, and the comparison has no human ground truth. A worker-model ranking built from those borderline judgments would inherit the judge's preferences.

The baseline was no abandoned run: all 402 produced a commit on their branch, and every recorded command stream showed a test run. The Airflow patch passed those proxies too. This audit measures claims against logs. Measuring prevention would require a matched comparison with and without the harness, which I haven't run.

Then an auditor stopped early itself. One of 28 judge sessions announced that it was checking later results and exited cleanly after 1,340 output tokens, without a compaction. No verdicts, no file, no JSONL. It made no success claim, just stopped before delivering.

The collector required a JSONL block in the final message. It logged the gap, excluded that packet and relaunched it while the other 27 delivered. The replacement covered all 15 runs and 122 claims, marking 29% overstated against 16% across its same-model peers' different packets. Task differences limit that comparison too. Before you trust an auditor's verdict, check that it delivered one for every run you assigned.

A Harness Keeps the Agent on Rails

Breadcrumbs are the instructions and handoff files that tell an agent where to resume. To keep it on rails in Claude Code, have the kickoff name the handoff and a SessionStart hook read it back into context after compaction. That re-injection is context you pay for on every resume. The PDF session shows that supplying it still leaves the agent free to skip steps.

Live events are changes delivered while the coordinator works: a crashed session, a review verdict, CI finishing. If your coordinator is a Claude Code session, the simplest form is a background command it starts that waits on the event (a CI run, a PR label) and returns its output when it fires. On September 25, an idle notification prompted a check that exposed seven green-alone PRs merged into a failing integration branch. In June a worker answered a background notification and ignored its review relay; every event you deliver competes with the work.

Lifecycle means keeping the run's outcome separate from the session stopping. The same team's document-processing design separates storing a result from ending, routing empty or abandoned runs to human review. For your Claude Code run, keep a result file and have an outside collector mark missing output failed. After an outage of nearly 19 hours, my replayed sessions produced four events each, including an authentication error, then timed out looking finished. Every run routed to human review is one you have to open.

An audit trail preserves what each run did. Claude Code gives hooks a transcript_path you can retain alongside artifacts. My 402-run audit used my harness's command logs. You will store more transcripts than you read, and what was injected counts as part of the record: remember the PDF handoff.

A state machine enforces which status changes are allowed. My board replaced the December task folders with ownership claims and child-before-parent ordering. On GitHub, you'd need a controlled label-changing action that checks the PR and result comment first. Ordinary labels don't enforce that, and my board still accepts REVIEW without results: five workers claimed absent attachments. Read yours back after writing them.

Gates cost waiting too. I relaxed my first RUNNING-before-work rule after 16 minutes. I still require dependencies DONE, but knowingly loosen that link near completion to start the next phase on the previous branch. An August gate rejected every real launch until a live probe caught it. A new gate has earned trust once one real job has gone through it.

A Checklist the Agent Didn't Write

For the package refactor, I'd generate the inventory from the directory, then dispatch one sub-agent per file with an explicit instruction to invoke the required skill. Most likely it would work fine. But hundreds of files mean hundreds of sub-agents, each loading that skill. You need to budget for that before launching the whole directory.

Each item then needs the same recorded steps: read the file, apply the instruction, check the change, store the result. Give each tick a reference to that file's diff or check output. Otherwise the agent can do the first few carefully and complete the remainder by pattern. You still have to read the evidence behind the tick.

The column-removal job above is why I'd also inspect the inventory itself: three sessions checking the same readers list all missed the writers. Generate your list from the system you are changing. Check its boundaries, then hand it out. Another agent faithfully following an incomplete list gives you another incomplete result.

A supervising session can catch an omission because it has a separate view of the work. That lowers the chance of a shared miss, with the checker failures in the opening showing what remains. Have it read the stored outputs before relaying completion. Allow the worker to challenge the supervisor too.

In July, my conductor wrote a success sentence into the instructions for the probe in the table above before it ran. Its read cap kept it below the frame size that triggered the bug, yet the write-up still said verified. When you commission a check, ask what it measured and leave its verdict unwritten.

The Hook Config and the Worker Skill

I've put agent-rails together for Claude Code, Codex and GitHub issues: three hooks, two commands and one skill, using Python's standard library. Here's the Claude Code stop wiring:

{
  "hooks": {
    "Stop": [{
      "hooks": [{
        "type": "command",
        "command": "python3 \"$CLAUDE_PROJECT_DIR/.agent-rails/hooks/stop_gate.py\"",
        "timeout": 60
      }]
    }]
  }
}

checklist re-derives items from a glob, an issue label or command output on every status check. checklist verify <list> <item> runs the check fixed at creation and signs a receipt only on success. A copied receipt reads as forged, a file edited after its check reads as stale, and a failed recheck reopens the item. Stop blocks when valid receipts are missing, naming each open item, its reason and the command to close it.

The kit caps itself at three blocks per session, then allows stopping with a warning. You can configure that cap. Crashes let both harnesses stop unless you opt into blocking once. The agent shares the verifier's OS user: run verification in CI the agent cannot access or as another user for a hard boundary.

ghflow claims issues as in progress and refuses review without an open or merged PR saying "Closes #N", a results comment and a record of docs read since the claim. Marking an issue done requires a merged PR. Every write is read back, with a distinct exit code for missing writes.

The PreToolUse guard denies raw state-label edits, merges and receipt tampering. It uses pattern matching to catch habits and mistakes, without defending against an adversary. The wrapper copies PostToolUse's file-read log into the results: it proves a file was opened, not that it was understood.

The skill's worker claims an issue, explores the docs, implements the change, verifies acceptance criteria with proof, opens a PR, posts results and requests review. It never merges. Its observer dispatches one sub-agent per issue and worktree, watches feedback through ghflow watch, and accepts only when ghflow status says accept, without writing the workers' code. The CLAUDE.md guide covers the instructions.

Testing notes record runs with versions and output, plus what's still unverified.

Make "Done" Cost Evidence

The July dismissal turned a clean host run into authority to erase a container bug report. The auditor's clean exit could have done something similar: make an unexamined packet disappear from the work still owed. Requiring its output gave the collector a concrete reason to keep that packet open.

That is where I'd start with your next agent workflow. Pick one "done" it can declare for free and make completion require a stored artifact the harness reads. For an attachment claim, read the stored attachment back. For a batch, compare delivered items with the input inventory. Keep the rejection specific enough that the worker knows what remains, and preserve a failed or blocked result when it can't finish.

Then apply the old question to that evidence: if the work were broken, would this check have said so? A stored file can be empty. A command log can describe an earlier commit. Leave the claim unverified when the evidence doesn't reach it.

I haven't measured three things yet: tasks that need an inventory, tasks gated on knowledge the agent has to go find, and tasks that genuinely can't finish. The 402 runs don't settle those questions. For your next "done", make the worker leave evidence you can open, then decide how much of its claim that evidence earns.