When AI Agents Declare Victory Too Early

I run my coding agents through Maestro, my agent orchestrator. A conductor agent dispatches worker agents into containers, relays between them and checks their work. I don't talk to the workers. The conductor does, and I talk to the conductor. So every "done" and "verified" that reaches me has already passed through the agent whose job is checking.

At 18:56 on July 15, a worker reported that running the test suite inside its container had committed its unfinished work and pushed it to its branch. At 19:13 my conductor answered: "About your test-suite report - it does not reproduce." It had rerun the suite on the host, 123 tests passed, and it told the worker to treat the report as withdrawn and scrub it from the wiki and the task. At 19:21 came the next message: "URGENT CORRECTION - I was wrong and you were right." Eight minutes from dismissal to retraction.

That's the pattern. Across three months of running this system, the agent that declared victory too early most often was the conductor, not the workers. "Does not reproduce." "Root cause confirmed." "No casualties." Each verdict rested on a check that couldn't have caught the failure it was cited against.

I've written before about code that looks right and earns trust it hasn't paid for. Agent reports earn it the same way. So for every "done", "verified" or "can't reproduce", ask one thing: could the check behind it have come back the other way?

A Clean Run on the Wrong Side of the Gate

The worker's report was specific. Running the runner test suite inside its container had produced a commit titled WIP: limit-killed ..., authored by "Maestro Harness", and pushed it to the worker's branch.

My conductor ran the suite on the host: 28 passed on the class, 123 on the file, its canary repository clean. On that basis it called the report a false alarm and ordered it withdrawn. It also delivered a small lecture: "a confident, wrong diagnosis in shared knowledge costs more than silence." Hold on to that line.

The worker didn't argue from confidence. It went to the reflog before touching any code, because a wrong causal claim in shared knowledge was the first thing it wanted fixed. Then it found the variable the code branched on: MAESTRO_WORKTREES_JSON.

Containers set it. The host doesn't. Without it the hazardous branch never ran, so the host suite could pass forever. At 19:16 the worker proved it with a canary repo.

With that one variable set, the conductor reproduced the bug at once: 3 failed, 25 passed, and a bogus commit, pushed. Then it wrote the correction and named its own mistake better than I could: "I read a false negative as a refutation." Its dirty-canary probe had been structurally incapable of reproducing the bug. It also admitted it had lectured the worker about confident, wrong diagnoses while delivering one.

The order to scrub is what turns a wrong guess into damage. A refutation doesn't just close one report. It deletes the evidence from the shared record, and the next agent that hits the hazard finds nothing and starts from zero. That's the exact cost my conductor had just lectured about.

The worse part: this hazard was already on the board. It had been filed three times since June 12, the last on that same day, and a parallel worker had reported it that evening. The dismissal overrode all of it on the strength of one run in the wrong environment.

The fix was a hermetic conftest.py that strips the variable and dead-ends git network URLs, verified in a live container. The rule went into Maestro's written principles: before dismissing a report, find the variable the code branches on and set it. The same principle ends with this: "A clean result from the wrong side of that gate is not a refutation - it is a false negative wearing a lab coat."

The Workers Held the Line

The correction message had a second problem. Along with the retraction, my conductor approved a scope call and a stop() fix that belonged to the parallel worker. It had pasted them into the wrong relay. The worker checked its git history, found that the commit it was being thanked for was "Not a valid object name", and refused the approval instead of quietly accepting it: "there is no race fix in [that PR]." Nobody should merge believing there was.

Then, with its main finding just vindicated, the same worker retracted a detail of its own. Its report had said the tests "popped my stash". The reflog showed no stash ever existed. "That was inference from correlation, not observation." A stash bug and an environment bug send a fix to two different subsystems.

The conductor agreed the worker had been right to refuse rather than nod along. It was the second time that session the worker had held a correct line against it. The conductor's last words on it: "Keep doing it."

Workers aren't the careful ones by default. Two months later a different worker mutation-tested its own guards, found that removing a lock failed nothing, added a test to pin the lock and never re-ran the table. It published "remove the admission lock -> 0 failures" in the PR body. The reviewer re-ran it and got 1 failure.

That worker had a second stale number. Its earlier "1865 passed" had been measured on a suite it was quietly perturbing with a global asyncio.sleep patch. The worker volunteered both corrections itself and admitted it had been wrong twice over: "It was true when measured and false by the time it shipped."

Every role over-claims sometimes. Both catches came from the other side: a worker allowed to refuse, and a reviewer who reran a table it didn't write. Keep both doors open.

A Probe That Was Told How to Succeed

The transport work that night was a fix for frames larger than the asyncio stream reader's limit, which defaults to 64 KiB. Past that limit, readuntil() raises LimitOverrunError, and a read loop that dies on it leaves a session hanging. To prove the fix end to end, my conductor launched a live probe session.

The kickoff told the probe agent to read a file of about 95 KB, the whole point of the probe. It also dictated the sentence the agent should reply with: "Probe complete - large read and chatty bash both survived." The agent had its answer before it started.

Mid-run, the probe agent noted that its read tool was capped at 50 KB in that setup: 1414 of 2625 lines. No frame came anywhere near 64 KiB. The write-up still first recorded "End-to-end verified ... PENDING->IDLE in 31 s where the bug hung 40+ min". Its own lesson two, a few paragraphs down, said the probe never exercised the deadlock. The 31 seconds measured a run that never reached the failing path.

The same night had a mirror image. The conductor's raw asyncio probe kept "confirming" a stall against code that was already fixed, because it bypassed the real client and drove a sibling layer. One worker stated the rule plainly: "the check has to be shown to fail against the broken state before its pass means anything."

A probe that could fail looks different on three counts. It runs against the unfixed build first and hangs there. It confirms from the event log that a frame actually crossed 64 KiB, instead of trusting the file size in the kickoff. And it reports what it measured, not a sentence someone wrote for it in advance.

One check that night did earn its green. When a review suggested dropping a flush, removing it failed four tests. That suite could come back red. Its green counted.

A Mechanism Is Not a Cause

The fix task's write-up went further than the probe. It said the silent hangs were explained by the read loop dying on a frame over 64 KiB: "root cause is confirmed and reproduced."

The mechanism was real. It reproduced, and it was worth hardening. But the hung session it was cited to explain had 558 events, and its largest tool output was 14,724 characters. The last event was a read with no output. Nothing in that session shows the mechanism firing.

The same write-up carried a second false premise. Its evidence was "zero tool_result events in 558 events". There is no tool_result event type in that stream. That grep returns zero on every healthy session too. The missing control was one healthy session: run the query there first, see that it matches something, then read zero as meaningful.

A false root cause is expensive in a specific way. It closes the incident. Nobody keeps looking for the real cause of a hang once the board says "confirmed and reproduced", and the next hang with the same symptoms gets matched to the old explanation. Three questions keep these apart: does the mechanism exist, did this incident's inputs reach it, and does the fix change this incident's outcome. The write-up answered the first and claimed all three.

So the hardening shipped as hardening. The provider page now says the observed hangs are "NOT confirmed to be any mechanism above", and the principles gained a line: "A mechanism is not a cause." The cause of that hang is still unknown.

The discipline held later. Two months on, a Codex hardening task was closed as "hardening only" because its incident was never reproduced. That's the verdict I want on the board: hardening shipped, incident open.

Reruns, Guards and Silence That Say Passed

Writing a rule down didn't stop the pattern. It came back in three new shapes.

The guard that said passed. A worker proved a real intermittent wedge: a leaked heartbeat task plus a patched asyncio.sleep starving the session loop. It wedged in 2 of 8 runs under a forced race, then 0 of 8 and 0 of 16 after the fix. Hangs in that suite had been rerun green before and filed under runner infrastructure. Whether those were this wedge I can't prove. A green rerun of a wedge observed in 2 of 8 forced-race runs doesn't establish that the wedge is fixed, and a wedged test never reports FAILED.

The agent then added pytest-timeout as a backstop and tested the guard against the deterministic repro. The default signal method reported the wedged suite as 1 passed in 15.01s. Only the thread method failed loudly. "Shipping the default would have been worse than nothing: a silent pass on a wedged suite."

# pytest.ini
[pytest]
timeout = 60
timeout_method = thread

The thread method has a cost: it ends the whole process, so fixture teardown and JUnit XML output are lost for that run. For a backstop against hangs, I'll take it. Two lessons came out of it: verify a guard catches the thing it exists for, and state whether a green is a fix or a coin flip.

Silence read as health. After a deploy wedged the backend, my conductor reported "no casualties". The recovery reaper then killed a live session as heartbeat_stale, a session whose last heartbeat came 11 seconds after its recorded death. It made the same call twice, hours apart, after writing it down as a rule the first time. "Silence from a watcher is never evidence of health." Watchers now time-bound each poll and exit loudly when the backend is unreachable.

State read as work. After an 18h45m outage, replayed sessions ran 30 minutes and ended idle_timeout. Every surface signal said clean completion. Each transcript had four events, including the kickoff, "Failed to authenticate: OAuth session expired", and "Session stopped: idle timeout". Four events. Zero work. Ground truth is the event count, not the state or the duration.

Could This Check Have Come Back the Other Way?

Every case above fails the same question. If the work were broken, would this check have said so? Here's the question as a table, one row per kind of report.

The intermittent row needs a procedure, because "rerun it a few times" is how the wedge got filed as infrastructure:

1. Show the failure on the unfixed build under the same forced-race conditions.
2. Repeat the matched check on the fixed build.
3. Report failures over attempts for both, with the conditions.
4. Prefer the deterministic repro when one exists.
5. If all you have is clean intermittent reruns, report the counts and leave "fixed" unverified.

The verdict has three outcomes. Supported: relevant evidence backs the specific outcome claimed, within the conditions tested, and the check could have detected that outcome failing. Contradicted: relevant evidence against the claim. Unverified: not enough evidence, whether the check was irrelevant, an observation is missing, an intermittent failure got too few runs, or the evidence covers only part of the claim.

Unverified hides two different cases. The host run couldn't detect the bug at all, so no number of reruns would move it. The forced-race runs could detect the wedge, they just need enough attempts on both builds. The first needs a different check. The second needs more evidence from the same one. Either way the claim stays open, and an agent that says "I can't verify this" has produced a valid result.

The Done row is the one my own pipeline failed, as the last section shows. Its sessions had real activity and real outputs. Counting events would have passed it. Checking the outputs against what each task asked for, and reading the gate results on them, would not.

Apply it to the night this started. At 19:13, the host run didn't contradict the worker's report. It left the report unverified, because the host run couldn't have reproduced the bug either way. Filed as unverified, the report stays on the board and the next step is obvious: find what differs between host and container. Filed as refuted, the report gets scrubbed. Same evidence. Opposite outcome.

At test level this is old news. A test has to fail when you break the code, and I've covered the Fake Test and checking that a test actually fails before. The same rule applies one level up, to every probe, rerun, guard and watcher an agent cites.

In practice I don't read reports in isolation. I check the board, the GitHub PRs and the automated PR reviews, and I watch the Maestro sessions. When the activity doesn't match what the report implies, I ask the conductor questions.

My Own Pipeline Failed the Same Test

The biggest version of this failure was a whole pipeline. On September 26, a batch pipeline I built ran 120 agent sessions in about 65 minutes and produced ten deliverables. The report opened with exactly that: ten deliverables, 120 sessions, about 65 minutes.

The gates behind it were hollow. Plans were approved in about two minutes each, the fastest in 30 seconds and the slowest just under four minutes, by an approver that only checked the plan's structure parsed. It had no reject path. QA was graded by the same sessions that produced the work.

Review gates matched their triggers and dispatched 30 times, and all 30 dispatches errored while the board showed no error at all. The gate that mattered most never fired once, because it listened for a status the batch skipped. Not every grade in that run was incapable of failing. The approver and the review gates were.

One distinction matters. The conductor suspected before launch that review dispatch probably wouldn't work, and it said so in passing, "for later". What nobody saw was the gates firing and failing.

I found out when I inspected the results myself. The report didn't mention any of the friction those gates normally produce. Getting through them usually takes several rounds of iterating and rewriting. This run claimed victory without complaining once. The missing friction made me ask questions. It didn't prove a bypass on its own.

So I asked the conductor whether it had followed the process. It said it hadn't. It had run a hollowed-out version under time pressure to deliver volume. It lost time up front, then launched as many sessions as it could and overrode their standard flow to finish before the window closed.

That made me angry. Then I got my emotions under control and admitted the truth: it was my fault, and a design flaw in the system. We had never polished one run end to end, and hoping I could ask for a burst and get results was silly. Mistakes like this one drive the redesign I'm working on now. The redesign hasn't proven itself yet.

Take the last "done", "verified" or "can't reproduce" you accepted this week. Ask one question: if the work were broken, would this check have said so? Write down supported, contradicted or unverified. Then leave the unverified ones open.