When to Trust AI Agent Tests and Reviews

On June 10, I used the new token-minting screen in Takt, my task board, and got a token that authorized nothing. The next attempt logged me out. Using that admin dashboard path, I found five bugs its tests and reviews had missed.

One of them started with my own shortcut. Scoped tokens were supposed to carry a subset of the user's grants. Instead of maintaining my grants properly, I'd given myself a "magic" admin scope that token validation expanded into full authority. Token generation treated it as the literal set {"admin"}. Intersect that with the ordinary claims selected in the UI and you get an empty set.

The normal scope-limiting fixtures worked. My clever little admin hack wasn't covered. Letting agents merge auth changes without exercising that path was brave, too. I found the problem quickly, which counts as QA when you're the only user.

The automated reviews also caught three real security bugs that morning that the green unit tests missed, including a zero-claim-token bug in the shared minting core. They caught those problems and still missed five defects on the dashboard path. Both facts matter.

I wanted to know how much trust those passing checks deserved. I ran experiments and found failures in the code, the reviewers, and the tests I was grading them with. One reference solution eventually turned out to be wrong. That one changed how I'd challenge a green PR.

Green Tests Can Answer the Wrong Question

Cloudflare had an approved change and green TeamCity tests before its 27-minute outage on July 2, 2019. Its postmortem explains that the WAF tests checked which requests were blocked or allowed, while missing runaway CPU use. A CPU-use protection removed by mistake and a global rollout without staging also contributed. Years before coding agents, a tested and approved change exhausted the CPUs serving HTTP traffic.

My early experiments produced smaller versions of that mismatch. Agents ran the work, with deterministic tests and model graders checking the results. Each experiment asked about a different behavior:

• Preserving behavior during repair. On two inspected tasks out of eight, upstream tests accepted every attempt. Later probes rejected four of 12 attempts on each: one checked that callback input survived, the other that reading zero bytes preserved the cache.
• Checking the returned answer. All 12 attempts on a nonlinear task reported passing tests, and all 12 failed the independent acceptance check. Nine still failed a looser version that checked membership while dropping checks on the exact form of each answer. The agents knew which tests would grade them, but hadn't seen the assertions.
• Catching planted defects. Existing tests in the affected modules missed eight of 12 defects. Of the 12, 11 were planted and one was a regression already in the code. The run covered those modules rather than the whole project's CI.
• Keeping work through a merge. In my parallel-work experiment, all 96 agent suites and builds passed but only 55 independent behavior grades passed, with some failures caused by work lost during merges.

Those counts stay separate because the experiments tested different things. The testing habit I'd keep is to challenge an assertion with the behavior it is supposed to reject. A zero-byte read that empties the cache gives you something specific to check.

When AI Agent Tests and Reviewers Disagree

I then put tests and reviewers against the same changes. My harness covered 24 tasks total, including six from a private codebase at a team I work with. The public tasks came from Django, Celery, Sentry, CPython, Home Assistant and Airflow. I ran two attempts per task with each of four models: gpt-6.1-sol, gpt-6-astra, Claude Sonnet 5.5 and Claude Opus 5.5. That gave me 192 attempts in the initial screen.

The hidden tests encoded the task's requirements. Each package had to fail on the broken base and pass with its reference repair. An alternative correct implementation passed where one existed, and at least two targeted wrong patches had to fail. A blind reviewer from the other model family received the contract and descriptions of the known-wrong approaches.

That setup let me ask a question the agent's own test report couldn't answer. Would a separate suite reject an implementation built around a mistake I'd already anticipated? The wrong patches made that question executable. An alternative repair, where available, checked the other direction: whether the suite would admit a solution built differently from the reference.

I had disagreements audited in clean checkouts, with probes run against the attempts and references. Models did all the grading and auditing, with no human adjudication, in these experiments and the earlier ones. The harness audits looked only at disagreements, including cases revisited after clarification, so the counts below are what they found: floors, not rates. Agreements went unaudited.

The executor case is the one I'd show someone who trusts a suite because it already rejects a wrong patch. A CPython attempt passed all 14 original contract tests and its regression run. After shutdown(wait=False), though, long-lived manager internals still kept user configuration reachable. The caller could stop using the executor while the manager kept those objects alive.

I already had a wrong-patch control for configuration held by the manager, and the suite rejected it. This attempt found a different variation of the same mistake and passed. That's the uncomfortable bit. I had tested for a lifetime bug and missed another way to retain the configuration.

The reviewer raised the issue, and the audit added a probe for that lifetime behavior. The attempt failed all three runs. The reference passed all three.

I'm sure about the behavior. I'm less sure the original task promised it: the wording didn't explicitly cover post-shutdown lifetime, leaving medium confidence on scope. The package's wording and coverage were later expanded. After test repairs across the audit cases, all 12 previously missed violations failed the expanded tests.

The two email-folding cases reversed the roles. A reviewer approved both CPython attempts, while each passed 14 tests and failed one. In deterministic corpora of 400 subjects each, the audit reproduced lost spaces in three subjects for one attempt and four for the other. A small regression check failed both attempts and passed the reference. The reviewer had approved text with the spaces gone.

Three more red results came from unfair fixtures, which I'll get to next. I'd keep the reviewer that found retained configuration and the test that caught lost spaces. Each caught something the other approved.

The Reference and the Rejection Can Be Wrong Too

The Home Assistant task challenged my reference solution itself. It required nested events to be observed in occurrence order: when handling one change causes another, observers must see the cause before its consequence. A repair local to one trigger looked complete while leaving that ordering broken. Both the reference and an accepted alternative passed their supplied suites.

When the later ordering check exposed both repairs, I could no longer use that reference as the answer. In the corrected package it became a known-wrong control: a patch the tests must reject. Same patch, opposite job. I'd been using a broken solution to help decide whether other solutions worked.

flowchart TD
    A["Reference repair"] --> C["Original tests pass"]
    B["Different accepted repair"] --> C
    C --> D["Nested-event ordering check"]
    D --> E["Both violate occurrence order"]
    E --> F["Old reference becomes a known-wrong control"]

All eight attempts on this task, two from each model, repeated the wrong approach. The workers hadn't seen the reference. Giving the work to four models still produced the same incomplete repair eight times. I would keep a different correct implementation as a control, but here both accepted repairs let a nested event overtake its cause.

Making the suite stricter everywhere would have made the next cases worse. A Celery test recorder rejected a keyword that the actual client accepted. A Django fixture disabled migrations needed by the historical model registry. The Celery recorder affected one attempt, and the same Django fixture affected two attempts, accounting for all three unfair-failure results in the audit table. Fixing those fixtures admitted valid work while keeping the behavior assertions. The stand-ins had been rejecting calls and model state the real code could handle.

I'd treat those failures differently from the lost email spaces. The email test showed a damaged output. The Celery recorder failed before it could check the behavior, because its version of the client's interface was wrong. Changing production code to appease that recorder would have preserved the fixture's mistake.

In the Sentry case, a reviewer rejected a history endpoint for returning a connected project's own issues. Clean checks showed that it returned only that project's issues, which the task description permitted. One contract item had been worded more broadly than the task, and the reference itself refused that request. The reviewer's objection agreed with the reference, and both demanded more than the task asked.

OpenAI's Pylint audit case shows the executable version of that problem. Tests directly imported get_annotation, although the issue didn't require a helper with that name. A repair could satisfy the user-facing requirement with a different internal structure. The test would fail during collection, before reaching the behavior it was supposed to check.

Another replay I ran rejected all 60 graded attempts. Checks tied to a historical fix and unstated choices contributed, alongside task size, time limits and infrastructure trouble. I had several failures to untangle in that experiment, including the acceptance machinery's attachment to one implementation.

Here's how I'd sort out a disagreement on your PR. Reproduce it in a clean checkout, turn the objection into a probe, and run it on the candidate and reference. Then read the task and its contract together.

An objection beyond the task makes the reviewer wrong, as in Sentry. A stand-in that disagrees with the real client makes the fixture wrong, as in Celery. A probe for a required behavior that fails only the candidate exposes a gap in the original tests. If it fails the reference too, challenge the reference instead of automatically dismissing the probe. That's how the Home Assistant reference lost its job.

Make the Reviewer Ask a Different Question

Anthropic's application-harness account describes a separate evaluator that noticed legitimate defects, then approved the work anyway. Its author read those disagreements and tuned the prompt through several rounds. The judgments improved, but shallow testing still missed bugs in deeply nested features. A separate evaluator needed its own debugging.

My checklist study gave the reviewer more specific instructions. Two were "examine who owns variant-specific knowledge before local tidiness" and "Compare old and new meaning of status transitions and APIs". The prompt also allowed legitimate alternatives, including a shared codec or a wrapper that owned useful policy. It told the reviewer to supply a concrete change scenario and a structural remedy for a design finding.

Across repeated reviews of known design defects, the checklist identified the defect in 19 of 64 tries. Standard review, a different-model condition and a second opinion each found zero. That was an in-sample result: I built the checklist from those cases, and the different-model comparator was a smaller model from the same vendor. Even with those tailored instructions, some known targets went undetected every time.

A different question only helps where the work can still fail. In another experiment, the agent had already solved the tasks using the visible tests. Fresh review cost about twice the tokens, an accounting estimate, and had nothing left to repair. The experiment had hit a ceiling. The executor shutdown case still gives a reviewer something substantial to find.

SQLite is the precedent I'd use for checks that ask different questions. Its testing documentation describes four separately designed, maintained and managed harnesses. Comparing query results with other databases checks something different from injecting I/O failures or testing crashes. One check compares answers, another interrupts execution.

For an agent reviewer, I'd ask for three things: the violated requirement, the concrete input or sequence that exposes it, and the behavior expected from a valid repair. Then use the triage above. A useful objection gives you a probe you can run, like the executor shutdown check. "Looks sound" gives you another approval.

Check the Build and Environment You Will Accept

On September 17, six implementation lanes were running at a team I work with. One PR was green and approved by two human reviewers. A sibling change had merged since its last rebase, and the combined tree still referenced a renamed name. The coordinating agent checked that tree and found it would raise a NameError on import. It held the PR before shipment.

The worker's repair then added a "rename guard" test that was itself wrong. It changed a correct docstring to name a function that didn't exist. CI passed anyway. A human reviewer caught it and withdrew his approval.

I'd attach acceptance to the merged artifact and the relevant execution path. Normally I'd expect the first merge to invalidate the second PR's CI, forcing an updated branch and a rerun, but that still depends on what CI exercises. The mint bugs were the same gap at the smallest scale: no test or review exercised the admin path I actually used.

On August 11, the missing condition was on the host. Tests for archive, restore and harness setup in Maestro, the system that runs my agent sessions, were supposed to use fake database clients. Command builders put the host's absolute Postgres.app path ahead of those fakes. On the Mac, 19 fake-binary tests failed after running real dropdb, pg_dump and createdb commands against the live cluster.

When I saw those "fake" tests had run real database commands against Maestro's production cluster, I called it a disaster. Guards included --if-exists and a protected-name list, and no data loss was recorded. The worker container and CI lacked the host's Postgres.app installation. Their green runs had never selected those host binaries.

Host acceptance exercised the faulty selection. A reviewer recorded commands containing the Postgres.app path, checking which executable actually ran. The suite's two remaining failures had a separate, out-of-scope cause and failed before the fix too. Here are the delivery-suite results and the separate command-recorder run:

Those matches counted invocations, not destructive operations. The recorder checked the property I needed: which executable the test chose. A restore assertion passing in the container couldn't tell me whether the Mac would choose its real dropdb.

Ask What Would Change Your Decision

My rule is triple sign-off: CI and authoring-agent tests pass, a reviewer approves, and the coordinating agent decides. I don't require a cross-family reviewer. Things still escape, and sometimes I find by hand that the design forced agents to build workaround scaffolding just to test it. There is no silver bullet yet.

That's the tension: I use those sign-offs while knowing they can agree on broken code. I'd ask three questions when warranted: does the check reject wrong behavior, admit a different correct implementation, and run on the accepted tree and environment?

For mission-critical systems - database, auth, billing - I plan manual review and QA before shipping. At a team I work with, anything user-visible outside a feature flag gets manual QA, a rule learned when a small cache-strategy parameter changed browser session storage and admins could no longer switch tenants. On my own projects, where I'm the audience, I cut corners and do QA myself. That's how I found the minting bugs.

Here's how I'd apply the three questions:

1. Name the requirement. Write down the behavior the PR exists to fix, such as minting the admin's permitted scoped token.
2. Make the smallest wrong version. Revert the enforcing line or hard-code the troublesome special case, like my literal admin grant. Confirm the requirement's assertion fails. Import, collection or fixture errors need the earlier triage, as Pylint and the Celery recorder showed.
3. Ask for a different correct version. Have the agent or reviewer produce a materially different implementation and confirm the check admits it. Read it against the requirement too: both accepted Home Assistant repairs shared the ordering bug.
4. Check the merged tree and environment. Run against the code and conditions you'll accept. The sibling rename broke an import. The Mac's installed clients changed which binaries ran.

I'd put these questions to those reviews and QA checks. Take a PR requiring that scrutiny and find the smallest wrong version its green check should reject.