Debugging With AI: The Agent Digs, I Steer

A teammate pinged me on Slack about a production problem, and I didn't start digging myself. I handed the ping to an agent with three pointers: search GitHub, because it has to be something merged recently. It might be our auth provider. And here's the control I think is misbehaving.

The agent fanned out in every direction at once and came back almost immediately: it's not the auth provider, it's us. A recent PR had flipped one variable in a colleague's auth cache gate. The agent reverted it, and production was fixed about ten minutes after the ping.

In January I wrote that debugging took the same time as always, sometimes longer with AI in the loop. Back then I asked questions, the model suggested what to check and sometimes wrote the SQL, and I pasted it into a shell, fed the results back and tried the next hypothesis. It was satisfying. By today's standard it was terribly slow: hours per issue.

Debugging with AI is now the best working experience I've had, and the speed is absurd. The agent does the digging. I do three things: the context it reads, the data pipes it can reach, and the steering when it gets confident about the wrong thing. The goal moved too, from "the right fix" to "the right fix now".

It didn't always look like this. In June, an agent and I spent days circling one outage with two confident stories, and a number settled it. Here's how I work now, and what June taught me about where the human still matters.

The Better the Context, the Sharper the Diagnosis

The accuracy of the diagnosis tracks the accuracy of the context. An agent with a stack trace guesses. An agent with the stack trace, the last three merged PRs and the Slack thread where a customer described the symptom usually doesn't have to.

So the first thing I run on any incident is a skill I wrote called /gather-communication-context. In one parallel batch it searches Slack, meeting transcripts from Fireflies, Gmail, GitHub issues and PRs, and the project wiki. It comes back with a timeline organized by topic, every line hyperlinked: who reported what, when, and what somebody already tried. The agent knows the symptom's history before it touches a log.

That pull regularly changes the shape of the problem. It finds more customers hitting the same thing, and more managers complaining about it in channels I'm not in. It finds the Fireflies recording where the team talked through the decisions that led to the current design. It answers two questions I'd otherwise spend an hour on: why is it like this, and what else is affected?

The investigation has rules too. Every claim the agent makes has to point at something: a log line, an event, a timing, a query result. When the evidence isn't there, the next step is instrumentation. Add the log line, time the operation, size the thing.

My three pointers in the revert worked the same way, one sentence each. "Merged recently" narrowed the search to recent PRs. "Maybe the auth provider" gave the agent a hypothesis to kill first. The hint about the control told it which diff to read closely.

The wiki half of that context lives in Takt, the task board plus wiki I built so agents can read and write project knowledge. Every incident note, decision and gotcha lands there. The next investigation starts from what the last one learned.

Then Claude Code or Codex fans the work out. Sub-agents pull the error codes, the bad timings, the time window, the specific Sentry errors, the Slack complaints and the recently merged PRs, all at once. They correlate on the fly, flag the areas that look wrong and iterate on hypotheses. The ten-minute revert was exactly this: parallel digging that ruled out the auth provider in the first pass and landed on a one-variable diff.

Connect Every Data Pipe to the Agent

Context tells the agent where to look. Pipes let it look. My agents read Sentry, Datadog, CloudWatch and the database themselves through CLIs. No screenshots.

Early July, in Maestro, the orchestrator I run my coding agents on, sessions started dying about 6 seconds after anyone attached a file. The bug report blamed mount paths.

One read of the runner log showed a 403 and no 307 redirect. An auth gate had refused the request. The file was never missing. The cause was a refactor labeled "no behavior change", merged 27 days earlier, that had dropped the runner token's access. We audited about 110 gates afterwards.

A report is somebody's story about the logs. The log line is the evidence. Maestro's diagnostic notes, which the agents read before any crash investigation, turn that into a ladder: is the container still there, then the error frame in the event history, then the runner log, then the container logs, and only then the status label. The top rule: read logs before theorizing. Exit code 137 means checking the reap lines and Docker's disk before anyone says "out of memory".

The same notes cover hung sessions. Never assume anything from a session showing RUNNING. Read the raw typed event history, not the text view, which hides the decisive frames. A trailing tool call with no result tells me where to look. Before I call the session hung, I check the process's elapsed time and state with ps, because a slow operation looks the same from the history alone.

Logs only help if the errors reach them intact. My global CLAUDE.md says no try/except by default, a rule I explained in my CLAUDE.md guide. A blanket except that logs only the error message and swallows the exception loses the traceback the agent needs. Without it, full traces flow to Sentry, and the agent reads them there.

Then permission. Giving an agent the error tracker, the logs, the database and the comms used to mean running it fully unattended, with every permission prompt switched off. As of September 2026, Claude Code's auto mode uses a classifier to review actions that would otherwise need permission. It can block risky actions, but it can also miss them. In my use it works almost perfectly. For me it removed most of the permission friction that made full data access impractical.

A Starting Path

You don't need my skills to start. Here's the smallest version I'd set up. Give the agent read-only credentials: for CloudWatch Logs, an AWS profile whose role has the CloudWatchLogsReadOnlyAccess managed policy. Then hand it one bounded query, a time window plus an error filter:

# Last hour of ERROR lines from one log group, read-only profile.
# Times are epoch milliseconds. `date -d` is GNU date.
aws logs filter-log-events \
  --profile incident-readonly \
  --log-group-name /aws/lambda/my-service \
  --start-time "$(date -d '1 hour ago' +%s)000" \
  --end-time "$(date +%s)000" \
  --filter-pattern "ERROR" \
  --max-items 200

Then ask for two things, in this order. First, a timeline of the incident with a citation on every line: a log line, an event or a PR. Second, the competing hypotheses, and the one measurement that would tell them apart. The Sentry CLI slots in the same way. The same CLIs work from Claude Code, Codex or any harness that can run a shell.

This setup has a price. Authentication for every source, the query rules, and keeping the skill and the glossary current all take time that isn't incident time. The parallel searches cost extra model calls on every run. So start smaller: the issue and PR history plus one log source. Widen to comms, more logs and parallel agents when a single query stops explaining the symptom, or when the incident clearly crosses teams.

Every Rule in My Debugging Skill Is a Mistake the Agent Made

In December 2025 I spent a long session on a client's background-job pipeline where tasks died partway through. I told the agent its theories were smart, and that now I wanted evidence. I gave it Datadog, the signal handlers and SQL on the production database.

It kept getting the tooling wrong, and I kept correcting it by hand. The time range was off because the log I'd pasted was in UTC+5. Attribute searches needed the @ prefix in Datadog's search syntax. Container ids follow an <id>-<int> pattern it didn't know. Then I had it run the log queries in sub-agents that return findings, not raw logs. The sub-agents found errors from the develop environment, decided they were the cause and celebrated early.

When I asked for a note on what we'd learned, it became a skill. env:production is mandatory in it and appears three times, flagged CRITICAL. Every sub-agent brief carries one exact query, an epoch time range and a limit. The agent walks the job lifecycle from published to received to started and finds the last log line before the failure. Then it must name a root cause from a closed set, each with an evidence signature:

That table isn't a general theory of failure. It's the three ways jobs died in that pipeline, and it stops the agent from inventing a fourth. Another line guards the opposite mistake: "no logs found" can mean retention expired, not that nothing happened.

By February I'd forgotten I wrote it. It was triggering on its own. In May I asked an agent to find evidence that documents were being ingested normally, and it reached for the skill without being told. Each correction I made by hand once now runs in every session.

Some lessons never made it in. Two things I told the agent in that December session are missing: search by task id without a service filter to see the publish event, and start at 7 days and narrow. My February method for chasing a deadlock never became a skill either. Search the error signature over 24 hours, pull the 10 minutes before each hit on that container, then compare what the other workers were doing. Every lesson I don't write down, I teach again.

The skill can't fix vocabulary. In that codebase the same thing has different names in different modules, and unrelated things share a name. The agent gets tangled every time. No skill rule covers it. A glossary in the knowledge base does, and writing that glossary is my job.

Two Confident Stories, One Measurement

June, in Maestro. Providers kept vanishing, runners kept dying, and sessions sat stuck in PENDING. We shipped two real fixes: one for duplicate provider rows per runner generation, one for auto-recovery that skipped re-advertising providers. Both bugs were real. The symptom stayed.

The runner log had a gap in it. The runner logs the start of its post-create step but never the end, so each death came after silence right after that step began. The agent read that gap as "the DB clone", with no completed timestamp to back it. I was almost sure it was the requirements install. Two readings of the same silence, and neither of us had a number.

Meanwhile I kept the platform alive by hand: fourteen manual supervisorctl restart runs of the runner over June 19 and 20, eight backend restarts on top. Each one papered over the problem. On June 19 the agent wrote up a "sharper root cause", then retracted it and decided my restarts were causing the churn.

What we had in June was enough to act on. The clone behind the gap was taking about two minutes, and the template it copied had grown to 9.8 GB, most of it two production datasets imported as-is. On June 20 we trimmed the template and the deaths stopped. My retro blamed a stale heartbeat: the slow clone starved it, and the backend gave up on the runner.

In September 2026 I went back to the incident with an agent connected to every log: runner, backend, supervisord. In minutes it put every clone next to every runner death. From June 11 to 18 the median clone took 63 seconds, and 3 of those 14 clones killed the runner. On June 19 and 20 clones took 122.9 to 130.8 seconds, and 15 of 15 killed it, each destroy within about a second of a slow clone finishing. Right before the gap, the log shows the dependency cache hit for requirements and the lockfile, done in about 3 milliseconds. That rules the install out.

It corrected my retro too. The runner ran the post-create step as a blocking subprocess.run inside its async dispatch handler, which froze its event loop: the log goes silent from 16:11:21 to 16:13:29 in one case. The frozen loop dropped the WebSocket. The backend waited out its 30-second WebSocket grace and destroyed a runner that was working fine. The heartbeat route would have needed at least 390 seconds.

And the agent's retraction was wrong as well: every manual restart resumed the same runner id. Every new id came from auto-recovery after a clone.

Every destroy wiped the runner's providers, which broke launches and PR reviews across all projects until someone restarted it. In-flight sessions were marked finished while their containers kept running untracked, and their databases leaked. The cleanup later turned up 35 near-identical 9.8 GB clones and took disk use from 688 GB to 69 GB.

The review also found the trigger in the very first bug report. On June 11 the first session to clone the new template took 68.6 seconds, the runner's WebSocket dropped the second the container started, and the backend destroyed the runner 30 seconds later. The report listed "~70s template-DB clone" under things that looked fine.

The agent's hunch pointed at the right component. Mine didn't. But neither of us had evidence until someone put the clone time next to the death time.

Steer With What's on the Wire

When an agent is confident, my most useful move is to put the raw bytes in front of it: the response body, the probe output, the log line.

In June I was dogfooding a new token-mint UI in Takt and found five bugs the tests and reviews had missed. On the fifth, a newly granted claim never showed up in the picker. The agent confidently blamed a stale PWA service-worker bundle, and a second agent was already building a client-side refetch to work around it. I pasted the actual request and response into the session. The server itself returned 7 claims, without the new one.

That one paste ended the frontend theory. The cause was a hardcoded claim list on the server used as a ceiling. I redirected the second agent mid-run, and it threw its refetch work away. The service-worker staleness was real. It just wasn't this bug, and it stayed an open thread.

In May it was a probe. The Maestro dashboard hung on "Connecting..." and the session was reaped at 52 seconds. The agent blamed Caddy for stripping the WebSocket upgrade and "confirmed" it with a curl that returned 200. But curl attempts HTTP/2 negotiation over TLS by default. To test the HTTP/1.1 Upgrade handshake we were investigating, the probe needed --http1.1 and the WebSocket handshake headers.

My counter-evidence was simpler: Takt runs behind the same Caddy with working WebSockets, so Caddy wasn't stripping upgrades across the board. The agent dropped Caddy, followed the session's database audit trail instead, and that trail exposed four real bugs.

Neither agent was stupid. Each had a plausible story and a check that seemed to confirm it. What I supplied was the thing on the wire: the actual response, the right probe, a working neighbor behind the same proxy. My rule: if the agent's evidence is its own reasoning, I ask for the response, the log line or the timing.

Stop the Bleeding, Then Fix the Trigger

Speed only helps if you know which kind of fix you're shipping. I keep two separate steps: stop the bleeding now, then fix whatever set it off.

The ten-minute revert restored production. Understanding why the variable flipped could wait.

In June the restarts were my first step. The real fix was trimming the template: drop the two datasets, keep the DDL backed up so the data could be re-imported. From 21:01 on June 20, clones took 1.2 to 1.5 seconds. Before the trim, runners lived 5 to 60 minutes each, 14 generations in about a day. The runner created after the trim lived 480.9 hours.

Trimming removed the trigger. It didn't remove the blocking call. At the September 27, 2026 re-analysis, the post-create step still ran as a blocking subprocess on the runner's event loop. The only protection then was a longer WebSocket grace, raised from 30 to 90 seconds. Between September 16 and 25 there were 27 slow post-create runs of 27 to 74 seconds, and none caused churn. It was a known, bounded risk. The shape of the proper fix looks like this:

import asyncio
import subprocess


async def post_create_blocking(cmd: list[str]) -> int:
    return subprocess.run(cmd).returncode


async def post_create(cmd: list[str]) -> int:
    proc = await asyncio.create_subprocess_exec(*cmd)
    return await proc.wait()


if __name__ == "__main__":
    print(asyncio.run(post_create(["sleep", "1"])))

The retro also said to watch template size as a health metric. As of that September re-analysis, nobody had built the monitor.

One more rule came from a later incident. A restart "fixed" a runner that refused dispatches, and it also swept away the evidence. I suspected a latch. The ticket blamed missing releases. Reading the source next to the leftovers settled it: 7 containers still sitting at Exited (0) and zero seat-release lines in an 18 MB log. The event subscription that should have noticed them had died silently.

The rule: capture first, restart second. For that runner the list would have been short: the incident time window, the runner and container ids, the current dispatch and seat state, and the runner and event logs for that window. If collecting it would noticeably delay recovery, restore service first and write down what evidence you gave up.

My Job Is the Context Now

In June I kept a runner alive by hand while two stories argued. In September an agent with every log settled the argument in minutes and corrected my own retro. I didn't find the correlation. I supplied the access, pointed it at the incident and asked what the logs actually showed.

That's the division of work I trust now. The agent reads faster than I ever will and holds fifteen clone timings next to fifteen deaths at once. I decide what it can reach, what it knows about the system, and when a story needs a number before anyone acts on it.

I wrote about agents that declare victory too early. That post is about checking "done"; trusting agent tests and reviews takes that question to accepting the fix. This one is about getting to the cause, and the work has the same shape: a knowledge base the agent can read, every data pipe connected, skills that remember its past mistakes and a glossary for the words it keeps tripping on. The model doesn't need my debugging instincts. It needs my context.

Next time an agent names a cause, or you do, ask for the number behind it. Time the operation, size the thing, paste the actual response.