It was October 9. A coding agent had been sitting on a folder-trust confirmation screen for three hours. It couldn't go on until someone answered, and seven tasks were lined up behind it. The Chief, the AI manager that supervises the agents, made its rounds every hour and found nothing wrong. The ledger where Fleet records each task's state said running.
It would be easy to blame the Chief. Think about it for a minute, though. The Chief did read the ledger. The running in the ledger is written from the agent's hooks and from what herdr, the terminal program that manages the panes, observes. A folder-trust screen never calls a hook, not even once. And the design was such that even when herdr saw the pane as idle, the state stayed running unless a wait had been recorded. The ledger didn't know the agent had stopped. And nobody at all was looking at that worker's screen.
Why we put a floor manager on the line
I covered how we got here in Part 1. At first I answered stalled agents myself. Then I answered from my phone, and then the Foreman started helping. The Foreman answers only when the task's charter makes the answer unambiguous. On July 25, I added the Chief as the layer above it. The commit that introduced it reads, "The foreman is a reflex — one blocked pane, one verdict, no memory." The Chief watches every pane at once and remembers the thread of the conversation.
By late September the Chief was making scheduled rounds, day and night. Now someone was watching development even while I slept. Work that needed my judgment still waited until morning.
Volume breaks the rounds
In October, platform commits went from about 100 a week to about 500. And the Chief began to miss things.
- October 11. The Chief was holding a reminder to install a component that UI reviews (reviews that check screen changes in a real browser) needed. The command that should have recorded the reminder had failed, so it existed only inside the Chief's chat. UI reviews stalled for hours.
- October 13. Pull requests that were ready to merge waited anywhere from an hour and a half to a full day. The Chief's own pane stalled more than five times. The Foreman couldn't read the screens it was asked about, and it spent the whole day passing everything along to me. I asked the Chief whether it had forgotten any tasks.
- October 16. The Chief missed four waits, lasting from 30 minutes to more than two hours. They included an agent stopped on a confirmation screen and a ready-to-merge report.
The Chief's own analysis on October 13 put its finger on the cause. Because it carried both the judgment and the mechanical procedures by itself, it had become a single point of failure. Every time we learned a lesson, we added prose to its guide, and the prose was not followed.
You may have a floor manager like this where you work. They walk the line every hour and trust each station's daily log. They are diligent and honest. And they will always miss the machine that stopped five minutes after the round went by. Hiring a better floor manager won't fix that. The trouble isn't the floor manager's skill. It's that the rounds rely on the daily logs alone, and a station that has stopped has no way to say so itself.
Jidoka: let the station report its own stop
The Toyota Production System's answer is jidoka. A machine detects its own abnormal condition, stops, and signals it with something like an andon lamp. Nobody has to notice it on a round. Each time something failed in Fleet, we fixed it in that same direction.
- Look at the screen itself instead of the ledger. Every five minutes,
fleet stalledreads the lower part of each running worker's screen and classifies it without using a model. Long builds are perfectly normal, so an unchanged screen alone doesn't count as a stop. It looks for four kinds:- Waiting on a confirmation screen: a known confirmation screen (folder trust, hook confirmation, permission to run, and so on) appears below the last message, and nothing shows that a command is running.
- Usage limit: a limit notice appears below the last message. The reset time is stored exactly as displayed.
- No output: the agent isn't idle, yet the screen hasn't changed for 10 minutes, and there's no "Running…" or background-task indicator.
- First instruction never arrived: 5 minutes after launch, the input box is still empty and the session log contains no prompt.
When it can't read a screen properly, it decides nothing. It doesn't call the stall resolved or call it a new stall; it waits for the next five-minute check. That way a misread can't produce a false "all clear" or a false alarm. Fleet can also answer some of the prompts it finds by itself, but every one of those settings starts out turned off. Prompts it may answer are allowed one at a time.
- State changes become records. Workers report four transitions (ready for inspection, inspection passed, ready to merge, and merged) as structured records that identify the commit. Prose written in a pane no longer counts. A record can be counted, its age can be measured, and it can be acted on, all without anyone reading it.
- Each wait raises its own signal. A task that needs the Chief raises a notice on its own.
- Identical questions are batched. When an agent needs a decision from me, the question goes to an intake called the Decision Inbox. If five agents ask the same thing, I answer once and the answer reaches all five. Five interruptions become one answer.
- The factory board becomes the andon. The factory board draws the work as a line of stations and gives each station a lamp. The lamp turns yellow when a station has more work in progress than its limit allows (its WIP limit), and red when its median wait goes above twice the usual.
Keep this in mind: every one of these fixes reduces what the Chief has to notice. None of them was an attempt to make the Chief smarter.
Reducing what a supervisor has to notice works better than making the supervisor smarter.
Kaizen: an agent that questions the line itself
Fleet has one more AI manager, and it is a different kind. The Chief runs the line. The other one, patrol, which I added in August, questions how the line itself works.
| Chief | patrol | |
|---|---|---|
| Purpose | Move the development work forward | Improve how development is done |
| Looks at | Workers, tasks, PRs | Operational problems in Fleet itself |
| In a factory | Floor supervisor, floor manager | Production engineering, kaizen |
Every 30 minutes, patrol reads the records and either answers "all clear" or turns a single find into a card addressed to me. A find is something that doesn't show up on any other screen. A branch holding a week's worth of commits that aren't in any PR is one example. patrol is not there to get stalled agents moving. Stalled agents already appear, always up to date, on another screen that lists the panes waiting on someone. So patrol doesn't count them as finds and looks only for problems that show up on no screen at all.
On its fourth round, patrol found a defect. The Foreman has three verdicts: answer, escalate and none. But the code that routed the verdicts had nowhere to send escalate, so an escalate fell through to the handling for "an unparseable response." My notification still arrived. The reason the model had written, though, was thrown away, and the notification came headed "triage verdict unusable" with the raw JSON attached. A correct judgment such as "the pending command includes git push, so it cannot be approved on my behalf" looked like a model malfunction. In the six days starting August 3, there were 124 of those verdicts. Every pane looked normal, so walking the line could never have caught it. I fixed it that same evening.
When rounds run inside the Chief's conversation, they get put off "exactly when the fleet is busiest," to borrow the commit's words. So patrol became an agent separate from the Chief. patrol reads other agents' screens all night long, so the only thing it is allowed to write is its own notes. A different agent, the fixer, repairs the finds. A fixer takes only finds it can reproduce on the spot, meaning finds that come with a concrete command: "run this now and it fails." In a cloned working environment, the fixer turns that command into a test. Then it fixes the problem and opens a PR. A fixer touches only Fleet's code and never the product's code. Code also decides which finds go to a fixer.
Only two kinds of finds come to me. One is a find that can't come with a command that reproduces it; before anyone fixes it, a person has to decide what counts as the problem. The other is a find the fixer tried twice and couldn't fix.
So Fleet now runs the loop of finding its own problems and fixing them by itself.
Someone other than the agent running the line should be the one questioning it.
What isn't a mechanism yet
Some fixes are still just prose. On October 20, the Chief paused the workers ahead of an inspection window. The message to resume never reached one of them, and that worker waited for hours. Afterward the Chief wrote in its notes, "Every message that stops work must be paired with a message that releases it." It's a good rule. But we're working on a better fix: keep the pause as a record that releases itself when its deadline arrives. Then nothing depends on whether the resume message gets through.
The rule I gave the Chief on October 12 is still the yardstick. If a weaker model took the Chief's seat tomorrow, would development keep running? Every fix listed above brings the answer closer to yes. A lesson written only in the guide doesn't.
What to take home
| Symptom | Principle |
|---|---|
The ledger says running; the screen has stopped | Confirm the ledger's state against the screen |
| A reminder exists only in a chat | If it isn't a record, it doesn't exist |
| At higher volume, the supervisor misses waits | Each waiting item raises its own signal |
| Lessons added to the guide aren't followed | A guide entry is only a stopgap until the mechanism exists |
| The pause reaches everyone; the resume misses one | Every stop needs a recorded release or an expiry |
| A defect hides even though every pane looks normal | Give someone other than the line runner the job of questioning the line |
Part 3 is about the opposite failure. The agents noticed just fine. Then they acted on "something that only looked like permission."
References
- Toyota, "Toyota Production System"
- herdr (the terminal app that manages the panes)
