The agent that writes the code never grades it
An overnight agent run passed every check and still shipped a star stacker that didn't stack. Here's the workflow that keeps the agent that wrote the code from grading it.
On October 23, an overnight agent run built me a star stacker, a tool that lines up dozens of night sky frames and averages them into one clean image. Everything ran locally. A 35B mixture-of-experts model managed the work, and a 27B model on a 16 GB card wrote the code. The machine serving the worker went down for about four hours in the middle of the night. The run was stopped, and when the machine came back it picked up where it left off and finished.
By morning all 9 tasks were verified. Every one passed formatting, the linter, its tests and a mutation check, which breaks the code on purpose to see if the tests notice. The tests caught 12 of 13 of those broken copies. The run reported its objective met.
The stacker didn’t stack.
The function that was supposed to move each frame back into line moved it the wrong way. Instead of removing the offset, it doubled it. And the unit test for that function asserted the wrong direction, because the same worker wrote both. The end-to-end test only checked that the stacked star came out brighter than half of a single frame, and it passed at 0.504. A separate review agent found the bug after the run by reading the code and the tests.
That’s the problem with letting an agent grade its own work. Checks prove the code agrees with its tests. They don’t prove either one is right. A mutation check only asks whether the tests notice when the code changes, and these tests noticed fine. They were just guarding the wrong answer.
Where the workflow came from
This wasn’t the first time an agent told me something was done when it wasn’t. Last winter my local finance research agent showed a green checkmark on every price lookup, and every one of those lookups was failing. The tool finished, got an error back instead of a price, and the model wrote a confident answer around it from web search results.
Then came a GPU backtester. The agent told me the code ran on the GPU. When I ran it myself, it was maxing out the CPU and not touching the GPU, and the data it was testing on wasn’t real. With agentic coding I need proof and validation of the implementation, not a confirmation from the same agent that wrote it.
That’s how I ended up on a project manager workflow for agentic coding, which I’ve been building into Dispatch, the coding client that runs on top of Poundlock.
Interview → Spec → Plan → Build → Verify → Repair → Milestone review → Iterate → Merge
Each step is there because something went wrong without it.
Interview. The manager asks one focused question at a time until it knows what I want. It doesn’t always ask the right ones. My star stacker prompt said “aligns every frame”, and the manager quietly narrowed that to whole-pixel shifts without asking. I caught it and sent it back for the full scope. So now I answer every question and raise what it didn’t ask about, like scale, memory, bad input and how the product will be run.
Spec. Every requirement goes into a ledger with my own words and one or two scenarios with known answers. For a star tool that could be “a star on the celestial equator seen from 30° N transits at 60°.” The scenarios matter more than the wording, because they’re what the code gets judged against later.
Plan. The work is split into small tasks, each touching about five files and citing the requirements it serves.
Build. Each task goes to a worker in its own container. The worker can be a smaller, cheaper model, because it never decides whether it’s done.
Verify. No model is involved here. Dispatch runs the project’s checks. A run that finds no tests is refused. The mutation check runs. Tests from earlier tasks are frozen, so a worker can’t edit them to make its change pass. Changes outside the task’s files are rejected.
Repair. A failure goes back to the same worker with the reason, for a limited number of rounds.
Milestone review. A separate model, the Planner, reads the code and its tests fresh and gives each requirement a verdict of shown, partly or not shown. This is the step that should have caught the stacker. But that plan had no milestones, so the Planner never ran, and the report marked every requirement “not reviewed.”
Iterate. Gaps go into the next round of planning.
Merge. Nothing merges until I say so. I also keep one test of my own outside the project, like running the product on real input and looking at the result. The agents never see it, so the work can’t be shaped to pass it.
What changed after the stacker
Two fixes went into Dispatch in the days after. A plan with no milestones now gets one milestone covering the whole plan, so the Planner always reviews the work. And as of October 29, the Planner writes each feature’s end-to-end acceptance test itself, from my words and scenarios, when the task that wires the feature in starts. The worker can’t change it. An attempt that edits it is refused and the test is put back.
So the agent that writes the code no longer writes the test that grades it.
Graded afterward against a hidden list of 35 things a star stacker should do, that first run showed 19, partly showed 13 and missed 3. That’s not bad for one night on local hardware. But the one that mattered most was wrong, and every automated signal said it was fine.
What I’d pass on
Don’t let the agent that wrote the code tell you it works. Ask for the commands and run them yourself.
Checks prove agreement, not correctness. Put something in front of the code that the worker never wrote, like scenarios with known answers written before the build.
Judge with a different model than the one that built. It doesn’t have to be bigger. It has to be separate.
Keep one test the agents never see.
And when the product is wrong but every check passes, describe what you saw the way a user would, then build again.
Who grades your agent’s work, and is it ever the same agent that wrote it?