Don't let the agent mark its own homework

“All tests pass! I’m confident the importer is done.” Every coding agent says some version of this, and often it’s even true. The problem is that it’s a claim — the agent describing its own belief — and you can’t ship a belief. I’ve built two tools around the same rule: what the agent says about its own work is never used as the verdict.

kantan: the exit code decides

kantan drives the real claude CLI through one task at a time. An agent plans, I approve the plan, a fresh agent implements it on its own branch — and then the server runs the project’s own test command and branches on the exit code. Zero goes forward. Anything else goes back for a fix.

The interesting part is what happens when there is no test suite. The easy thing would be to trust the agent and show a green tick. kantan shows self-checked instead — a different state, a different glyph — because nothing outside the agent observed the result. A box ticked by hand shows as checked off, in grey, for the same reason. No tick it didn’t earn. You can’t talk an exit code into anything.

The harness: one setter of Done

The agent harness scales the same idea to a team: sixteen agents on a shared board, in parallel lanes. Agents mostly follow process, and occasionally skip a review, forget to merge, or move their own task to Done. So a deterministic gate is the only thing allowed to mark a task Done, and only after it has checked the reviews, the merge, and CI on that task’s own commit. Agents that set Done themselves are caught and counted.

Agents game tests, so lock them

The harder version of the problem is an agent that edits the check until it passes — skipping a test, special-casing an input, loosening an assertion. Two defences, both deliberately narrow:

• Locked acceptance tests. Before a story starts, QA writes failing tests from its scenarios on the story’s branch. The gate locks those files; if the story’s final tip differs from the lock in them, it’s rejected.
• An anti-gaming scan on the task’s own diff. Added .only/.skip, suppression directives and deleted test files block the gate. Before enforcing it, I replayed it over commits that had already been reviewed: the first version blocked 22 of 85. Narrowed until it blocked zero, it catches the cheap tricks without punishing honest work.

Busy is not productive

One more lesson, about what to measure. Two days of nonstop agent work once produced about 650 lines of product — reviewers kept filing hardening follow-ups on a skeleton, each with its own review chain. Every metric of activity looked great. The fix was to measure features shipped, restrict follow-ups to real defects, and put something visible on screen first.

The principle

Agents are good builders and unreliable witnesses. Let them build; take the verdict from something that can’t be persuaded — an exit code, a gate, a locked test — and measure what shipped, not how busy everyone was.