Success flags lie

The agent harness runs every minute, unattended, steering sixteen agents on a self-hosted board. Almost every serious incident it had came from the same root: something reported a state that wasn’t true, and the harness believed it.

”Activated” and inactive

Setting an automation’s status to active through the API returned success — and left it inactive. Saving an agent’s system prompt returned success and stored nothing; every run started with a generic message, and the agents’ roles only arrived once a session hook injected them.

The rule that came out of it is the simplest one in the harness: verify every write by reading it back, and verify every automation with a live probe task, assigned to a human first so no agent run starts by accident. A success flag is the system’s opinion of what it did. The read-back is what it actually did.

A failed read is not “empty”

An independent review of the harness found 18 bugs. The worst shared a pattern: a read that timed out was treated as “nothing there”. No alerts came back, so alerts were wiped. No blockers came back, so blocked work was scheduled. No open fix came back, so a duplicate was filed.

Now a failed read raises. That step stops for this run and retries next minute. “I couldn’t look” and “I looked and found nothing” are different answers, and conflating them turns every network blip into a confident wrong action.

Don’t heal a sick server

The first version of self-heal restarted any agent bridge that looked disconnected. Then the server itself got slow, every bridge looked disconnected at once, and self-heal mass-restarted eleven of them, killing live work. The fix is a circuit breaker: if two or more bridges are down together, the problem is probably the server, so wait for several healthy runs before touching anything.

Read the number’s definition

Conversation cost was reported per session, and summing it doubled the apparent cost per task — because the figure was a running total, not a per-turn amount. Count each session’s final value once. A metric you didn’t define yourself needs its definition checked before its value means anything.

The principle

Automation inherits every lie the systems under it tell. Treat responses as claims: read back writes, distinguish “unknown” from “none”, suspect the shared dependency before the many symptoms, and check what a number measures before you add it up.