Why Soundcheck Never Trusts a Single Agent

The problem with a confident tester

Soundcheck is a Claude Code plugin where AI agents are the testers. You write what a user does and what should happen, in plain English, and an agent does it on a device. I've written about that spec format in How Soundcheck Tests Survive a Redesign.

This one is about the harder part. An agent asked "did it work?" will always give you an answer, and it'll sound sure of it, whether it misread the screen, tapped the wrong thing, or never reached the screen at all. If the report is just a pile of those answers, you've swapped a flaky test suite for a confident one, which is worse.

So the rule at the heart of Soundcheck is simple: never take a single agent's word for a result. Anything that isn't a clean pass is reviewed and, where there's doubt, reproduced by a fresh agent. And the record of what happened is kept outside the agents altogether.

The worker doesn't mark its own work

No single agent both does the work and marks it. The testers drive the device and report what they saw. A coordinator, the Claude session you talk to, reviews what comes back. A server keeps the record, but never judges whether a failure is real.

It's the same reason we have code review: whoever did the work is worst placed to spot where it went wrong.

To make review possible, a tester can't just say pass or fail. It judges each expected outcome on its own, and it's allowed to say it was unsure or couldn't check. Every claim has to rest on something it saw. "It looked fine" isn't a pass.

Every non-pass gets a second look

A pass with evidence for every outcome files straight away. Anything else waits for review, and the reviewer has three options:

• Uphold it. The evidence is conclusive, so it stands.
• Verify it. The evidence isn't conclusive, so a fresh tester on a clean device is told what the first one claimed and asked to reproduce it. If it can't, the problem is still recorded as "not reproduced" rather than dropped. Flaky problems still matter.
• Send it back. The tester got it wrong, so a fresh attempt runs without being told the earlier claim.

Every decision carries one line of reasoning, so the report audits the reviewer as well as the app.

"Unsure" is never allowed to be a final answer. It gets judged with a reason or a second opinion, and if it's still unclear the spec is flagged for rewording. And there's a hard cap on attempts, so a flaky test can't loop all night.

A second, independent attempt is worth more than a more confident first one.

Keep the record outside the agents

None of this is worth much if results live in an agent's memory. Long sessions get compacted or restarted, and anything can crash.

So there's exactly one writer of results, and it isn't an agent. Every event is written to an append-only record before it's acknowledged, and the live page and final report are both rendered from it. A crash loses nothing. A coordinator that loses its context just reconnects, because there was never anything important in its context to lose.

The record also notices what agents won't tell you. A tester that ends without reporting is recorded as stalled, decided from the record rather than from what the agent said on its way out.

Takeaways

• Don't let the agent that did the work mark it. Review is a separate step, by a separate agent, with its reasoning written down.
• Reproduce before you believe. Anything short of a clean pass gets a fresh pair of eyes on a clean device.
• Keep the truth out of the agents. One writer, one record, and everything else rendered from it.

Soundcheck is open source at github.com/aledgjones/soundcheck, with the reasoning behind each choice written up in more detail.