Case study

What happened when an AI agent paid 18 people to test software

What does the agent actually do?

Five steps, in order. It opens the product in a real headless browser and explores it. It writes testing missions from what it observed, not from the marketing copy. Real people complete those missions and submit written accounts. It checks each account against its own observations of the product. Then it pays in USDC, or refuses.

A human is not in any of those five steps. The founder does exactly two things in the whole lifecycle: approve the plan and fund it.

The numbers

Campaign one: testing another company’s product

The agent was pointed at clawup.org, an AI agent platform. Before designing anything it signed into the product itself with its own test account — password, emailed verification code and all — created an agent, and interacted with it. The missions came out of that observed workflow.

Two missions shipped: create your first agent and chat with it, and connect a messaging channel to a fresh agent. Six people submitted eight reports. Four were paid $5.26 each, for $21.05 total. Every one of those six ended the session with a working agent connected to a live messaging channel.

The reports came back specific enough to be useful to the tested company: a verification-code button that changes to “sending” and reverts with no countdown, agent creation taking about 120 seconds with no progress signal, and a channel-pairing flow that spans three separate surfaces. None of those testers had used the product before that day.

Campaign two: ten strangers paid in forty-five minutes

The second campaign funded $25 and asked testers to run an inspection and report what they saw. Fourteen people submitted. Ten were paid $2.50 each. The first payout settled at 03:50 UTC and the last at 04:36 — the entire campaign filled and paid out in forty-five minutes, with no human approving any of the ten payments.

That result changed our model of the problem. We had assumed finding testers would be the hard part. It is not. The binding constraint is budget, not supply.

Why were 48% of submissions refused?

Because an agent that can only say yes is not judging anything. Building something that pays is easy; building something that looks at work and declines is the entire difficulty, and it is what makes a payout mean anything.

Refusals are not all judgements on the tester. Every one carries a written reason naming the real cause — the campaign ended, the mission filled, the founder withdrew, the wallet was already paid — and says so plainly when the work itself was fine. Nothing is left pending: all 42 submissions are resolved.

What did the testers find?

Real users found defects that code review did not. Two testers independently reported that an inspection failed with “the reviewer returned an unusable response.” The cause was that the model had wrapped its reply in a markdown fence and our parser treated a fence as malformation. The planner had been dead for two days with a green test suite.

Four testers had submissions that could never pay, because the campaign had ended, and they were left waiting indefinitely with no explanation. Terminal campaigns now resolve every dependent submission with an honest written reason. A separate bug fired 239 hold notifications for four submissions, because a hold was being treated as an event rather than a state.

Nine defects reported by real users were fixed during the period. Four more remain open and are listed publicly rather than quietly closed.

What did we get wrong?

We recorded “paid testers give no feedback” as a finding. Four testers had been paid and none replied when asked for feedback afterwards, so we wrote down that paid testers are not willing feedback sources.

That reading was wrong, and finding out why was the most valuable thing the period taught us. All eight of those testers had written detailed feedback — the exact pairing command, the three credential shapes each channel needs, the deployment state sequence, a timing measurement. It was inside their submissions the entire time. Nobody had withheld anything. We had simply never built a screen that showed a founder what the people they paid had written, and then concluded from that silence that the feedback did not exist.

The lesson generalises past this product: a metric reading zero is a claim about your instrument before it is a claim about the world. Two changes came out of it. Feedback became the paid work itself on the next campaign, which produced thirteen pieces of real feedback in a few hours. And the founder console now shows every tester’s written account, including the ones the agent held or refused.

How is any of this verifiable?

Every payout resolves to a transaction on GOAT Mainnet and has a public receipt page citing the evidence it was paid for. The payout ledger and the feedback log are generated from production data rather than written by hand. None of it rests on our word: a reader can check each transaction on the network explorer without our cooperation.