My Agents Try to Break My Apps

Reviewing everything our agents output has long been the bottleneck, so I’m always looking for new ways to let my agents improve the output quality before it gets to me. My latest version of this is having adversarial agents take the finished app or pending PR build and just go to town trying to find bugs and break the app. It’s actually been a great way to find weird edge cases, funky layout issues, and other bugs my normal review process might miss. Think of it as a QA team instead of another code reviewer.

I’ve been using it across a bunch of our products lately and it’s helped me knock out dozens of real bugs, optimize performance, automate long-tail platform testing, improve our onboarding flow, and fix a bug in some edge-case payment failure scenarios. One example of a bug this process caught was with Nibit. If you typed quickly right after a snippet trigger in a long Gmail message, letters could get dropped. On the phone it worked 1 time out of 6 during the bug hunt. After the hunters’ fix it worked 6 out of 6. These kinds of bugs would almost never be found in a regular code review or my usual testing. I schedule these hunts with Redline, so they run on subscription capacity I’d otherwise lose. Anything they find is basically free.

Here’s my approach:
I try to use hunters from two providers to check each other. I’ve found Sol and Opus sometimes find different kinds of issues, so it’s nice to get a mix from different providers when possible. I also make sure they:

1. Write a failing test before the fix to prevent future regressions
2. Stop after 3 review rounds and escalate whatever’s left to prevent review churn
3. Ask me clarifying questions before they start hunting. This has caught cases where they would’ve tested the wrong environment or the wrong part of the app, or not cleaned up after themselves.

It’s not all been perfect, though. During one Travally hunt, a background worker left running in a different worktree picked up my test data from the shared local database and spent real third-party API credits. Another agent thrashed against a Cloudflare Worker setup and got a huge, unexpected write spike. So now I’ve set up sandboxes in VMs to limit the scope of what the agents can access and I confirm the environment details ahead of time.

A new approach to long-running sessions

The bug hunts are also where I lean hardest on a new tool, because the hunters have a lot of questions. I built out an MCP integration I’m calling Async Decisions (not public yet, reply if you’d want it) that lets them ask me questions async and queue them up for me. I get a push notification on my phone and have a web and Android app that I can check in when I have a few moments. It gives me the context of the question, any relevant media attachments (screenshots, videos, test results, diffs) and a multi-choice question with a recommended answer and a free-form input in case none of the choices were right. I’ve had agents run through 20+ questions in a session.

Once the agent is properly instructed I can have it just spin out questions as it has them and keep working until it’s truly blocked. Then I can just answer the questions when I’m free instead of feeling like I need to answer right away to unblock the thread. It feels much more like working with a coworker instead.