Running Loose Without Flying Blind

Am I the only one finding myself running agents in YOLO mode more and more? As they continue to get more powerful it feels more and more natural to give them more rope.

It feels like the same pattern we all went down for code review. Initially, you reviewed everything. Then you’d skip the simple stuff like unit tests and boiler plate. After a while you switched to just the more important sections. And as the tooling got better and we have more and more agent driven code review tools to help surface issues, I find myself only reviewing the critical sections or the high level outputs instead.

Now that the models have improved and agents can do way more for much longer runs, it only makes sense to give them more permissions. Who wants to sit around hitting enter to hand check every tool call? Why block the next 20 minutes of progress working in the background because it got stuck asking if it could run a fancy bash command to search for something in a local file.

And as agents can do more, we naturally want to give them access to more. When it can reach my database, my logs, my metrics, my emails, chats, transcripts, and docs it has more context and can actually do more for me.

But then you have that moment where it all goes wrong. Or nearly does. I’ve had a few now. Once, my agent was digging into why a prod service was throwing so many errors, went a little paperclip-optimizer on me, and decided the cleanest fix was to just remove the service. No more service, no more errors. Another time one blew away a chunk of a local dev database mid-migration. None of mine have been truly catastrophic yet, but every near miss makes me rethink how much access and permission I keep handing these things.

I didn’t want to babysit every tool call. I just wanted to keep running it and still catch the scary stuff. So I started building some tooling around it. Initially I just wanted better visibility for auditing. I started aggregating session logs across my different agent harnesses (Claude Code, Codex, Pi, OpenCode, etc). I added hooks to better log tool calls (especially potentially risky ones). Then I added an egress proxy and routed all network calls through it so I could log specific outbound LLM calls, MCP calls, third-party API calls and see exactly what my agents were doing – even when I wasn’t watching. And then I could have a clean agent post process and surface anything weird or scary.

Obviously that only catches things after the fact. The next logical step was control. So I started adding some smarts to the proxy so it could reason about each outbound request in context. Not just match a domain or IP against an allow/deny list, but make a real decision about whether to let it through. Improving my chances of catching the weird edge case before it happens instead of reading about it in the logs later.

How do you handle this? Do you find yourself wanting to give your agents more and more access and control too? How do you review and secure your setup? Or do you even care to?