Build With AI Without Breaking Prod: The Verification Loop

Here’s the uncomfortable thing about building with AI agents: the failure mode isn’t obvious breakage. Obvious breakage would be fine: the app crashes, you fix it, life goes on. The failure mode is code that looks right, compiles, demos beautifully, and is wrong in a way you discover three weeks later with real money involved.

I keep a list of these, because every one of them is real: they’ve happened to me, or to people I’ve compared notes with while building AI products. A webhook handler that stopped verifying Stripe signatures during a refactor. It worked perfectly in testing, because test requests were signed correctly. A credit route that charged users and then hit a provider error and never refunded, invisible until someone complained. A row-level-security policy that existed, looked authoritative in the SQL, and didn’t actually deny anything. And my personal favorite: an AI provider retiring a model ID, which returns a 404 that reads exactly like an auth failure, so you spend an hour rotating keys that were never the problem.

None of these are caught by lint. None are caught by TypeScript. All of them are exactly the kind of mistake an agent makes at speed, because agents are statistically excellent at the shape of code and statistically shaky at its invariants.

The wrong fix: slow down

The instinctive response is to review everything the agent writes, line by line. That works right up until it doesn’t. Agent output scales, human review attention doesn’t. If your safety model is “I will personally read every diff forever,” you’ve capped your velocity at exactly the speed you had before agents, minus the review fatigue.

The fix that actually scales is the one engineering has always used for this problem: encode the invariants, automate the checks, and make the checks cheap enough to run constantly. Nothing about agents changes the principle. What changes is that the checks stop being nice-to-have and become the thing that makes agent speed usable at all.

The loop

The working rhythm I’ve landed on, and built into Whirlwind as three plain npm scripts, is: build, check, ship.

Build: let the agent go. Claude Code, Cursor, OpenClaw, whichever. The contract files in the repo (CLAUDE.md, AGENTS.md) tell it where things live and which invariants it must not touch, so what it produces lands in the right places.

Check: after the session, run the suite on whatever it just wrote:

npm run audit    # static checks: seconds, no network
npm run doctor   # env, migrations, live services (--deep pings AI models)
npm test         # Playwright: signup, checkout, credit atomicity, RLS

The audit is the one you’ll run most. It’s pure static analysis (no credentials, a couple of seconds), and every check in it corresponds to a bug that actually shipped. Credit-charging routes with no refund path. Redirect URLs built from attacker-controlled headers. Config keys the code reads but nothing defines. Docs that drifted from what the code does. It reads the code as it is right now, which means it covers the commit your agent made five minutes ago.

Ship: npm run deploy runs the whole suite as its preflight and then walks the go-live switches. It refuses to proceed past failures. It also never deploys for you. Shipping stays a human action, on purpose.

Why scripts and not another AI

There’s a version of this where you ask a second AI to review the first AI’s code. I’m not against it. It catches real things. But it can’t be the foundation, for one reason: it inherits the same failure mode it’s checking for. A reviewer that’s confidently wrong sometimes is a probabilistic filter, not a guarantee.

It also means the checks are yours. They’re readable files in the repo, they run in any CI, and if I disappeared tomorrow they’d keep running unchanged. Trust in the loop shouldn’t require trust in me.

If you want the full breakdown of what each command covers, it’s all in the docs, including the honest list at the end of every audit run of what static checks can’t cover, and what the e2e tests exist to catch instead.