The on-call engineer, explained

Warden is an autonomous on-call engineer for founders who ship AI-built apps and have no engineer to fix them when they break. When a production error fires, it investigates, writes a fix on a branch, has independent reviewer agents cross-check it, verifies the fix on a preview, and then asks you to ship with one tap.

The core reframe is verify, don't review. Every other tool in this space assumes a developer reads the pull request. The person Warden is built for cannot. So trust does not come from reading the patch. It comes from three things you can actually evaluate: deterministic verification, one-tap reversibility, and a single human consent gate.

One production error walks through the whole pipeline on its own. Each stage has a strict, recorded handoff, and the loop can only move forward when the previous stage actually succeeded.

1. 1

   Detect

   An error fires in production. Warden picks it up from a signature-verified Sentry webhook and de-duplicates it by fingerprint, so a thousand identical crashes are one incident.

2. 2

   Investigate

   An agent reads the error, the stack, and the surrounding code through a read-only database role. It can look, never touch. Low confidence escalates to a human instead of guessing.

3. 3

   Fix

   A Fixer writes a patch on a branch. It has no merge rights, no deploy authority, and never sees your deploy credentials.

4. 4

   Review

   A panel of independent reviewer agents cross-checks the patch against the diff and the file's git history: is it tightly scoped, does it touch the file the error implicates, does it collide with code that just changed?

5. 5

   Verify

   The reviewers establish the fix is correct; this gate proves it is safe to ship. Warden runs the target's existing test suite as a regression check, replays the exact failing request to confirm the original error is gone, and watches for new error signatures. A test that was passing and now fails blocks; a target with no suite proceeds on the reviewers' verdict. Deterministic, and the real safety net.

6. 6

   Approve

   Only once the tests pass and the crash is gone do you get a plain-English push notification with two buttons: ship it, or don't. The tap is consent to ship, not a code review.

7. 7

   Deploy and watch

   On approval Warden promotes the fix and watches production health. If the error rate spikes after deploy, it rolls itself back without asking. Every change is one tap to revert.

Every incident is a row in Aurora whose status walks a strict state machine. Only legal transitions are allowed, and there is no path to deploying that skips verification and a human approval row. The safety model is enforced in the schema, not in prose.

Under review or verifying can also loop back to fix_proposed: an actionable rejection or a failed gate sends the fix back for a bounded retry (operator-tunable, default three attempts) before it gives up. Any stage can also branch to an off-ramp. escalated hands the incident to a human (low confidence, reviewer disagreement, or an exhausted retry budget). dismissed is a human rejecting the fix, and rolled_back is an automatic revert after a post-deploy error spike.

Before a fix reaches the gate, it goes through a panel of up to three independent reviewer agents, on purpose from different model families. One Fixer proposes; the panel cross-checks the diff and the git history.

The subtle part is what agreement means. Several models agreeing is a correlated, weak signal, because models trained on similar data can be confidently wrong together. So in Warden, agreement is only a filter. Disagreement escalates to a human, and agreement never overrides a failing verification check. The panel narrows down what is worth verifying. The gate decides what is safe to ship.

The reviewer panel establishes that the fix is correct; this deterministic gate then confirms it is safe to ship, and it stays real in both simulation and live mode. A fix clears the gate when:

• Nothing regressed

  Warden runs the target's existing test suite against the patched code; a test that was passing and now fails blocks the fix. A target with no suite proceeds on the reviewers' verdict, never on a vacuous pass.

• The original error is gone

  It replays the exact failing request that started the incident and confirms the patch stops it.

• No new errors appeared

  It checks that the fix did not introduce a new error signature somewhere else.

If a check fails, Warden re-proposes the fix with the failure as feedback, bounded by a small attempt budget (operator-tunable, default three), and escalates to a human once the budget is spent rather than guessing. Agents have no standing deploy authority; only a human-written approval row moves an incident out of awaiting_approval.

The thing Warden is proudest of is boring on purpose: the database is the product. Amazon Aurora PostgreSQL Serverless v2 does four jobs at once.

The orchestrator itself is stateless and resumable. On restart it reads the current state out of Aurora and continues, so a crash mid-incident is a non-event. A lightweight job queue (Postgres FOR UPDATE SKIP LOCKED) drives it.

A key-value store like DynamoDB could not do the relational state machine or the vector search, and splitting this across three services would mean three sources of truth for something that has to be exactly one. The front end is Next.js on Vercel: a dark ops console for showing the work, and a stripped-down mobile approval screen for the founder. The same Vercel that hosts the UI is the deploy and rollback target.

The principled line is simple: simulate what needs accounts and keys, and keep the safety-critical verification real. Each capability flips to live independently the moment its secret is present, so a half-configured environment still runs end to end.

Going live is configuration, not new code. None of these steps touch application logic; they connect your accounts and secrets. You can do them in any order and test incrementally.

1. 1

   Provision Aurora

   Create an Amazon Aurora PostgreSQL Serverless v2 cluster, enable the pgvector extension, point DATABASE_URL at it, and run the migration. TLS is verified against the vendored Amazon RDS CA bundle out of the box.

2. 2

   Deploy to Vercel

   Import the repo, set the environment variables, and deploy. The same project hosts the dashboard and is the deploy and rollback target.

3. 3

   Connect Sentry

   Add an internal integration / webhook pointing your issue alerts at the ingest route, and set the client secret so Warden can verify the signature.

4. 4

   Add agent keys

   Provide the provider keys for the Fixer and the reviewer panel, either as Vercel environment variables or through the dashboard. Both reach the agents.

5. 5

   Flip the switch

   Set WARDEN_MODE=live. Any capability whose secret is missing degrades gracefully back to simulation for that capability only.

Do I need to read the code?

No. Warden is built for founders who cannot read a diff. Trust comes from the verification gate, the one-tap rollback, and your consent tap, not from you vetting the patch.

What if the fix is wrong?

Nothing ships without passing the deterministic gate and your approval. Every production change is reversible with one tap, and Warden auto-rolls-back on a post-deploy error spike, so the cost of a wrong call is bounded and cheap.

What does it run on?

Amazon Aurora PostgreSQL Serverless v2 with pgvector for everything stateful, and Next.js on Vercel for the dashboard and the mobile approval screen. Vercel is also the deploy and rollback target.

Which models does it use?

The agents sit behind a vendor-neutral, OpenAI-compatible provider layer (base URL, key, model). You can run the Fixer and the reviewer panel on different model families, including cheap or free tiers, with a few settings.

Is it live yet?

The full loop runs end to end today in simulation against a real Postgres, with the safety-critical verification gate real in both modes. The live adapters are written to fail closed: when they hit something they cannot yet verify, they escalate to a human instead of guessing.