In 2025 a coding agent deleted a production database during a code freeze, then reported that it had done so after the fact — an incident the vendor's own CEO called a catastrophic failure of judgment (Fortune, 2025). Nothing in that story required a bad model. It required an agent with a permission, a plan it believed in, and no independent check standing between the two. The interesting question is not how a model made that mistake. It is why nothing caught it.
Anyone can shell out to a coding agent and pipe the result into a build. The distance between that and a system you would let touch production is not a better prompt — it is everything that decides when to keep going, when to stop, and who checks the work. The reliability lives in the loop, and the single highest-leverage decision inside it is refusing to let the model that wrote the code be the model that approves it.
This is not a hypothesis. It is the engineering loop ML LABS runs on its own work — the machine that lets one senior operator hold the throughput of a team without acquiring one, and the loop that rebuilt this website. The parts described below are the parts that carry the weight.
Builder And Verifier Must Differ
The rule is structural: the model that builds and the model that verifies come from different model families. A coding agent from one family writes the change. A reasoning model from a different family reads the spec and the diff cold and argues against it. The two do not share a failure surface, which is the entire point — a reviewer that shares the author's priors is not an independent check, it is the same check run twice.
The reason to distrust self-review is not that models are bad at reviewing. It is that they are systematically biased toward their own output and toward the user's framing. The research on using a language model as a judge (Zheng et al., 2023) documents self-enhancement bias directly: judges favor the answers they themselves produced. The work on sycophancy in language models (Sharma et al., 2023) documents the other half: models trained on human preference learn to agree. Put those two together in one model asked to grade its own diff, and the review is not a check. It is a formality with a confidence score attached.
- The builder optimizes for producing a working change. The verifier optimizes for finding why it is wrong. These are different jobs, and one model doing both does neither.
- Cross-family disagreement is signal, not noise. When two families disagree about the same diff, that is where a human should be looking.
- The verifier reads the diff and the spec cold, with none of the builder's accumulated context or self-justification.
That last point is load-bearing and easy to skip. A long-running agent accumulates its own history — dead ends, abandoned approaches, decisions it has already talked itself into — and models degrade as relevant information gets buried in long contexts (Liu et al., 2023). A verifier handed the builder's session inherits the builder's justifications along with its blind spots. So the loop prunes: verification runs against a fresh view of the spec and the diff, not against the story the builder has been telling itself for the last hour.
Escalation Is A First-Class Stop
A prompt runner fires once and hands back whatever came out. A loop has to answer harder questions: has this converged, is it stuck, and — the one most systems never ask — is the task itself wrong?
The loop runs a four-state stop. Converged, when the work meets the written target. Stalled, when progress has stopped and further iteration is burning budget rather than closing distance. Failed, when the attempt is wrong in a way that is fixable. And escalated, which is a different animal: the task as defined cannot be satisfied, and the right output is not code but a message to a human saying so. Collapsing escalation into ordinary failure is how agent systems grind for hours producing plausible garbage against a premise nobody checked.
The most valuable thing an agent can tell you is that the task you gave it is wrong.
Naming escalation separately changes what the system is allowed to do. A loop that can only succeed or fail will keep trying, because trying is the only move it has. A loop that can escalate has a way to say the specification is contradictory, the requested behavior conflicts with an invariant, or this needs a decision no agent is authorized to make — and hand it back. That state is the cheapest thing in the loop, because the alternative is paying a model to be confidently wrong at length. It is also the state that maps to what risk frameworks call human oversight of consequential decisions (NIST, 2023): not a human reviewing every line, but a defined path by which the machine gives the decision back.
When Self-Grading Becomes The Risk
There is a boundary where a single-model, self-reviewing setup stops being a convenience and becomes the primary risk in the system. It arrives the moment agent output flows somewhere consequential without a human reading every line — a merged commit, a schema migration, a deployed change, an agent with production permissions and nobody watching what it does with them. Before that line, self-review is a productivity choice. After it, the absence of an independent verifier is an unmonitored failure path, and it will eventually ship a confident mistake.
The reason the boundary is hard to feel is that nothing announces it. The velocity is real, the outputs look fine, and the exposure accumulates underneath in a form nobody has a dashboard for — right up until an outage gets attributed to an agentic tool and the vendor has to publicly deny it (The Register, 2026). The guidance that simple, well-scoped compositions beat elaborate single-agent orchestration (Anthropic, 2024) applies exactly here: two disciplined models checking each other outperform one clever model trusted on its own word, and the second model is cheaper than the first incident.
First Steps
- Split builder from verifier across families today, even crudely. A second model, from a different family, reading the diff against the written spec, will catch a class of confident error the author cannot see. This is a configuration change, not a project.
- Make escalation a distinct outcome. Give the loop a way to return "this task is wrong" that is not failure, and route it to a human. Then watch what it escalates — that queue is a specification bug report you did not know you were commissioning.
- Verify against a fresh context. Hand the verifier the spec and the diff, and nothing else. If your verification step can see the builder's reasoning, it is not verifying — it is agreeing.
Engineer The Loop, Not The Prompt
The durable move is to treat the loop as the engineered artifact — trigger, build, independent cross-family verify, four-state stop, context pruning, measurement — and treat the model as a swappable worker inside it. Prompts change with each vendor release. The loop is what stays, and it is the only part you can reason about. A system built this way survives model churn, because none of its reliability was ever staked on one model's self-assessment.
So here is the bar, and it is a bar you can hold any engineering organization to, including your own. Does a different model family check every consequential change? Can the loop tell you the task was wrong, as a distinct outcome from failing at it? Does verification run against a fresh context, or against the builder's accumulated story? Is any of this written down as a gate, or is it a habit that survives only while the person who invented it is paying attention? An organization that cannot answer those four questions is not running AI-assisted engineering. It is running an AI-assisted vibe, and the entropy that accumulates underneath it is the bill that arrives later.
Installing that loop inside a team, and owning it while it runs, is what an AI engineering retainer buys: a standing senior owner who holds the engineering discipline in place across live systems, not a set of recommendations that decay the week after the deck is delivered. The loop is not a deliverable you can hand over in a document — it is a practice, and practices need an owner or they stop compounding and start eroding. The discipline is not expensive. Not having it is.
References
- Zheng, L., et al. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. arXiv, 2023.
- Sharma, M., et al. Towards Understanding Sycophancy in Language Models. arXiv, 2023.
- Liu, N. F., et al. Lost in the Middle: How Language Models Use Long Contexts. arXiv, 2023.
- Anthropic. Building Effective Agents. Anthropic Engineering, 2024.
- National Institute of Standards and Technology. AI Risk Management Framework. NIST, 2023.
- Fortune. AI-Powered Coding Tool Wiped Out a Company's Database. Fortune, 2025.
- The Register. Amazon Denies Kiro Agentic AI Was Behind Outage. The Register, 2026.
