Research on temporal degradation in deployed models (Scientific Reports, 2022) tested 128 model-dataset pairs across healthcare, weather, finance, and transportation, and found 91% of them degrading over time — many with no drift in the underlying data at all. Not one of those degradations would have moved a DORA metric. Deployment frequency, lead time, change failure rate, and mean time to recovery would all have stayed exactly where they were, while the systems they described got worse.
DORA earned its place as the default engineering scoreboard, and the four metrics still measure something real. But they rest on an assumption so deeply baked in that it was never written down: that the humans shipping the code are the humans who understand it, review it, operate it, and can explain it under pressure. When much of a change set is drafted by a coding assistant, glued by an agent, or copy-edited by a model before review, the deployment record stops being a faithful proxy for what the team knows. Field reporting on this gap (LaPosta, 2026) frames it as a legibility problem, and the framing is right.
The control that answers it is not a new metric — it is a separation of duties, and it has to be structural. In the engineering loop ML LABS runs, the builder and the verifier are drawn from different model families, so the model that wrote a change is never the model that approves it; the verifier reads the spec and the diff cold, with none of the builder's accumulated justification; and the loop has four named stop states, one of which is escalation — the task as written is wrong — because the most valuable output of an automated change is sometimes a refusal. None of that produces a number on a delivery dashboard. All of it decides whether the change should have shipped.
What DORA Measures And Misses
The four metrics measure the surface of delivery. Deployment frequency and lead time describe how fast work reaches production; change failure rate and mean time to recovery describe how the system behaves when a change misbehaves. Read together they give a useful picture of throughput and resilience — as long as the assumption underneath holds.
The assumption is that every artifact in the pipeline reflects a deliberate engineering decision someone can defend, which is why the metrics correlated with outcomes that mattered in the first place. The multi-dimensional framing of developer productivity (ACM Queue, 2021) made the adjacent case a decade into the DORA era: throughput is one dimension among several, and reading it alone was a compression even then. When changes flow through that nobody on the team can reconstruct, the metrics still register the activity. They cannot register the gap behind it.
Three Ways DORA Misleads Now
Legibility Drops While Velocity Holds
AI-assisted code can produce volume indistinguishable from senior output on a dashboard. Pull requests merge, deploys succeed, lead time compresses. What is missing is the mental model that used to be a byproduct of writing the code — the engineer who accepted a suggestion did not necessarily build the understanding that suggestion implies.
Review degrades from the other side of the same trade. As the volume of plausible-looking diffs rises, review compresses toward a plausibility check: does the output look right, do the tests pass. The substantive judgment review exists to apply — whether this change belongs here, whether the abstraction is right, whether it composes with the surrounding system — is the part that gets crowded out first, because it is the part that costs the most attention. Work on coding assistants already shows that developer perception of productivity tracks suggestion acceptance more than retained code (Ziegler et al., 2022), which is the same signal mismatch DORA inherits: activity is easy to count, comprehension is not.
Hidden Changes Bypass The Pipeline
A growing share of operationally critical change lives outside the surface DORA observes. Vendor toggles, identity provider rules, feature flags, model prompts, agent tool definitions, and infrastructure drift all reshape system behavior without producing a pull request. None appears in deployment frequency. None counts against change failure rate when it breaks.
Earlier work on the dominance of glue, configuration, and data dependencies as hidden debt (NeurIPS, 2015) generalizes cleanly here: the change surface is wider than the codebase, and the parts furthest from review are the ones most likely to cause an outage. A dashboard scoped to the repository is measuring the part of the system that was already under control.
Fast Recovery Masks Brittle Systems
A low mean time to recovery reads as operational strength. It can also be the opposite: the team rolls back aggressively, the critical path returns, the metric records a clean recovery, and the question of what actually broke is never answered. The rollback ends the incident without ending the condition that produced it, and the next expression of that condition arrives against a different shadow surface — where the dashboard, once again, looks healthy.
The recognizable version is the unacknowledged dependency the on-call team was certain could not be the cause. A rollback proves it was, service is restored, and the metric reports a win while staying silent about the comprehension gap that produced the incident. Fast recovery without root-cause clarity is a leading indicator dressed as a lagging one.
graph TD
A["AI-Assisted Delivery"] --> B["Deployment Frequency Rises"]
A --> C["Team Comprehension Drops"]
B --> D["Dashboard Looks Healthy"]
C --> E["Critical Path Unexplainable"]
D --> F["False Confidence"]
E --> F
F --> G["Outage From Hidden Surface"]
style A fill:#1a1a2e,stroke:#ffd700,color:#fff
style B fill:#1a1a2e,stroke:#16c79a,color:#fff
style C fill:#1a1a2e,stroke:#e94560,color:#fff
style D fill:#1a1a2e,stroke:#16c79a,color:#fff
style E fill:#1a1a2e,stroke:#e94560,color:#fff
style F fill:#1a1a2e,stroke:#ffd700,color:#fff
style G fill:#1a1a2e,stroke:#e94560,color:#fffWhen the dashboard and the team's actual grip on the system point in opposite directions, the dashboard is the one that is wrong.
Pair DORA With Comprehension
The fix is not to throw DORA out — the pipeline metrics still tell you whether the delivery machinery works. They need a companion frame that observes what they cannot, so the two signals can be read against each other rather than one standing in for both.
Three augmentations carry most of the weight. Every change set carries a short scope note naming its risk profile, the surfaces it touches outside the main repository, and the vendor or identity configuration it depends on. Quarterly metric reviews are paired with two or three real incident narratives, root cause written out rather than compressed into a recovery-time number. And the team tests its own legibility on a cadence: can members independently explain the critical path in plain language, and where do the explanations diverge. The divergence is the measurement.
The deeper shift is in how the numbers get used. The moment DORA becomes a scoreboard for grading individuals or feeding promotion decisions, the team optimizes for the dashboard, and the gap between the dashboard and the system widens in the one direction nobody is watching — which is what a live AI system costs when nobody owns it, arriving through the metrics rather than around them.
When DORA Alone Is Still Enough
The critique has limits. On a team with a single owner of the production system, where AI assistance is light and most changes pass through the person who wrote the surrounding code, DORA on its own remains a reasonable read of engineering health. The comprehension signal is implicit, because the system fits in one head and that head is on the on-call rotation.
The same holds for mature systems with a low change rate, regulated environments where every change already carries an auditable trail by mandate, and codebases where the deployment log is the authoritative record of what exists. Augmentation earns its cost exactly where the change surface is widening faster than the collective understanding of it — which is the condition AI-assisted delivery creates by default, not by accident.
First Steps
- Add a one-paragraph scope note to every change, naming the risk profile and the out-of-band dependencies — vendor configs, identity rules, agent definitions, feature flags — that the diff alone does not reveal.
- Pair the next quarterly DORA review with three real incident narratives from the period, written as plain-language post-mortems rather than reduced to recovery times.
- Run the critical-path test: ask each member of the team to independently explain the end-to-end path under load, and record where the explanations diverge. Divergence is the finding, not a failure.
Measure Comprehension, Not Velocity
The bar is short enough to hold your own organization against this quarter. The generator of a change is never its verifier, whether both are people or one is a model. Every change carries the surfaces it touches outside the repository. Every incident produces a written root cause, not just a recovery time. And somebody can explain the critical path without opening the code — because a system nobody can explain is a system nobody is operating, whatever the dashboard says. The original DORA emphasis on outcomes over targets (Forsgren et al., 2018) supports this reading, and it lines up with the older observation that essential complexity in software cannot be engineered away (Computer, 1987), only made legible.
Where the gap is already open and nobody inside the team has the vantage point to see it, that is an ownership problem before it is a metrics problem, and the risk accumulates on the shadow surfaces the dashboard was never pointed at. Managed AI operations puts a senior owner on a defined scope of live systems — monitoring, drift and cost review, incident response, and a written operating brief every month, cancellable on 30 days' notice — which is what turns comprehension from an aspiration into somebody's job. The separation of duties that keeps the shipping side honest is why AI coding agents need separate verifiers.
References
- LaPosta, P. DORA Metrics Are Lying To You, And AI Is Making It Worse. LeadDev, 2026.
- Forsgren, N., Storey, M.-A., Maddila, C., Zimmermann, T., Houck, B., Butler, J. The SPACE of Developer Productivity. ACM Queue, 2021.
- Forsgren, N., Humble, J., Kim, G. Accelerate: The Science of Lean Software and DevOps. IT Revolution Press, 2018.
- Ziegler, A., Kalliamvakou, E., Simister, S., Sittampalam, G., Li, A., Rice, A., Rifkin, D., Aftandilian, E. Productivity Assessment of Neural Code Completion. arXiv, 2022.
- Sculley, D., et al. Hidden Technical Debt in Machine Learning Systems. NeurIPS, 2015.
- Brooks, F. No Silver Bullet: Essence and Accidents of Software Engineering. Computer, 1987.
- Vela, D., Sharp, A., Zhang, R., Nguyen, T., Hoang, A., and Pianykh, O. S. Temporal Quality Degradation in AI Models. Scientific Reports, 2022.
