A production AI system is not an asset that sits still. It is a position, and a position has to be held. The argument for building on shared foundations is sound — the second system costs less than the first, the fifth costs less than the second, the platform absorbs the fixed work — but a condition rides along with it, and the condition decides everything: every one of those gains is held in place by somebody. Remove the somebody and the same machinery runs in reverse.
The research has been reporting the reverse case for a decade. Research on hidden technical debt in ML systems (NeurIPS, 2015) found that model code is a small fraction of a real production system; the rest is machinery that keeps the model attached to reality, and machinery is maintained or it is not. The models do not hold still either. Research on temporal degradation in deployed models (Scientific Reports, 2022) tested 128 model-dataset pairs across healthcare, weather, finance, and transportation, and found 91% of them degrading over time — many with no drift in the underlying data at all. Nothing attacked those systems. They got old while running.
The expensive version of this is not a model that visibly gets worse. It is a system that gets worse in a direction nobody is watching, while every surface anyone looks at stays green. ML LABS met that system inside a hedge fund: unnecessary and polluted data accumulating at scale, unnoticed, while everything that was supposed to run ran.
The Data Nobody Was Reading
The fund was storing large volumes of unnecessary and polluted data without noticing. Three things were wrong at once, and not one of them was an error. The aggregation mechanism feeding the models was discarding information that mattered for training — detail that went into the pipeline and did not come out the other side. The storage structure duplicated data, so the fund was paying to keep several copies of records it did not need in the first place. Nothing in the system objected, because objecting was not available to it: a pipeline that duplicates still returns rows, and an aggregation that throws away variance still produces a clean, readable table.
Correcting the foundation cut the fund's storage costs by more than 60%, improved the detail and accuracy retained for training, and made their models perform 2% better. For a fund whose edge is the model, two percent is not a rounding error — it is the thing the entire modeling effort exists to produce, and it was recovered from the storage layer rather than from the models. The link is not exotic: the effects of data quality on machine learning performance (Mohammed et al., 2022) are measurable, and duplicates and dropped fields are two of the dimensions that get measured.
"We came to Omar with messy data and a deadline we'd already missed once. He turned it into a working system in days, kept us updated throughout, and it held up in production — reliable in a way this field rarely is." — Head of Data, hedge fund
The engagement did not begin as a data-quality project. It began as a deadline. The storage cost and the lost training signal were not what anyone asked us to look at — they were what nobody had noticed, and nothing had failed. A loss with no error message runs until someone goes looking for it, because an alert is what ends a loss and there was no alert to raise. The data foundation a working AI build actually needs is not the clean one; it is the one somebody reads.
Decay Is The Default Direction
Left alone, a live AI system moves in three directions at once, and all three are quiet. Spend creeps wherever usage grows and deletion is nobody's budget line. Quality ages whether or not the world changes around it — the Scientific Reports work above is a study of models that decayed on their own. And context leaves, because the knowledge of why the pipeline drops that field lives inside a person, and people move.
The third one is the compounding loss, and no dashboard has a widget for it. Research on data drift mitigation in large-scale systems (MLSys, 2022) treats drift as a permanent operating condition to be managed rather than an incident to be fixed, and managing is something a person does. Work on how complex systems drift into failure (Dekker, 2011) describes the mechanism exactly: failure arrives through an accumulation of locally sensible decisions, each defensible on its own day, with no broken component anywhere in the chain. Nothing pages anyone either: alerting fires on symptoms someone named in advance (Google, 2016), and a decay that produces no error offers no symptom to name.
graph TD
A["Live AI System"] -->|"no owner"| B["Spend Creeps"]
A -->|"no owner"| C["Quality Ages"]
A -->|"no owner"| D["Context Leaves"]
B --> E["Nothing Alerts"]
C --> E
D --> E
E -->|"someone looks"| F["Cost Cut, Model Improved"]
style A fill:#1a1a2e,stroke:#fff,color:#fff
style B fill:#1a1a2e,stroke:#e94560,color:#fff
style C fill:#1a1a2e,stroke:#e94560,color:#fff
style D fill:#1a1a2e,stroke:#e94560,color:#fff
style E fill:#1a1a2e,stroke:#ffd700,color:#fff
style F fill:#1a1a2e,stroke:#16c79a,color:#fffEach live AI surface decays in its own dialect of this. An approval path decays once the person who defined it leaves, because an exception route with no owner has nothing pushing back against it — which is why oversight of AI agents in production is an operating discipline rather than a launch checklist. Code accumulates faster than the understanding of it when AI writes the code, and the ceiling that produces is its own argument. Delivery metrics can report green while the risk moves somewhere they were never pointed, which is what DORA metrics miss about AI-era engineering. One mechanism under all of them: the system still runs, so nothing raises a hand.
What An Owner Actually Does
ML LABS runs an ongoing engineering retainer for HeartSciences, the medical-device company whose cloud ECG backend the practice designed and built. Delivery, then ownership, is not a theory in that engagement — it is its literal history, and what the ownership has produced is not a maintenance log. It is systems.
Two of them are published. The clinical billing automation built under that retainer ran in shadow mode and was not allowed to cut over until it held 98%+ agreement with expert-adjudicated determinations; the same work fixed the line at which the data rather than the model is the blocker — extraction leaving more than 15–20% of records with missing critical fields. The multi-site clinical operations program reaches further out, automating across facilities whose EMR behavior and data paths differ, without pretending they do not. Neither system is part of the ECG platform build. Both were delivered under the retainer that followed it — which is the argument for a standing owner in one line: the engineer who already holds the context is the one positioned to find the next system worth building.
The contract structure behind that is ML LABS' own, so it can be stated in full. Three terms carry it:
- An independent engineering partner — not staff augmentation, not a managed resource, not a body against a seat.
- AI-assisted delivery, acknowledged in writing. A senior operator leveraged by agents is what the client signs for, not what they discover. The builder/verifier split across model families behind it is a system the practice runs — the same one rebuilding this site.
- Cancellable on thirty days' notice, without penalty, with a refund if the practice consistently fails to perform.
That last clause is the load-bearing one. A retainer that can be ended at any time has to earn the month it is in, and a retainer nobody can end becomes exactly the drift it was hired to prevent. The owner is also there to make the team stronger rather than more dependent — what an organization loses by adopting AI without learning anything is the failure mode on the other side of that.
Where The Leverage Accumulates
The upside case is real, and it hangs on the same hook. A second system is cheaper than the first only if the first one left something behind, and things get left behind when somebody is responsible for leaving them. A shared ML platform initiative (LinkedIn Engineering, 2019) set out to double ML engineer effectiveness through reusable feature schemas; one production ML platform (Del Balso & Hermann, 2017) centralized roughly ten thousand shared features and removed the duplicate engineering underneath them. Three assets carry the leverage:
- A feature layer computed once — documented, versioned, discoverable, so the next project searches before it builds.
- One serving and monitoring surface instead of four, so the on-call path does not multiply with the project count.
- A written record of decisions, benchmarks, and failures, which is what turns one project's experience into the next project's starting position.
None of the three survives unattended. A registry nobody curates cannot be authoritative: the moment two definitions of the same customer metric coexist and nobody may retire one, it is a menu rather than a source of truth. A platform with no owner has nobody to argue with, so it gets routed around whenever routing around it is the faster path — and then there are four serving stacks, the monitoring surface has quadrupled, and what breaks first under growth is four different things.
Three trend lines say which way it is going, plotted against project sequence rather than the calendar: time-to-production per project, cost per project, and the share of a new build that is assembled rather than written. Flat lines mean every project is rebuilding the same scaffolding under a different name, whatever the architecture diagram claims. The dashboard those lines live on needs an owner too — measuring what a live AI system is actually doing is a different discipline from measuring a build.
When A Retainer Is Wrong
Ownership is a purchase, and there is a size below which it is the wrong one. An organization running a single AI system, with the people who built it still in the building and still on it, does not need a standing owner — it needs to not lose them. The purchase starts paying when the number of live systems exceeds the number of people who genuinely understand them, or when the ones who did understand have moved on and the system is running on documentation.
The other failure condition is authority. An owner with no mandate to retire a duplicate pipeline, refuse a fifth serving stack, or hold a release until a gate passes is a consultant with a recurring invoice, and no contract structure repairs a missing mandate. Where the mandate cannot be granted, the lighter mechanism is the honest one: a written record of decisions, benchmarks, and failures that the next project can actually read. It builds value more slowly than an owner does, and it still builds.
First Steps
- Price what nobody reads. Take one live system and total what it costs to store data that no model, no report, and no auditor has touched since it landed. Storage is where unowned cost accumulates, because nothing in a system ever asks to be deleted.
- Trace one aggregation end to end. For a single production model, follow the path from raw capture to training input, and name every field the pipeline collapses, drops, or writes twice. The fund's 2% was in that path.
- Name the owner. For each live AI system, write down the person accountable for its cost and its quality. A blank line there explains every flat trend line above it.
One Owner, Written Targets, Cancellable
Concentrate the ownership rather than distributing it: one senior engineer accountable for every live AI system, targets written down before the work starts, the authority to retire what should not exist, and a contract the buyer can end on short notice. The cancellation clause is what keeps the other three honest, which is why the bar to hold a fractional owner to can be set as high as it is. Whether the next engagement should be a standing owner or a scoped build turns on how many live systems are already running without one.
The hedge fund's storage bill did not fall because anyone shipped a new model. It fell because someone with the authority to change the data path went and read it, and the models improved as a byproduct of the same look — the return on ownership arrives in places no launch metric was ever pointed at. That is the position, and it pays only while it is held. An AI engineering retainer puts a senior owner on your live systems permanently: the same engineering that builds them, still attached, still accountable, still cancellable the day it stops earning its keep.
References
- Sculley, D., et al. Hidden Technical Debt in Machine Learning Systems. NeurIPS, 2015.
- Vela, D., Sharp, A., Zhang, R., Nguyen, T., Hoang, A., and Pianykh, O. S. Temporal Quality Degradation in AI Models. Scientific Reports, 2022.
- Mallick, A., Hsieh, K., Arzani, B., & Joshi, G. Matchmaker: Data Drift Mitigation in Machine Learning for Large-Scale Systems. Proceedings of Machine Learning and Systems, 2022.
- Dekker, S. Drift into Failure. Ashgate/Routledge, 2011.
- Mohammed, S., Budach, L., Feuerpfeil, M., Ihde, N., Nathansen, A., Noack, N., Patzlaff, H., Naumann, F., & Harmouch, H. The Effects of Data Quality on Machine Learning Performance. arXiv, 2022.
- Google. Monitoring Distributed Systems. Site Reliability Engineering, 2016.
- LinkedIn Engineering. Scaling Machine Learning Productivity at LinkedIn. LinkedIn Engineering Blog, 2019.
- Uber Engineering. Michelangelo: Uber's Machine Learning Platform. Uber Engineering Blog, 2017.
