Pilot purgatory has a precise definition, and it is not "too many experiments". A pilot is in purgatory when nobody can promote it and nobody is allowed to kill it — when the criteria that would settle either decision were never written down, so the pilot survives on the only currency available to it, which is optimism. It keeps its budget line. It keeps its slide. It never reaches a user. Research on AI in business (MIT Sloan, 2024) put the shape of it plainly: the belief that AI confers competitive advantage is close to universal among executives, and the share of organizations that have incorporated it at scale is not.
That gap is where budget and credibility drain away together, and it is a criteria problem before it is a technology problem. A pilot's job is to answer a feasibility question. Feasibility is the easy half — the half a demo can settle — and organizations that conflate it with production-readiness end up promoting systems that were never evaluated against the thing that decides their fate. This article is about which pilots to promote; the engineering of getting one across the line is covered in the engineering path from AI pilot to production.
The distinction is not academic. On the cloud ECG backend ML LABS built for HeartSciences, the numbers that decided whether the AI layer was working in clinical production were never the aggregate accuracy figures on a model vendor's data sheet. They were per-provider tail latency at the 95th and 99th percentile, retry rates broken down by failure class, and the share of studies where two providers disagreed on a finding a clinician would act on. A pilot that clears a benchmark has answered none of those questions, and none of them are visible from inside the pilot.
Feasibility Is The Easy Gate
A demo proves that a model can produce an output. Production asks whether an organization can absorb one. "We can predict churn accurately enough to act on it" is an answer to a technical question, and it leaves every operational question open: can the system reach the CRM, can it act inside the window where acting still matters, and does anyone own it after the data science team moves to the next thing. Without answers, the pilot sits exactly where the definition puts it — too promising to kill, too incomplete to ship.
The most expensive version of that limbo is the pilot that clears every technical check and lands with no operational home. The team that built it wants to move on. The team that must run it did not build it and has no context for it. The system deploys with nobody accountable, degrades where nobody is looking, and eventually fails in a way that reads as an AI failure rather than an ownership failure — which is precisely what a live AI system costs when nobody owns it.
Portfolio breadth is not the cure for this, and there is reason to think it is part of the disease. Research on AI deployment rates (Deloitte, 2024) tracks how few enterprise pilots reach scaled deployment. The mechanism is structural: a gate is a decision to stop something, a committee cannot stop anything without a majority, and every additional concurrent pilot dilutes the attention of whoever would have to build the case for stopping it. Concentration is what makes a kill decision cheap enough to actually make.
Three Gates, Each One Harder
Moving from experiment to impact needs explicit criteria, one accountable owner, and kill conditions defined before anyone is emotionally invested in the answer. Three gates, each progressively harder to clear. Only the first is a question the pilot can answer on its own terms; the other two have to be answered by the organization that would have to live with the system.
graph TD
A[Hypothesis] --> B[Pilot]
B --> C{Gate 1: <br/>Technical<br/>Viability}
C -->|Pass| D[Integration<br/>Prototype]
C -->|Fail| X1[Kill or<br/>Redesign]
D --> E{Gate 2:<br/>Operational<br/>Readiness}
E -->|Pass| F[Limited<br/>Production]
E -->|Fail| X2[Return to<br/>Pilot]
F --> G{Gate 3:<br/>Business<br/>Impact}
G -->|Pass| H[Full<br/>Production]
G -->|Fail| X3[Sunset or<br/>Pivot]Gate 1: Technical Viability
The lowest bar, and the only one a demo is built to clear. Verify performance on representative production data rather than a curated demo set, confirm that latency fits inside the business process the system is supposed to serve, and check that a second engineer can reproduce the pipeline end to end from what exists in the repository. If performance on representative data falls below the threshold at which the output is useful to the business, the pilot is killed and the finding is written down. A documented negative result is a cheap asset; an undocumented one is a cost the next team pays again.
Gate 2: Operational Readiness
This is the gate that decides the pilot's fate, and the one a demo cannot rehearse. Research on ML systems in production (NeurIPS, 2015) established the reason: the model is a small fraction of a real production ML system, and the surrounding machinery is the rest. The gate tests the machinery. Real data sources with defined contracts. Drift monitoring pointed at the signals that would move first. Documented failure procedures owned by a named team. A completed security review. And the property that matters more than any metric — every record reaches a definite terminal state, and that state is queryable, so reconciliation is a query rather than a heroic act. If no team will accept production ownership, the honest outcome is a return to pilot or a kill, because deployment without an owner is not a graduation.
Gate 3: Business Impact
Evaluated after a defined production window — long enough to accumulate signal, short enough to stay accountable, and named in advance so it cannot be extended by whoever fears the answer. Measure the business metrics against their pre-deployment baselines, confirm that the people the system was built for have actually abandoned the old pathway, and verify that the economics hold at peak load rather than at demo load. Adoption is the honest proxy: a system users route around has failed regardless of its accuracy. If impact cannot be shown inside the window, the system is sunset. Sunk cost is not an argument.
The Zombie Zone
Between Gate 2 and Gate 3 sits the most expensive state a system can occupy: technically deployed, generating cost, generating no value, and generating no alarm either. A zombie is worse than a failed pilot, because a failed pilot stops billing. Every deployed system therefore needs a scheduled impact review with a remediation window and a decommissioning path that someone has the authority to walk down.
Making The Gates Stick
Gates decay into paperwork unless two things hold them up. The first is accountable ownership: research on scaling AI (McKinsey, 2024) associates dedicated AI leadership with higher rates of production deployment, and the mechanism is not mysterious — a gate is only a gate if someone with enough authority to absorb the disappointment is standing at it. The second is a shared vocabulary for what "ready" means, which is what the AI Risk Management Framework (NIST, 2023) supplies: governance, mapping, measurement, and management as named functions rather than as a feeling about the system. Invest proportionally to gate progress — minimal before Gate 1, moderate between Gates 1 and 2, full commitment only after Gate 2 — so that a kill is cheap right up to the point where the evidence says it should not happen.
When Nobody May Kill Anything
The framework has one hard precondition, and no amount of process substitutes for it: someone must be permitted to kill a pilot. Where leadership has committed publicly to every initiative, gate reviews still happen, decks still get built, and no pilot is ever stopped — the gates become a reporting ritual attached to a portfolio that only grows. The same failure arrives more quietly when the graduation owner exists but any department head can overrule a kill.
That authority is worth more than it looks, because a kill is the only decision in this process that returns its entire remaining budget — and the decision not to build at all returns the whole thing. A major US TV network came to ML LABS having already been quoted a full software system for a workflow that did not require one. The technology was sound and the build was buildable; nobody had asked whether the workflow needed custom software at all. The scoping session's deliverable was that it should not be built.
"Omar delivered in two weeks what our team estimated would take six months. The scoping session alone saved us from a $200K mistake." — AI Program Manager, a major US TV network
First Steps
- Inventory the pilots and apply Gate 1 retroactively. If the list cannot be assembled from memory and a spreadsheet, the inventory is the first finding, not a preliminary step.
- Hunt the zombies. Run Gate 3 against everything already deployed. Any system whose business impact cannot be demonstrated with data is a zombie candidate, and its running cost is the number that makes the case.
- Name a graduation owner with kill authority. One person, not a committee, with the standing to stop a project on the evidence and no career exposure for doing it.
The Bar A Pilot Must Clear
The bar is short enough to hold your own portfolio against it right now. Every pilot has written promotion criteria and written kill criteria, both agreed before the work starts. Every gate has one named owner who can stop the thing. Every deployed system has a scheduled impact review with a decommissioning path attached. And nobody is promoted for shipping a system that no user adopted, because adoption — not accuracy — is what the third gate measures.
An organization that clears the bar promotes on evidence and kills without blame, and its capacity for production work stays intact instead of being consumed by systems nobody will defend and nobody may retire. An organization that clears none of it does not have a pilot problem; it has an authority problem wearing a pilot problem's clothes. Deciding which experiment deserves production budget is a discrete decision that can be bought on its own: an AI scoping session turns the portfolio question into a written recommendation with a go or no-go and a next step, at $750 credited against the work if the work goes ahead — and, as the network above found, the recommendation is sometimes that the most valuable thing you can do with a pilot is stop it. For a first use case rather than a queue of them, how to find the one worth doing is the companion piece.
References
- MIT Sloan Management Review. Artificial Intelligence in Business Gets Real. MIT Sloan Management Review, 2024.
- Deloitte. State of AI in the Enterprise. Deloitte Insights, 2024.
- Sculley, D., et al. Hidden Technical Debt in Machine Learning Systems. NeurIPS, 2015.
- McKinsey & Company. The State of AI. McKinsey Global Survey, 2024.
- NIST. AI Risk Management Framework. National Institute of Standards and Technology, 2023.
