Measure AI Impact By Decisions, Not Accuracy

Model accuracy is not a business result. It is a property of a model measured against a dataset, and no budget owner has ever been able to spend one. A system can clear every target on its model card and change nothing about how a company operates — the predictions arrive too late, land in a tool nobody opens, or answer a question that was never blocking anyone.

A survey of 1,000 C-suite executives (BCG, 2024) found that 74% have yet to show tangible value from AI, even as budgets increase. That is not mostly a modeling gap. Research on scaling AI (MIT Sloan, 2024) found that the organizations which do scale define business outcomes before model metrics, and research on the productivity paradox (NBER, 2018) showed that even national statistics fail to capture AI's benefits — a project with no pre-AI baseline measures under the same handicap and none of the excuses.

The fix is not a better model metric. It is a measurement stack that follows the value from what the system does to what the business decides. On the cloud ECG backend ML LABS designed and built for HeartSciences, the numbers that decided whether the platform worked in clinical production were never the accuracy figures on a model provider's data sheet. They were per-provider tail latency at the 95th and 99th percentile, retry rates broken down by failure class, and the share of studies where two providers disagreed on a clinically actionable finding.

Four Levels Of AI Measurement

Measurement that survives a budget review moves through four levels: inputs describe what the system is fed, outputs what it produces, outcomes what people do differently because of it, and impact what the business banked — net of what the system costs to run. The bottom two come free with the infrastructure. The top two have to be built deliberately, which is why a dashboard can be entirely green while the BCG finding above stays true of the company that owns it.

graph TD
    A["INPUTS<br/>Data freshness, uptime"] --> B["OUTPUTS<br/>Accuracy, latency, coverage"]
    B --> C["OUTCOMES<br/>Decisions taken, overrides"]
    C --> D["IMPACT<br/>Revenue, cost, risk"]

    style A fill:#1a1a2e,stroke:#ffd700,color:#fff
    style B fill:#1a1a2e,stroke:#e94560,color:#fff
    style C fill:#1a1a2e,stroke:#0f3460,color:#fff
    style D fill:#1a1a2e,stroke:#16c79a,color:#fff

The outcome level decides the rest. Research on AI and human decision-making (HBR, 2021) found the primary determinant of AI value is whether the system changes decision-making behavior — which makes override rate the most diagnostic signal available, and the most misread. Research on algorithm aversion (Journal of Experimental Psychology, 2015) showed that people abandon a model faster than they abandon a human forecaster after identical errors, so a climbing override rate can be trust collapse rather than model degradation. Retraining a healthy model because its users stopped believing it is an expensive way to make the problem worse.

None of the levels is legible without a baseline: current performance, the method used to measure it, and the variance range that separates an improvement from a good week. Research on AI value metrics (Gartner, 2024) recommends substantial baseline data and a meaningful post-deployment window before anyone claims impact — and where that impact lands on the calendar is a payback question of its own.

Signals A Data Sheet Hides

The levels are a frame. What fills them is domain work, and that is the part nobody can buy. The ECG platform's AI layer runs studies through multiple model providers, each with its own latency behavior, availability pattern, and billing consequence — and a provider's published accuracy is a laboratory claim about a model, not about the system a clinician touches. What decides clinical trust lives in the tail: the slowest requests, the malformed response, the retry that must never become a second billable result for the same study.

Each of those three signals changes an engineering decision. Tail latency is read per provider, because an average across providers hides the one that is failing. Retries are counted by failure class — network, malformed response, model error — because the three have different owners and different fixes. And cross-provider disagreement on an actionable finding is the closest thing to a live accuracy signal a production system can have without a ground-truth label beside it.

Aggregate accuracy is a claim about a model. Tail latency, retry rate by failure class, and cross-provider disagreement are claims about the system a clinician actually uses — and only the second kind can be acted on.

Not one of those is an accuracy number, and each names its decision: which provider to route to, which failure class to engineer against, which finding to escalate. That is the test for whether a metric belongs on the dashboard at all — name the decision it changes, or take it off. Applied to a tool budget rather than a model, the same test separates a stack from a subscription pile, which is the whole of the ROI question for AI tooling.

Where Value Leaks Between Levels

Value does not leak inside a level. It leaks at the transitions, and each transition has a signature the dashboard should surface on purpose rather than bury inside a level-internal average.

• The technical break. Healthy throughput with degrading accuracy, or a rising tail that pushes consumers back to their defaults. The system is alive; its predictions have stopped earning their place in the workflow
• The adoption break. Strong accuracy, flat action rate. Slice overrides by cohort and decision type: when one cohort or one class of prediction concentrates them, the problem is workflow fit, not model quality
• The attribution break. Decisions are changing and the business metric does not move — the model is influencing the wrong decisions, or the right ones in volumes too small to register

A model that is right and ignored and a model that is wrong and ignored produce the same business result. Only one of them shows up as a problem on the model dashboard.

A schematic scenario makes the adoption break concrete — invented, not a client record. Say a demand-forecasting model serves a dozen regional planners and the output metrics look good enough to open the steering-committee deck with. The outcome metrics say otherwise: few planners open the forecasts, overrides run high among the ones who do, and the spreadsheet the model was built to replace still decides most orders. The scenario is invented; the split it illustrates is not, and retraining fixes neither half of it. The work is in the workflow — move the forecast into the tool the other planners already use, then read the overrides from the engaged few for a blind spot or a class of decision the model should never have been handed.

Attribution Without Self-Deception

A naive before-and-after comparison cannot separate the model's effect from everything else that moved in the same window: a pricing change, a seasonal swing, a competitor's outage. Three approaches, in descending order of rigor:

1. Randomized A/B against a holdout. The only method needing no assumption about what would otherwise have happened — and it needs an experimentation surface you own, which is why borrowed analytics cannot run the test
2. Difference-in-differences. Compare the before/after change against a comparable metric the system does not touch
3. Interrupted time-series. Project the counterfactual from the pre-deployment trend, then measure the deviation

Pick the method before deployment and write it down, because a method chosen after the results are in is a method chosen to fit them. There is also a fourth leak the hierarchy does not draw, because it opens after the dashboard is built: nobody owns the measurement, and it quietly stops matching the system. On a hedge-fund engagement — told in full in the ownership argument for AI systems — polluted data accumulated unnoticed, an aggregation step was discarding information the models needed, and nothing in the client's own reporting surfaced either. Once someone whose job was to look actually looked, storage costs came down by more than 60% and the models performed 2% better.

When Outcomes Cannot Be Observed

The hierarchy assumes outcomes can be observed and attributed, and two conditions break that. The first is a workflow with no capture point: when the decision happens in someone's head, in an email, or in a spreadsheet the system cannot see, no adoption or override metric exists to collect, and a dashboard built before the capture point will faithfully report zeros.

The second is causal density — where a dozen forces move the same number, no method yields a clean attribution, and insisting otherwise produces a figure that will not survive an audit. Neither is terminal. Instrument the human act first; that payoff is forward-looking and worth more than another point of model accuracy. Under high causal density, run on leading indicators and let the lagging data accumulate.

First Steps

1. Name the decision each metric changes. Walk the current AI dashboard line by line; every metric that cannot be tied to a decision someone would make differently comes off it.
2. Find the capture point. Locate where a human acts on a prediction and instrument that act — adoption, override, action — before adding anything to the model layer.
3. Write the attribution rule before the next deployment. Pick a method, then record the baseline and its variance range while the pre-AI state still exists to be recorded.

The Bar For A Real Dashboard

The pattern is a layered stack: one metric per level on a single page, with baselines and the attribution method agreed before deployment rather than negotiated after the results arrive. Inputs and outputs prove the system is alive, outcomes prove somebody changed what they did, and impact proves the company banked it — the four together make the failure point visible instead of leaving it to be found in a budget review.

That gives a bar any program can be held to. Every metric on the page names the decision it changes. The baseline predates the deployment and carries its variance range. The attribution method was chosen before the results existed. And one owner stays accountable for the numbers still meaning what they meant at launch — a measurement stack is not a build artifact; it ages with the system it watches, and thresholds calibrated at launch keep reporting green against a distribution that has moved out from under them. Where that stack has no accountable owner, managed AI operations puts a named owner on it: the practice that runs the system also holds the dashboard that says whether it is worth running. A program clearing all four lines of that bar is measuring its AI spend — one clearing two is reporting on it, and the gap between them is the number the next budget review will ask for.

References

1. Boston Consulting Group. Where's the Value in AI?. BCG, 2024.
2. MIT Sloan Management Review. Winning With AI. MIT Sloan Management Review, 2024.
3. Brynjolfsson, E., Rock, D., and Syverson, C. Artificial Intelligence and the Modern Productivity Paradox. NBER Working Paper, 2018.
4. De Cremer, D., and Kasparov, G. AI Should Augment Human Intelligence, Not Replace It. Harvard Business Review, 2021.
5. Gartner. 5 AI Metrics That Actually Prove ROI. Gartner Research, 2024.
6. Dietvorst, B. J., Simmons, J. P., and Massey, C. Algorithm Aversion: People Erroneously Avoid Algorithms After Seeing Them Err. Journal of Experimental Psychology: General, 2015.