Turn Vague AI Requirements Into Checkable Targets

Onboarding a new organization onto HeartSciences' cloud ECG platform is a configuration task. Its HL7 field mappings, the AI models enabled for it, its invoice pricing, its storage provisioning — each one is an entry in a configuration surface, and none of them is a code change. ML LABS built that backend, and the platform was designed from the start so that bringing on a new organization would be configuration, never code.

Read that as a requirement and it has the two properties most requirements never acquire. It names the thing precisely enough to build against — onboarding an organization, not "scalability" — and it carries a pass condition a stranger could check: bring one on, and see whether an engineer has to touch the code. It was a design-time target, and it held.

Business intent arrives as an outcome — predict which customers will churn, automate invoice processing, detect fraudulent transactions. Each hides a dozen decisions that change the system's design, cost, and scope: how far in advance, at what confidence, from what data available at prediction time, and what happens to the prediction once it exists. A systematic mapping study on requirements engineering for AI systems (Ahmad et al., 2022) locates the damaging gaps not in the requirements that went missing, but in the ones that were clear in business terms and ambiguous in technical ones.

Traditional requirements assume deterministic behavior: given input X, produce output Y. AI systems are probabilistic: given input X, produce output Y with confidence Z, and sometimes Y is wrong. An analysis of AI project failures (RAND, 2024) attributes the majority of failed initiatives to misaligned requirements rather than to technical limits, and a case study of software engineering for machine learning at Microsoft (Amershi et al., 2019) reports the mechanism from the inside: ML behavior is entangled with data and produces non-monotonic errors, so it resists the clean modules a deterministic spec assumes. Written in that shape, a requirement encodes the wrong contract.

A specification can be satisfied in full and the work still fail, when nothing in it could be graded pass or fail by someone who had not helped write it.

What Makes A Target Checkable

ML LABS writes acceptance targets into the statement of work before work starts, and the standard each target has to clear is published with the contract: a target is objectively checkable when it names a workflow, names the data, and states a pass condition a third party could verify. If a target cannot be written that way, it does not go into the SOW. That is a filter with teeth, because the same document says a missed target — for reasons within our control — refunds the fee.

Read those two clauses together and the usual incentive inverts. A vague target stops being a diplomatic convenience and becomes an unpriced liability, because a promise nobody can grade cannot be shown to have been met either. A practice that has to refund against its own targets is a practice that cannot afford to write a soft one, and requirements discipline stops being a virtue and turns into arithmetic.

graph TD
    B["'Automate invoice<br/>processing'"]
    Q1["Which workflow,<br/>on which data?"]
    Q2["What is the<br/>pass condition?"]
    Q3["Who verifies it<br/>without us?"]
    T["Written SOW target"]
    R["Missed target<br/>= refund"]
    X["Not writable<br/>= not in the SOW"]

    B --> Q1
    Q1 --> Q2
    Q2 --> Q3
    Q3 -->|"all three answered"| T
    Q3 -->|"any one missing"| X
    T --> R

    style B fill:#1a1a2e,stroke:#0f3460,color:#fff
    style Q1 fill:#1a1a2e,stroke:#ffd700,color:#fff
    style Q2 fill:#1a1a2e,stroke:#ffd700,color:#fff
    style Q3 fill:#1a1a2e,stroke:#ffd700,color:#fff
    style T fill:#1a1a2e,stroke:#16c79a,color:#fff
    style R fill:#1a1a2e,stroke:#16c79a,color:#fff
    style X fill:#1a1a2e,stroke:#e94560,color:#fff

Four Ways A Target Fails

Four tests catch the softness before a line of code is written. Run them against the draft spec you have now — each maps to a piece of the definition above that is missing.

• Acceptance is qualitative. "The model should be accurate", "errors should be rare" — no metric, no sample, no threshold. If two reviewers cannot independently grade the same output pass or fail, there is no pass condition to accept against.
• The error budget is symmetric by default. A spec that never names which error the business prefers to absorb leaves the model optimizing whatever loss the framework defaults to — and that default holds no view about which of the two costs more in operations.
• Inputs are named but not sourced. "Customer engagement" appears in the spec, and nobody can point at the table, the refresh cadence, or the join path that produces it at prediction time. Training-serving skew is born here, before any code exists.
• Nobody owns the unhappy path. Ask what happens when the model is wrong or unavailable, and the answer is a shrug or a future-tense plan. A system with no defined fallback has no pass condition for its worst hour.

Where Targets Have To Land

A checkable target spans more than one sentence — one per surface the system crosses. These four are where the pass conditions have to land.

• Data. Source, schema, quality constraints, and features written as computations rather than nouns — "average monthly spend over the last six months" is a feature, "customer data" is not. Poorly defined features (Google, 2024) are a standing source of ML system bugs, and the groundwork that makes a data path buildable is what makes this surface writable at all.
• Model. The class of approach — supervised classification, regression, retrieval-augmented generation — never the specific model, which experimentation decides. What belongs in the target is the baseline: performance today without AI, the minimum acceptable for deployment, and the number that counts as success.
• Integration. Input and output contracts, the auth flow, the versioning scheme, and the error responses — what the system does on timeout, on low confidence, on missing data, and who hears about it.
• Operations. Monitoring, alerting, retraining triggers, and a rollback that has been exercised rather than documented. Research on scaling AI in organizations (MIT Sloan, 2019) puts all four before deployment, not after launch.

What A Missed Target Costs

What happens when a target is missed is the half that makes the first half real. Acceptance runs against the written targets and nothing else: ML LABS delivers written notice that the targets are met, attaches the evidence the SOW named, and an evaluation window of 15 business days opens. A rejection that names no specific unmet target does not stop that window. The obligation to be precise is symmetric — a buyer who cannot name the target that was missed stands where a vendor who could not write one stands.

A claimed miss is adjudicated against those written targets, on the evidence the SOW names — logged runs, recorded outputs, the pass conditions themselves — with one cure period of 10 business days from written notice. Misses caused by factors outside our control are not refund triggers, which is why every SOW lists its client dependencies: access, data, third-party systems. That list is a requirements artifact in its own right, dragging the assumptions a build would otherwise carry unspoken into the document that prices them.

Regulated work forces the sharpest version of the discipline. Where data may not leave the client's network, the build runs inside their VPC and the acceptance targets are defined over de-identified or synthetic data — a target gradeable only against production data cannot be graded before production, which is precisely when acceptance has to happen. Validation against real data follows acceptance, under the agreement signed at SOW. The constraint reads like a compliance concession and functions as a requirements upgrade: every pass condition has to be provable on data the engineer is allowed to hold.

It is the same discipline that let the onboarding target on the cloud ECG backend be stated the way it was. "Configuration, never code" is gradeable by a stranger holding a diff and one new organization. "Easy to onboard" is gradeable by nobody — and it would have survived every review meeting in the world.

When Goals Resist Measurement

This approach assumes stakeholder goals can be made concrete. Some cannot, at least not yet. "Make the customer experience better" with no defined metric, or two executives whose success criteria contradict each other, produces a specification that encodes the wrong target precisely and expensively — a translation pipeline is a machine for making intent buildable, and it will faithfully build the wrong intent.

Where the ambiguity persists, the resolution is ownership rather than more analysis. One decision-maker defines success in measurable terms and says the trade-off out loud — "we would rather miss a tenth of the fraud than freeze a twentieth of the legitimate transactions" — and only then is there something to write down. If that person does not exist yet, that is the finding, and the honest next move is to go find them rather than build against a committee's average. The test for whether your data is ready to build against has the same shape: the answer is a fact about your organization, not about the model.

First Steps

1. Grade your own spec. Take the requirement closest to shipping and try to write its pass condition: named workflow, named data, and a check a third party could run without you. Where the sentence will not come, you have found the work.
2. Name the error you prefer. Decide, in writing, which failure the business absorbs and which it refuses. That one sentence moves the model, the threshold, and the review queue.
3. Write the unhappy path. State what the system does when it is wrong or unavailable, and who is notified. A requirement with no defined fallback is not finished.

Make Acceptance The First Artifact

Write acceptance before you write architecture. The order matters more than the format: a target that survives being written down — named workflow, named data, a pass condition a stranger could check — is the one that survives contact with production, because it was never permitted to mean two things at once. Everything downstream becomes a derivation rather than a negotiation, and the expensive arguments happen while they are still cheap.

Targets that survive that test are what an AI system design is built to produce, and it puts money behind them: the kickoff scope note fixes the target in writing — the workflow, the dataset, the path a working spike has to demonstrate in your stack — and if the spike does not run as scoped, for reasons within our control, the fee comes back. A recommendation not to build is not a miss — the guarantee protects the spike, never the answer. What that engagement hands over beyond the spike — the plan, the fixed quote, the integration map — is the subject of the companion piece on what a good AI system design includes.

References

1. Ahmad, K., Abdelrazek, M., Arora, C., Bano, M., & Grundy, J. A Systematic Mapping Study on Requirements Engineering for AI-Intensive Systems. arXiv, 2022.
2. RAND Corporation. Analysis of AI Project Failures. RAND Corporation, 2024.
3. Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., & Zimmermann, T. Software Engineering for Machine Learning: A Case Study. ICSE 2019.
4. Google. Rules of Machine Learning: Best Practices for ML Engineering. Google Developers, 2024.
5. Ransbotham, S., et al. Winning with AI. MIT Sloan Management Review, 2019.