A request arrives at the edge. Before any origin server is touched, a Cloudflare Worker hashes the visitor with murmur3, drops them into a bucket, and hands back a signed cookie so that the same person meets the same variant tomorrow.
If that cookie cannot be signed, the Worker does something that looks like giving up: it assigns no variant at all and serves the control. That rule is deliberate, and it is the most opinionated line in the system.
That system is Versions, an experimentation and first-party analytics layer ML LABS builds and operates in the internal portfolio at Escape Velocity Labs, a sister practice under the same owner. A visitor whose assignment cannot be made sticky is a visitor who might be counted twice, in two different buckets, on two different days. They do not produce a missing number — they produce a plausible one that quietly poisons the result.
So Versions refuses the data point rather than accept a corrupt one. That refusal is the whole difference between owning your measurement and borrowing it, because a third-party tool cannot make that trade on your behalf. It reports what it can see, on terms it set, about a population it can still reach, and hands you a number with no way to ask how it was made.
Attribution Needs Controlled Comparison
A dashboard showing that a metric moved cannot tell you that your change moved it. Correlation is not attribution, and a change shipped into the noise of everything else happening that week is close to impossible to isolate afterwards.
The only mechanism that separates your effect from everything else in motion is the controlled comparison: variants shown to comparable groups at the same time, measured against the same definition. Everything else is a story told over a line chart.
Versions runs that comparison at the edge rather than inside the page, and the choice has consequences. Bucketing in a Worker puts the assignment before the origin responds, so a variant can be the entire site — a different structure, a different narrative, a different route — rather than a swapped button colour.
An element-level tool can only express an element-level question. If the change worth testing is a different shape of page, the tool cannot state it, so the question is not answered badly; it goes unasked.
- A concurrent comparison isolates your change from seasonality, campaigns, and everything else shipping that week.
- Whole-site variants let you test the changes that matter, not merely the ones that are cheap to instrument.
- The experiment and the metric live in one system, so "what changed" and "what it earned" resolve to one query instead of two tools and an argument.
Running trustworthy experiments is harder than the tooling makes it look. Sample-ratio mismatch, peeking before the horizon, and misread significance all produce confident numbers that are wrong, and the documented pitfalls of online controlled experiments (Microsoft ExP) are a long catalogue of them. Owning the layer is what makes those guardable — a borrowed dashboard has no opinion about your sample ratio.
AI Reads, Humans Decide
What AI adds to experimentation is not the decision. It is the compression: which variant moved which metric, where a result is too noisy to trust, what the next test should be.
That turns a pile of experiment output into a short ordered list a person can act on. Research on effective agent design (Anthropic, 2024) makes the same argument structurally: the useful pattern is a well-scoped model doing a legible job inside a system a human still owns, not a model handed the keys because its summary read well.
The restraint is the point. A system that auto-ships winners will eventually promote a false positive with total confidence, or a change that wins on the target metric while losing on something nobody instrumented.
A model can tell you which variant won. Only a person can say whether winning that was worth what it cost somewhere nobody was looking.
First-Party Data Is Now Survival
The case for owning measurement used to be accuracy. It is now availability.
A third-party analytics tool has to be allowed to run and allowed to keep state, and neither is a safe assumption any more. Safari became the first mainstream browser to block third-party cookies outright (Wilander, 2020), and every user a borrowed tool cannot see is a user missing from its sample.
The users it can still see are not a random subset of the rest. They differ in device, in technical sophistication, and potentially in the exact behaviour the experiment was built to measure — which makes the returned data not merely thinner but bent, in a direction nobody can inspect.
That is worse than having less data. Biased data with no visible shape is indistinguishable from good data right up until a decision made on it goes wrong.
First-party measurement, collected by your own system on your own domain, does not have that hole. It sees the users a third-party tool no longer reaches, and it collects them without shipping your users' behaviour to a company with its own plans for it. Privacy is the familiar argument for owning this and it holds — but first-party is now, more simply, the only data complete enough to bet a roadmap on.
When Nobody Owns The Measurement
Every experimentation practice eventually meets a boundary, and it is not a technical one: the metric you can move is not the outcome you want. Optimize hard enough against a single number and you will find local wins that damage the unmeasured whole — a conversion bump that raises churn, an engagement gain that erodes trust. The experiment was clean. The target was wrong. The discipline of choosing objectives that stand for the thing you actually care about (Google, 2016) was written for reliability targets and transfers exactly: a number is worth optimizing only while it remains a proxy for something you would defend out loud.
Underneath that sits a quieter and more expensive failure — measurement with no owner. Metrics rot the way any unowned system rots. A definition drifts, an event stops firing after a release, and a dashboard keeps rendering a number that has stopped describing the business, with nothing in the stack designed to announce that it is no longer true.
A dashboard that has stopped describing the system is more dangerous than no dashboard, because people are still deciding from it. This is not a hypothetical. The case for owning the AI systems you run tells it from a real engagement: a hedge fund whose storage bill and whose model quality had both been degrading unnoticed, where storage costs came down by more than 60% and the models performed 2% better once somebody was finally looking.
Nobody had hidden anything. Nobody had been looking — which is the same outcome with a longer fuse. Guardrail metrics catch the wins that are really losses only if somebody owns the guardrails, and the companion piece on measuring AI impact is where that ownership question gets its own treatment.
First Steps
- Make controlled experimentation a native capability rather than a retrospective argument. A concurrent comparison is the only thing that turns correlation into attribution, and no amount of dashboard sophistication substitutes for one.
- Move the measurement your decisions depend on to first-party collection on your own domain, because the third-party sample is now bent in a shape you cannot see from inside it.
- Decide in writing what your system does when it cannot measure cleanly. Serving the control and recording nothing is a legitimate answer, and it beats a number you will not be able to defend later.
Own What You Measure By
The durable approach owns both halves — the experiment and the metric — so that "what should we change" and "what did the last change earn" are answered by one first-party system rather than inferred from partial data gathered on someone else's terms. That is what lets a team learn something trustworthy from every change instead of accumulating dashboards nobody quite believes.
Standing that system up, though, is the easy half. A measurement stack is mostly not the clever part: research on hidden technical debt (NeurIPS, 2015) made the general case a decade ago — the interesting component is a small fraction of a real system and the surrounding machinery is the rest — and an experimentation layer is almost entirely surrounding machinery. Event definitions, bucketing, cookie signing, and guardrail metrics are live positions, each one correct only while somebody is checking that it still is.
The characteristic failure is therefore not an outage. It is a slow drift that surfaces as a confident decision made on a number that quietly stopped being true — the measurement-layer version of the spend drift the companion piece on cost reduction works through — and drift has no owner unless somebody is given the job. That job is what managed AI operations is: a senior owner on a defined scope of live systems at $12,000 a month — monitoring, cost control, incident response, and a written operating brief every month — cancellable on thirty days' notice, because ownership of your measurement should be earned monthly rather than locked in.
References
- Microsoft. Experimentation Platform. Microsoft Research.
- Wilander, J. Full Third-Party Cookie Blocking and More. WebKit, 2020.
- Anthropic. Building Effective Agents. Anthropic Engineering, 2024.
- Google. Service Level Objectives. Site Reliability Engineering, 2016.
- Sculley, D., et al. Hidden Technical Debt in Machine Learning Systems. NeurIPS, 2015.
