Strong language models agree with human preferences about which of two answers is better more than 80% of the time (Zheng et al., 2023) — roughly the rate at which two humans agree with each other. That is a real number, and it may be the most dangerous number in AI evaluation, because of what the same work names sitting right beside it: position bias, verbosity bias, and self-enhancement bias. The judge leans toward the answer that came first, the answer that ran longer, and the answer that sounds like something it would have written itself.
Agreement on a forced comparison is not a verdict on a piece of work anyway. Ask the open question — what do you think of this — and a different failure arrives: assistants tuned on human preference data drift toward the answer the user already believes (Sharma et al., 2023), and human raters sometimes prefer a convincingly written sycophantic response over a correct one. An underspecified question earns a diplomatic answer, and diplomacy is not something you can ship on.
GreatFeedback is the system ML LABS built against that gap, and it is live. Sixty built-in personas critique work across ten orthogonal lenses before launch; a widget on the shipped page collects reactions from real visitors after. It sits in the internal portfolio at Escape Velocity Labs, a sister practice under the same owner — ML LABS builds it, and ML LABS runs it. What follows is the architecture of a system that is running, not a proposal for one.
Score Dimensions, Not Vibes
The first design decision is that no reviewer is ever asked for an opinion. Work is scored against named dimensions, each carrying an explicit rubric, and a reviewer forced to rate accuracy, clarity, and persuasiveness as separate axes cannot return a blended shrug. The score is worth exactly what the rubric behind it is worth — which is why the rubric, not the model and not the prompt, is where the engineering goes.
- Orthogonal dimensions stop one strong quality from covering for a weak one inside a single blended number — the axes are chosen so they cannot trade against each other.
- A defined rubric per dimension hands the reviewer a fixed standard, which is what converts an opinion into a repeatable measurement.
- Custom dimensions let the work carry its own criteria: the ten lenses ship with the system, but the standard for this piece of work does not.
Sixty personas across ten lenses is not a feature count. It is a refusal to average. A skeptical buyer, a domain expert, and a first-time visitor reading the same page disagree in structured ways, and a mean taken over their scores destroys the only information worth having. Writing the standard down before measuring against it (Google, 2016) is the discipline that makes a service objective more than a dashboard, and it is the same discipline here: the target exists before the number, or the number means whatever the model wants it to mean.
A single reviewer returns an average. A panel built to disagree returns a map of where the work is contested.
Critique From Both Sides Of Launch
Pre-launch and post-launch feedback answer two different questions, and GreatFeedback ships both because neither one substitutes for the other. The persona panel says what is likely wrong before anyone has seen the work, instantly, and cheaply enough to justify running it on a draft. The widget says what is actually wrong, reported by people under no obligation to be kind. Synthetic critique has speed and no ground truth; real reaction has ground truth and no speed.
The artifact worth having is the gap between them. A predicted objection nobody raised, and a real objection no persona anticipated, are both calibration data about the critic — and a review system with no way to measure its own miss rate is a review system that should not be gating anything. This is the same split as the difference between output metrics and outcome metrics: the panel's scores describe the review, and the visitor's reaction describes whether the work landed.
That is why human ratings of the AI's feedback feed back into persona quality. The critic is itself under review, and the correction signal is a person marking a note as useful or as noise. An unrated critic is an unmonitored model: the finding that ML systems accrue hidden debt wherever monitoring and feedback are missing (NeurIPS, 2015) lands on the reviewer exactly as it lands on the thing being reviewed. Its rubric interpretation drifts, its calibration on your material decays, and the output reads as confident the entire way down. Managing model risk through traceability and measurement rather than trust (NIST, 2023) applies to the judge as much as to the judged, because the judge is a model in production too.
Where Rubrics Run Out
The rubric approach has a boundary, and it is worth naming out loud, because the score does not stop when it crosses one. Genuine originality, whether a joke lands, whether a design has any soul — these resist decomposition into scored axes, and forcing them in produces a confident measurement of the wrong thing. A reviewer will happily put a number on creativity, and the number will mean nothing at all.
The honest architecture uses the panel for everything a rubric covers well — clarity, correctness, structure, coverage, whether a claim survives a skeptic — and routes the irreducibly aesthetic judgment to a human who owns it and signs for it. A rubric is a floor under quality, never a ceiling on it. Sold as the ceiling, a scoring system eventually gates out the work that was worth shipping precisely because it did not fit an axis.
The boundary also moves. A lens that was orthogonal when the rubric was written stops being orthogonal once the work changes shape underneath it, and two dimensions that quietly measure the same thing will agree with each other forever — producing a score that looks stable and has stopped being informative. Auditing the panel for collapsed lenses is maintenance rather than setup, and a rubric that is never re-audited ages out of the work it grades. Naming the boundary out loud is what keeps the score trustworthy inside it.
First Steps
- Write the dimensions and their rubrics before you write a prompt. If you cannot say what a good score means on each axis, the reviewer will decide for you, and it will decide generously.
- Replace the single reviewer with a panel across lenses that genuinely differ, then read the disagreements before the consensus — the split is the finding.
- Rate the critic. Capture a human judgment on each piece of AI feedback, and route a real post-launch signal back against the pre-launch prediction, so miscalibration becomes visible instead of assumed away.
Build The Rubric, Then The Reviewer
The critique call is the cheap half of a review system. Ten lenses that do not collapse into each other, sixty personas that stay coherent under them, custom dimensions the work itself defines, and a human rating loop that corrects all of it — that is the half that is engineering, and it is the half a prompt cannot buy. The model underneath stays swappable, which is exactly what it should be: interchangeable, and bound to a standard it did not choose.
The moment this stops being a prompt and becomes a system is the moment the review has to gate a real decision — a ship, a spend, a release. Guidance on placing explicit checkpoints in front of consequential actions (Anthropic, 2024) applies to review as squarely as it applies to execution, and it is the same instinct that keeps the model on the surfaces while a deterministic core owns anything irreversible. A gate nobody can audit is a gate nobody will stand behind, which is the general condition everything it takes to run agents in production has to satisfy.
Anyone shipping AI-generated work at volume with no rubric behind the review is standing exactly where this pays. The next step is one contained workflow taken all the way to production: a production workflow build writes the acceptance targets into the SOW before work starts and includes its first 30 days of operation, run by the person who built it. The rubric and the correction loop are the deliverable. The model is the part you are free to change your mind about later.
References
- Zheng, L., et al. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. arXiv, 2023.
- Sharma, M., et al. Towards Understanding Sycophancy in Language Models. arXiv, 2023.
- Sculley, D., et al. Hidden Technical Debt in Machine Learning Systems. NeurIPS, 2015.
- National Institute of Standards and Technology. AI Risk Management Framework. NIST, 2023.
- Anthropic. Building Effective Agents. Anthropic Engineering, 2024.
- Google. Service Level Objectives. Site Reliability Engineering, 2016.
