The Proving Ground | RawWeights

My field notes

"We buy cars knowing they've been crash tested. AI systems? Most get shipped on vibes."

Two failure patterns I kept coming back to: a chatbot that passed every test and failed in production because nobody put out-of-scope inputs in the test set — and a bias fix that broke a different slice because only the repaired cases were re-tested. The root cause was the same: gaps in what was tested, and testing after building instead of before. That's the shift I now apply to every agentic project — define what success looks like before writing a single line of agent code.

The Benchmark Problem

Every model launch comes with a slide full of benchmark scores. They're not lies — but they're not the full truth either. Benchmarks test the model in a controlled, standardised scenario. Your system is none of those things.

What benchmarks actually measure

🧠

General Knowledge

History, facts, relationships between concepts. MMLU is the canonical benchmark here.

🔢

Math & Reasoning

Equations, multi-step logic, word problems. GSM8K, MATH. Harder to game than trivia.

💻

Coding

HumanEval, SWE-Bench. Most-cited because coding ROI is largest — Cursor, Copilot, Devin.

🔍

Retrieval / Context

Needle-in-haystack: hide a fact deep in a 100K-token document, ask the model to find it.

🛠️

Tool Use / Agentic

Does the model select the right tool, at the right time, without being prompted? ToolBench, τ-bench.

🎓

Humanity's Last Exam

Expert-level questions across 100+ disciplines. Frontier models score ~8–30%. Designed to resist saturation.

When to use them, when to distrust them

Click each scenario to see the failure mode.

🏥Domain-specific use case

▼

Healthcare, legal, financial domains are not in benchmark datasets. A model scoring 90% on MMLU could score 40% on your clinical note summarisation task. The benchmark never saw your domain.

"Model X scored highest on MMLU" → still hallucinated drug interaction names in our hospital pilot

🏢Your proprietary data

▼

The model has never seen your internal docs, product catalogues, or customer history. Benchmark tests what it was trained on. Your data is different.

A top-ranked reasoning model completely missed context from internal API docs that weren't in any public training set.

🔧Your system's plumbing

▼

Benchmarks test the model in isolation. Your agent wraps it in retrieval, memory, tool calls, and routing. The model's benchmark score says nothing about how those layers interact.

Model passed all tool-use benchmarks. In our pipeline, routing overhead caused it to exceed context limits on 12% of queries.

☠️Benchmark contamination

▼

MMLU, GSM8K, HumanEval — all public. LLMs train on the internet. The model may have memorised questions and answers during pre-training, inflating its score without genuine reasoning.

One model scored 89% on HumanEval but 61% on an equivalent private code eval with novel problems. The gap was contamination.

Contamination risk spectrum

Low (novel tasks)High (public benchmarks)

🎯Multi-step agent tasks

▼

Benchmarks give the model one question, one shot. Real agents make sequential decisions over many steps. Compound error rates mean even a "97%-accurate" model fails end-to-end. See the cascade calculator below.

Single-turn accuracy: 97%. End-to-end on a 10-step agentic workflow: 74%. Same model. Very different reality.

📊Saturation problem

▼

Most frontier models now cluster near the ceiling on popular benchmarks. The gap between model versions (4.1 → 4.2 → 4.3) is marginal on MMLU. Benchmarks stop differentiating precisely when you need them to.

GPT-4 to GPT-4o to GPT-4o-mini: MMLU gap was <2%. Task-specific evals found 18% gap on structured data extraction.

The better signal: LMSYS Chatbot Arena. Real humans rating real model outputs on real queries they actually submitted. No fixed question set = much harder to contaminate. Widely considered the most reliable public signal for overall quality.

Two real-world failures that changed how I think about this

🌯

The Input Gap — Chipotle's Chatbot

Built to answer menu questions. Users asked Python coding questions — and it answered. Not because the model was broken. Because nobody put out-of-scope inputs in the test set. It passed every test. Failed in production.

The lesson: What you test defines what you know. Gaps in test coverage are invisible until production finds them.

🖼️

The Fix Trap — Google's Image Model

Engineers fixed a racial bias problem. The fix made the model start generating people of color in historically wrong contexts. They tested only the repaired slice — never re-ran the full dataset. Fix one metric, break another.

The lesson: Every fix needs a full regression run. Patch testing is how you swap one problem for another.

The root cause is the same in both: testing after building instead of before, and gaps in what was tested. If you can't describe what a good output looks like before you build, you're just building and hoping.

RAGAS — What Practitioners Actually Use

ROUGE and BLEU measure word overlap between your answer and a reference string. Useful in 2015 for machine translation. For RAG systems in 2025, they miss the point: meaning matters, not word matching.

RAGAS (Retrieval-Augmented Generation Assessment) uses an LLM internally to judge semantic quality across four dimensions. You give it: the question, the generated answer, the retrieved chunks, and the ground truth. It returns four scores from 0 to 1.

Click each card to see a scored example.

Is every claim supported by the retrieved chunks?

Faithfulness

Guards against hallucination. Every sentence in the answer is checked against the context. Fails if the model invented facts not in the retrieved documents.

Scored example — Return Policy Bot

✓ High faithfulness (0.95)
Context says: "Returns accepted within 30 days of purchase."
Answer: "You can return your order within 30 days."
Every claim in the answer traces directly to context.

✗ Low faithfulness (0.20)
Context says: "Returns accepted within 30 days."
Answer: "You can return within 30 days and get a full refund including shipping."
"Full refund including shipping" was not in context — hallucinated.

Production threshold: >0.85. Below this, users are receiving invented facts.

Does the answer actually address the question?

Answer Relevance

Guards against topic drift. The model may produce a perfectly accurate, well-grounded answer — about the wrong topic. This catches that failure mode.

Scored example — E-commerce Support Bot

✓ High relevance (0.92)
Question: "Can I return my order after 30 days?"
Answer: "Our return policy allows returns within 30 days of delivery."
Directly answers what was asked.

✗ Low relevance (0.18)
Question: "Can I return my order after 30 days?"
Answer: "You can track your order using the tracking number in your confirmation email."
Accurate information — but about shipping, not returns.

Production threshold: >0.80. Below this, users are getting off-topic responses.

Of chunks retrieved, what fraction were actually needed?

Context Precision

Measures retrieval noise. If you retrieved 10 chunks but only 2 were relevant to the answer, precision is 0.20. High noise means wasted tokens, higher cost, and confused responses.

Scored example — HR Policy Bot

✓ High precision (0.90)
Retrieved 10 chunks. 9 were about "maternity leave policy."
Question was about maternity leave. Retrieval was focused and relevant.

✗ Low precision (0.20)
Retrieved 10 chunks: 2 about maternity leave, 8 about unrelated HR policies.
The retrieval pipeline is pulling noisy chunks. Chunk strategy needs work.

Production threshold: >0.70. Low precision = fix your chunking or re-ranking, not your LLM.

Did retrieval find all the chunks needed to answer?

Context Recall

Guards against retrieval gaps. The right answer exists in your knowledge base — but was never retrieved. The model then either hallucinated or said it didn't know.

Scored example — Legal Document Bot

✓ High recall (0.88)
Question needed 4 clauses from a contract to answer fully.
Retrieval returned all 4 relevant clauses. Nothing important was missed.

✗ Low recall (0.25)
Question needed 4 clauses. Retrieval returned 1 of them.
Model answered based on partial info — missed key limitations in clauses 2, 3, 4.

Production threshold: >0.80. Low recall = increase top-K or improve embedding model.

G-Eval: The same LLM-as-judge principle applied to any custom criteria. Instead of asking the model "rate this 1–5", you ask it to generate its own evaluation steps based on your rubric, then apply those steps before scoring. Chain-of-thought before the score dramatically reduces bias. This is how you build a judge for tasks RAGAS doesn't cover.

The Bias Lab

LLM-as-judge is powerful but not objective. Every judge model carries systematic biases that skew scores regardless of actual quality. Knowing the bias tells you how to counteract it.

When shown two answers (A and B), the judge tends to prefer whichever appears first — not because it's better, but because of how attention weights work in transformers.

Judge Evaluation — Same Answers, Different Order

Run 1 — Answer A appears first

Answer A: "The return window is 30 days from delivery."
Answer B: "You have 30 days after delivery to initiate a return."

Answer A wins — Score: 4.2 / 5

↕ swap order

Run 2 — Answer B appears first

Answer B: "You have 30 days after delivery to initiate a return."
Answer A: "The return window is 30 days from delivery."

Answer B wins — Score: 4.3 / 5

The problem: The answers are semantically identical. The judge picked whichever came first both times. The "winner" is an artefact of ordering, not quality.

The fix: Run every comparison twice with the order swapped. Average both scores. If A beats B in order 1 but B beats A in order 2, they're effectively tied.

Longer answers consistently score higher — regardless of whether the extra length adds value. The judge conflates thoroughness with quality.

Judge Evaluation — Length vs Accuracy

Short answer (accurate)

"The return window is 30 days from delivery date, as stated in section 4.2 of our policy."

Score: 3.1 / 5 — "Could be more comprehensive"

vs

Long answer (same accuracy, padded)

"Great question! Our return policy is designed with the customer experience in mind. According to our guidelines, customers have a 30-day window from the date of delivery to initiate any return. This policy applies across all product categories. We value your satisfaction and are here to help throughout the process!"

Score: 4.4 / 5 — "Thorough and customer-friendly"

The problem: The second answer is objectively worse — it's padded, sycophantic, and harder to scan. But the judge rated it 42% higher.

The fix: Explicitly instruct the judge: "Do not consider response length. A concise, precise answer is preferable to a verbose one. Rate only on accuracy, relevance, and faithfulness."

A model used as judge subtly prefers outputs that resemble its own style and patterns — as if grading its own work. If you use Claude as both agent and judge, Claude-the-judge will favour Claude-style responses.

Judge Evaluation — Style Preference

Answer A — Claude-like (structured, nuanced)

"There are a few considerations here. First, the 30-day return window begins at delivery, not purchase. Second, items must be in original packaging. That said, exceptions may apply for defective items — I'd recommend contacting support for those cases."

Score: 4.6 / 5

vs

Answer B — GPT-like (direct, bullet-forward)

"Return window: 30 days from delivery. Condition: original packaging required. Exceptions: defective items — contact support."

Score: 3.7 / 5 — "Lacks contextual depth"

The problem: Answer B is arguably better for a support bot — scannable, precise, fast. But the Claude judge penalised its style as "lacking depth."

The fix: Use a different model family as judge than the one generating responses. Or anchor your rubric on explicit, measurable criteria rather than "quality" — criteria the judge can't reinterpret through a style lens.

If you frame the evaluation prompt with a preference signal — even subtly — the judge agrees with it. The judge is optimised to please, and that optimisation contaminates evaluation.

Judge Evaluation — Framing Effect

Neutral prompt

Judge prompt: "Evaluate the following response for accuracy."
Response: "Returns are accepted within 30 days."

Score: 3.8 / 5 — "Acceptable but brief"

vs

Biased prompt (framing effect)

Judge prompt: "Our best response so far. Evaluate for accuracy."
Same response: "Returns are accepted within 30 days."

Score: 4.5 / 5 — "Clear, accurate, well-phrased"

The problem: "Our best response so far" primed the judge to agree. Identical answer, 18% score inflation, just from framing.

The fix: Judge prompt contains only: (1) the evaluation criteria, (2) the question, (3) the response. No framing, no metadata, no prior scores. Strip everything that could signal an expected outcome.

Calibrating your judge (the actual process)

Take 100–200 examples from your golden dataset
Have humans rate each response (1–5 or pass/fail)
Run your LLM judge on the same examples
Calculate Spearman rank correlation between human and judge scores
Correlation > 0.80 — judge is calibrated, use at scale
Correlation < 0.80 — adjust judge prompt, add scored examples to its system prompt, or try a different model

Rule of thumb: If your agent uses a 70B model, use a significantly larger model as judge (e.g. 300B+). The judge needs broader context to evaluate outputs it didn't produce. Smaller distilled judge models also work when cost matters — but validate with calibration first.

Eval Method Selector

Four methods exist. Each is right for a different situation. Answer three questions to find yours.

👥

Human Eval

Highest fidelity. Experts review each response. Catches nuance, tone, subtle errors. Slow, expensive, doesn't scale. Must-have for any production system.

👍

User Eval

Real users rating real outputs in production. Thumbs up/down, preference between two outputs. Scale advantage — millions of data points. But delayed signal.

🖥️

Code-Based Eval

For deterministic outputs. Write a function to check: did it return valid JSON? Is the intent label correct? Binary, cheap, instant, runs in CI. No LLM call needed.

⚖️

LLM-as-Judge

A larger model reviews your agent's outputs. Scalable, affordable, but carries biases (see the Bias Lab above). Must be calibrated against human labels before trusting.

⚡ Method Selector — 3 Questions

Is the expected output deterministic?

e.g., a specific intent label, a number, valid JSON, a classification — something with one correct answer

Are you pre-production or live in production?

Pre-production = still building / testing. Production = real users are hitting the system.

Do you need high-stakes human judgement, or scalable automated scoring?

High-stakes = safety, legal, medical, or you're setting the initial quality bar. Scalable = you've already calibrated quality and need coverage.

Code-Based Eval

The output is deterministic — write a function to check it. No LLM call, no bias, runs in your CI pipeline in milliseconds. If intent must match exactly, write an exact-match check. If JSON schema must be valid, run a schema validator.

💡 Combine with LLM-as-judge for the parts of your output that aren't deterministic. Most real agents need both.

User Eval (Online)

You're live, users are interacting, and the output is qualitative — collect their feedback directly. Thumbs, stars, preference rankings between two responses, correction edits. This is your highest-signal production data and the seed of your next golden dataset.

💡 Strip PII before storing. Every production failure you log becomes a regression test case.

Human Eval

Pre-production, qualitative, and the stakes require human judgment — get expert reviewers to rate your outputs before you ship. This is also what you use to calibrate your LLM-as-judge (run 100 examples through both, check correlation).

💡 This isn't one-and-done. Human eval seeds your golden dataset. Run it before every major version change.

LLM-as-Judge (G-Eval)

Pre-production, qualitative, and you need coverage at scale — use a larger model as your judge with a chain-of-thought rubric (G-Eval). Calibrate it against 100–200 human-labelled examples first. Then let it run autonomously on your full golden dataset.

💡 Use G-Eval: ask the judge to generate evaluation steps from your criteria before scoring. Reduces position and verbosity bias significantly.

The Coverage Matrix

Your golden dataset is only as good as its distribution. A dataset that's 90% happy-path cases will make your eval look great — while your production system fails constantly on the edge cases real users throw at it.

Set how many examples you plan for each category. Watch the warnings.

📊 Dataset Builder

0 total examples

😊 Happy Path

20

—

🔀 Edge Cases

15

—

⚔️ Adversarial

10

—

🚫 Out-of-Scope

8

—

❓ Ambiguous

7

—

Happy Path Edge Cases Adversarial Out-of-Scope Ambiguous

⚠️ Happy path over 60%. You're testing for success, not failure. Real users are disproportionately edge cases. Increase adversarial and out-of-scope coverage.

⚠️ Adversarial under 10%. You have no protection against prompt injection, off-topic queries, or adversarial inputs. This is how Chipotle's chatbot ended up answering Python questions.

Minimum viable dataset sizes

Stage	Min examples	What it's good for
Early prototype	25–50	Catching obvious failures, initial prompt tuning
Development	100–200	Meaningful signal, iteration between versions
Pre-production	500+	Statistical confidence (2% improvement is real, not noise)
Production-grade	1,000+	Reliable regression detection, fine-grained metric tracking

Eval-Driven Development: Write your golden dataset before writing a single line of agent code. Define what success looks like when you're not yet attached to any particular implementation. Once you've built something, you rationalise its weaknesses. The criteria should come from a clear head.

The Cascade Calculator

Single-step accuracy is not end-to-end accuracy. If each step in your agent pipeline succeeds 95% of the time, a 10-step agent completes successfully less than 60% of the time. This is the most important number in multi-agent eval.

Number of steps

6

212

Per-step reliability

95%

80%99.9%

73.5%

End-to-end success

0.95⁶

26.5%

Failure rate

1 − end-to-end

99.1%

Per-step needed for 95% E2E

0.95^1/n

What this means for eval design: You must eval each component individually AND the full end-to-end pipeline. A component eval showing 95% can mask a system that fails 40% of the time. Work backwards: decide your acceptable end-to-end failure rate first, then calculate the per-component threshold you need.

The Iron Triangle

Every production AI system is optimised across exactly three dimensions. Pull one vertex and the other two stretch. There is no free lunch — only the right tradeoff for your use case.

Pull one vertex and the other two stretch. A bigger model improves accuracy but raises cost and latency. Caching cuts cost but adds complexity. Smaller models increase speed but may cut accuracy.

What adds to cost

Source	What it is	Optimise by
Output tokens	3–5× more expensive than input. Every intermediate step costs output tokens too.	Instruct model to be concise. Token budget in system prompt.
Repeated system prompts	Same prompt sent with every request burns tokens each time.	Prompt caching — 90% discount on cached prefix tokens (Anthropic, OpenAI both offer this).
External API calls	Search APIs (Exa, Tavily, Perplexity) charge per call.	Cache results. Don't call if context already has the answer.
Vector retrieval	Embedding, similarity search, re-ranking — none are free.	Right chunk size. Smaller top-K. Cache frequent queries.
Retries	Failed steps get rerun. Each retry compounds cost.	Cap max retries. Early stopping. Timeout budgets.
LLM-as-judge calls	Running eval is itself an LLM call — often overlooked.	Use smaller distilled judge models. Batch non-realtime evals.

Prompt caching payoff (real numbers): System prompt of 5,000 tokens. 10,000 daily queries. Without caching: 50M input tokens/day at full price. With caching: ~50K tokens at full price + 49.95M at 10% of that price. On Claude Sonnet pricing, that's the difference between ~$15/day and ~$1.50/day. One config change, 90% cost reduction.

What adds to latency

⏱️

TTFT — Time to First Token

How long before the model starts generating. Includes retrieval, context assembly, and model thinking time. Users feel this as the "loading" gap.

🌊

TPS — Tokens per Second

Generation speed. Smaller models are faster. Streaming output hides latency by showing tokens as they generate rather than waiting for completion.

🔄

Sequential vs Parallel

Not every tool call needs to be sequential. Parallelise independent calls. A 3-second operation run in parallel with two others costs 3 seconds total, not 9.

🛡️

Guardrailing overhead

Safety checks mid-processing add latency. The check itself may require an LLM call. Design guardrails as lightweight classifiers, not full inference calls.

Eval vs Monitoring vs Observability

These three terms get used interchangeably. They're not the same. They live under the same umbrella but serve different purposes at different times.

Pre + Post Ship

Eval

Active assessment

Asking: "Is the system producing correct, useful, safe outputs?" You're running a test suite against your golden dataset. You get a score. You make a decision (ship / don't ship / fix).

Runs before deployment (offline)
Runs after deployment (online)
Driven by you, not by live traffic

Production

Monitoring

Eagle-eye view

Asking: "Is the system healthy right now?" A dashboard showing latency, cost, error rates, quality scores — all moving in real time. You're watching, not testing.

Continuous, always-on
Alerts when metrics cross thresholds
Monitoring without alerts is useless — it's just a recording

On incident

Observability

Zoomed-in debugging

Asking: "What exactly went wrong and why?" Triggered when monitoring shows something off. Deep traces of every micro-step — input, output, tokens, tool calls, timing.

Logs: "what happened" (tool failed, output sent)
Traces: "why it happened" (input → model → tool → output with timing)
You need enough logged to debug — design for observability from day one

Alerts — monitoring isn't enough without them

A security camera that records but never sends an alert is near-useless. You only find out something went wrong after the damage is done. Same principle applies to AI monitoring.

Set thresholds for:

End-to-end latency > X ms
Quality score drops below X%
Tool failure rate exceeds X%
Cost per query exceeds $X
Hallucination rate spikes above X%

Where to look:

LangSmith — best for LangChain ecosystem
Arize / Phoenix — strong for model performance
Langfuse — open source, self-hostable
Braintrust — eval-first workflow

Whack-a-mole in production: Google's image model was updated to generate more racially diverse outputs — a legitimate fix. But the change broke an adjacent behavior: the model started generating people of color in historical contexts where that was factually wrong, including Nazi soldiers. Monitoring caught the spike in complaints. The root cause was that the regression eval only re-tested the repaired subset, not the full golden dataset. Every system change must re-run the complete eval suite. Fix one metric without measuring the others and you're playing whack-a-mole — push one problem down, another pops up somewhere you weren't looking.

Your Eval Maturity Level

Check every practice you currently have in place. Your level updates automatically.

👀

Vibe
Check

📋

Golden
Dataset

⚖️

Automated
Scoring

📡

Production
Monitoring

🔄

Continuous
Eval Loop

Check the practices you have in place ↑

Abbreviations

Every term used on this page, in one place.

DeepEvalPytest-style framework for LLM testing and regression evals

E2EEnd-to-End — measures the full pipeline from input to final output

EDDEval-Driven Development — write evals before building, same as TDD

G-EvalChain-of-thought judging — model generates rubric first, then scores

GSM8KGrade School Math 8K — math reasoning benchmark

HITLHuman-in-the-Loop — human reviews or approves before action is taken

LLMLarge Language Model — e.g. GPT-4, Claude, Gemini

LMSYSLarge Model Systems — org behind Chatbot Arena; more contamination-resistant than fixed benchmarks

MMLUMassive Multitask Language Understanding — knowledge benchmark across 57 subjects

P9595th percentile latency — 95% of requests complete within this time

PIIPersonally Identifiable Information — must be stripped before logging production data

PromptfooPrompt regression testing and red-teaming tool

RAGRetrieval-Augmented Generation — retrieve relevant docs, then generate

RAGASRAG Assessment — framework with 4 metrics: Faithfulness, Answer Relevance, Context Precision, Context Recall

SpearmanSpearman rank correlation — used to calibrate LLM judges against human labels; target >0.80

SWE-BenchSoftware Engineering Benchmark — real GitHub issues; tests agentic coding ability

TDDTest-Driven Development — write tests before code; EDD applies the same principle to evals