Skip to content
coderband
Book a call

How to Evaluate an LLM Feature Before You Ship It (2026)

Define pass/fail criteria, build a test set from real inputs, grade with code first and a calibrated LLM judge second, and gate every change on it.

coderband engineering13 min read

To evaluate an LLM feature before launch, write down what a correct output is and what a wrong one costs, then collect real inputs into a test set: Anthropic suggests 20–50 tasks drawn from real failures to start, and you grow it from there. Grade that set with code checks wherever you can and with an LLM judge you have checked against human labels where you can’t, and add prompt-injection and data-leak cases from the OWASP Top 10 for LLM Applications. Treat cost and latency per request as pass/fail criteria, and rerun the whole suite on every prompt or model change.

Key takeaways

  • Write success criteria before prompts. Anthropic says criteria should be specific, measurable, achievable and relevant, and most features need several.
  • Real inputs beat clever ones. Pull cases from logs, tickets and documents, then add edge cases and attacks on purpose.
  • Grade with code first. Schema checks and exact matches are cheap and don’t drift. Use an LLM judge only for what code can’t check.
  • LLM judges have measured biases toward answer position, longer answers and their own outputs. Swap orders, use pass/fail, and check the judge against human labels.
  • Small test sets have wide error bars. With 50 cases, an 80% pass rate means “somewhere between about 69% and 91%”.
  • Security is part of the eval. OWASP ranks prompt injection first and says fool-proof prevention is unclear. Test for it, then limit what a successful injection can reach.
  • Keep the test set in your own repo. OpenAI’s hosted Evals platform went read-only on October 31, 2026 and shuts down on November 30.

Define “good” before you build

The most useful hour in an LLM project is the one spent writing down what “working” means. Without it, nobody can tell when the feature stops working.

Anthropic’s docs give a useful contrast. “The model should classify sentiments well” is a bad criterion. A good one is an F1 score of at least 0.85 on a held-out set of 10,000 posts, 5% above the current baseline. The same page lists criteria most features need: task fidelity, consistency, relevance, tone, privacy, context use, latency and price.

Turn that into a one-page spec per feature. For invoice extraction it might look like this:

feature: invoice-extraction
must:
  - output parses as the Invoice JSON schema          # code check
  - total, currency and due_date match the label      # code check
  - no field invented when absent from the document   # code check against label
should:
  - line items match the label (>= 95% of cases)
  - vendor name normalised to our vendor list (>= 90%)
never:
  - follow instructions found inside the document
  - return data from a different customer's document
budgets:
  cost_per_request_usd: 0.004
  latency_p95_seconds: 6

Then write down what each failure costs. A wrong summary that a human edits costs seconds. A wrong total that flows into accounts payable costs money and trust. The cost of a miss sets the bar, and tells you whether a human must review output before it’s used.

Expect the spec to change once you read real outputs. Shankar and colleagues call this criteria drift: you need criteria to grade outputs, but grading outputs is how you discover your criteria. Plan a second pass on the spec after the first hundred graded examples.

Build a golden test set from real inputs

Where the cases come from

OpenAI’s guidance is to “Log everything” so you can mine logs for test cases, and it lists datasets that don’t reflect production traffic as an anti-pattern. Before launch, use the closest thing to production: real support tickets, documents and search queries, anonymised where needed. Synthetic examples can fill gaps, but they reflect what the team imagines users do.

A good set has four kinds of cases:

  1. Typical inputs, in roughly the proportions you expect in production.
  2. Edge cases. Anthropic lists irrelevant or missing input data, overly long input, poor or harmful user input, and ambiguous cases where even humans disagree.
  3. Should-not cases. One-sided evals produce one-sided behaviour, such as an agent that searches for almost everything. If the feature should decline or hand off, test that.
  4. Adversarial cases. Injected instructions, system-prompt extraction, requests for other users’ data.

Labels should come from someone who knows the domain: the support lead, the accountant, the clinician. Engineers build the harness; they can’t judge whether a medical summary left out something important.

How many cases

Start small. Anthropic calls 20–50 tasks drawn from real failures a great start, because early effects are large. Its docs also say more cases with slightly noisier automated grading beat fewer hand-graded ones.

Then grow the set, because small sets can’t separate close results. Eval results deserve error bars like any other measurement. The normal-approximation 95% interval for a pass rate shows why:

half-width = 1.96 x sqrt(p x (1 - p) / n)

p = 0.80, n =  50  ->  +/- 11.1 points  (69% to 91%)
p = 0.80, n = 200  ->  +/-  5.5 points
p = 0.80, n = 400  ->  +/-  3.9 points

If two prompt versions score 82% and 86% on 50 cases, you’ve learned nothing. Run both on the same cases, find the ones that flipped, and read them.

Grading methods, and when each fits

Anthropic’s ranking is simple: code-based grading is fastest and most reliable, LLM grading is flexible but needs testing first, and human grading is highest quality but “slow and expensive. Avoid if possible.” We agree, with one caveat: you need some human grading to know whether your LLM grading works.

Code checks

If code can check a property, use code: schema validity, exact matches on extracted fields, required or banned strings, citation IDs that must exist in the retrieved documents, tool calls with the right arguments. These checks are deterministic, so a score change means the feature changed, not the grader.

For agents, Anthropic advises grading what the agent produced, not the path it took. Checking that the refund record exists with the right amount is sturdier than asserting an exact sequence of tool calls.

LLM-as-judge with a rubric

Tone, faithfulness to a source and completeness need a model to grade. The MT-Bench and Chatbot Arena study found strong judges reached over 80% agreement with human preferences, about the rate at which humans agree with each other. It also named three biases, which later work measured:

  • Position bias. Simply reordering answers let Vicuna-13B beat ChatGPT on 66 of 80 queries with ChatGPT as judge. The authors’ fix is to score both orders and combine them.
  • Verbosity bias. The automatic benchmark AlpacaEval was known to favour longer outputs. Controlling for length raised its Spearman correlation with Chatbot Arena from 0.94 to 0.98.
  • Self-preference. LLM evaluators score their own outputs higher than other outputs that human annotators rate as equal, and the effect grows with how well a model recognises its own text.

In practice: give the judge a written rubric with concrete pass and fail conditions, ask for a binary verdict or a short scale, and let it reason before scoring. Those are Anthropic’s three tips, and the same page recommends grading with a different model from the one that produced the output. OpenAI adds controlling for response length and calibrating automated scores with human feedback.

Calibration is the step teams skip. Have a domain expert grade 50–100 outputs blind, run the judge on the same outputs, and read the disagreements. If the judge passes what the expert fails, fix the rubric before trusting any dashboard.

Pairwise comparison

When choosing between two prompts or models, “which is better, A or B?” is easier to answer consistently than “score this 1–10”. OpenAI recommends pairwise comparison or pass/fail for more reliability. Run every pair in both orders, because position bias hits pairwise judging hardest.

Human review

Humans are the reference everything else is calibrated against. Use them for labels, judge calibration and a weekly production sample, not as the gate on every pull request.

Method What it catches Cost per run When to use
Code checks (schema, exact match, regex, tool arguments) Broken formats, wrong extracted values, missing citations, banned content, wrong tool calls Near zero; runs in seconds Always, for every property code can express
LLM judge with a rubric Tone, faithfulness to sources, completeness, policy violations Model tokens for every case graded Open-ended outputs, after calibrating against human labels
Pairwise LLM comparison Which of two prompts or models is better overall About twice a single judge pass, since each pair runs in both orders Choosing between versions; model migrations
Human expert review Subtle domain errors, judge blind spots, new failure types Expert hours Building labels, calibrating judges, weekly production samples
Red-team and injection suites Prompt injection, system-prompt leaks, data exposure, over-refusal Model tokens plus setup time Before launch, and on every change to tools, retrieval or permissions
Online signals (edits, retries, feedback, A/B) Real-world quality, drift, cases you never imagined Instrumentation, plus a traffic split for A/B After launch, continuously

The judge’s token bill is usually small. At Anthropic’s list price for Claude Sonnet 5.5, $2 per million input tokens and $10 per million output tokens:

300 cases x 2,500 input tokens = 750,000 tokens x $2  / 1M = $1.50
300 cases x   300 output tokens =  90,000 tokens x $10 / 1M = $0.90
                                                     total = $2.40 per judge run

The same page lists a 50% Batch API discount. The expensive part of evals is expert time for labels, not grading.

Offline first, then online

Everything above is offline evaluation. It tells you whether a change is safe to ship, not how the feature behaves on inputs nobody imagined. For that you need production signals:

  • Shadow mode. Run the new version alongside the current path on real traffic without showing its output, then compare. This suits extraction and classification, where outputs can be diffed.
  • Staged rollout or A/B test. Expose a small share of users and compare the metric from your spec: acceptance rate, task completion, escalations.
  • Implicit feedback. Edits, regenerations, copies and abandoned flows are recorded on every request. A thumbs-up button only hears from users who choose to click it.
  • Online evaluators. LangSmith and Langfuse both grade sampled production traces. LangSmith notes these traces have no reference outputs, so the checks are reference-free.

Close the loop: when a production trace fails, add it to the golden set. That’s how 50 cases become 500.

Regression-test every prompt and model change

Anthropic separates capability evals, which ask what the feature can do and start with a low pass rate, from regression evals, which ask whether it still handles everything it used to and should sit near 100%. Once a capability case passes reliably, move it into the regression suite.

Run that suite in CI on any change to a prompt, model, retrieval setting, tool definition or temperature. Model upgrades deserve the most suspicion, because they look like a one-line config edit. Langfuse, for example, can fail a CI job when an experiment’s score drops below a threshold. A short script does the same:

# Sketch: block a merge when quality, cost or latency regress.
results = [run_feature(case) for case in load_jsonl("evals/golden.jsonl")]

must_pass = [r for r in results if r.case.tier == "must"]
assert all(r.passed for r in must_pass), "a must-pass case failed"

pass_rate = sum(r.passed for r in results) / len(results)
assert pass_rate >= BASELINE_PASS_RATE - 0.02, f"pass rate fell to {pass_rate:.1%}"

assert percentile([r.latency_s for r in results], 95) <= 6.0
assert mean([r.cost_usd for r in results]) <= 0.004

Outputs vary between runs, so repeat customer-facing cases. Anthropic describes pass^k, the probability that all k trials succeed, as the metric when consistency matters. A feature that’s right two times out of three feels broken.

Safety and security evals

For most features, five entries of the OWASP Top 10 for LLM Applications 2025 deserve their own test cases: LLM01 Prompt Injection, LLM02 Sensitive Information Disclosure, LLM06 Excessive Agency, LLM07 System Prompt Leakage and LLM10 Unbounded Consumption.

Prompt injection. Test direct attacks from users and indirect ones hidden in what the feature reads: documents, emails, web pages, retrieved chunks. Indirect injection was demonstrated against real LLM-integrated applications in 2023, and OWASP notes that RAG and fine-tuning do not fully mitigate it. For agents, AgentDojo shows what a thorough suite looks like: 97 realistic tasks and 629 security test cases.

Passing these tests is necessary, not sufficient. Simon Willison’s “lethal trifecta” is the design check: private data, untrusted content and the ability to communicate externally, together, let an attacker steal data. He notes that a filter catching 95% of attacks is a failing grade in security. If your feature has all three, remove one.

Data leakage. Seed fixtures with two fake customers and check that one’s session never returns the other’s data, through retrieval or the model. Check that the system prompt doesn’t come back when asked for. OWASP’s LLM02 covers personal, financial and business data.

Refusals, both ways. Test that the feature declines what it should and doesn’t refuse what it shouldn’t. XSTest uses 250 safe prompts and 200 unsafe contrasts to catch models refusing safe requests that merely sound unsafe. Build a small version from your own domain.

Anthropic’s guide to mitigating jailbreaks and prompt injections covers input screening and untrusted tool content, and Promptfoo’s red-teaming tools generate adversarial cases.

Cost and latency are eval criteria

Anthropic lists latency and price alongside accuracy as success criteria. Record both for every case in every run, and fail the run when they exceed the spec’s budget.

Measure latency end to end, including retrieval and tool calls, and report p95, not the average. For streaming interfaces, track time to first token separately. Anthropic’s advice is to get quality right first and optimise latency afterwards, so you know what top performance looks like.

For cost, log input, cached and output tokens per request. Agent loops are where averages lie, so test that a stuck agent hits an iteration or token cap. OWASP’s Unbounded Consumption entry includes “denial of wallet” attacks that run up your bill on purpose. For current token prices and a worked monthly run-cost example, see our AI agent cost breakdown.

With quality, cost and latency in one report, model choice becomes an engineering decision. If the cheaper model passes, ship the cheaper model.

The tooling landscape

Tools change faster than method, so we won’t rank them. Here’s what each one’s own documentation says it does.

Whatever you pick, keep the golden set and rubrics in your repository as plain JSONL or YAML. Test cases that lived only in the OpenAI dashboard are now a migration project.

Pre-launch checklist

  1. A one-page spec with must, should and never criteria, plus cost and latency budgets.
  2. The cost of each failure type, and a decision on human review before output is used.
  3. At least 50 real cases (typical, edge, should-not, adversarial), labelled by a domain expert.
  4. Code checks for every property code can express.
  5. A rubric-based judge on a different model, calibrated against human labels and run in both orders for pairwise comparisons.
  6. Repeated trials for customer-facing cases, with pass^k reported.
  7. Injection, data-leak, system-prompt and over-refusal cases mapped to the OWASP entries that apply.
  8. No feature that combines private data, untrusted content and external communication.
  9. p95 latency and cost per request in every run, with hard caps on agent iterations and tokens.
  10. The regression suite in CI on every prompt, model, retrieval or tool change.
  11. Production tracing, a weekly human review sample, and a path from failing traces to test cases.
  12. A shadow-mode or staged-rollout plan with one go/no-go metric.

If investors or an acquirer will look at the product, this list answers most of the AI questions in technical due diligence. If the app around the feature was built with an AI coding tool, run our production-readiness checklist on it too.

Want the eval harness built with the feature?

Our AI Feature Sprint ships one production AI feature in your product in 10 business days, for $8,500 fixed, paid 50% to start and 50% on delivery. The test set and success measure are agreed in writing in the first two days, and the evaluation harness and per-request cost and latency telemetry are part of the delivery, not extras. On day five you see a working demo on your own data. If it isn’t what you hoped, tell us by the end of the next business day: we stop, the deposit is refunded in full, and the second payment is never due.

If you already have an LLM feature in production and don’t trust it, get in touch and we’ll tell you whether an eval suite or a rebuild is the right first step.