How to Evaluate LLM Outputs: Metrics, Rubrics, and Human Review

A prompt that works on ten examples will still surprise you on the eleventh. That gap between the demo and the thing you actually run is where evaluation lives.

Three tools do most of the work. Automated metrics are cheap, fast and narrow. A language model acting as judge scores at a volume no person can match, while carrying biases you have to design around. Human reviewers with a written rubric are slow and expensive, and they are the only ones who can tell you whether an output was genuinely useful. Used together, the three cover each other's blind spots.

Decide what "good" means before you measure anything

Write the job down in one sentence. "Summarise a support ticket in under 60 words for an agent scanning a queue, keeping the order number and the requested resolution." Now list the ways that could go wrong: the order number is wrong, the customer's name is invented, the resolution is missing, the tone blames the customer.

That list is your evaluation plan. Every metric you add later should trace back to one of those failure modes. If a metric does not map to a failure you actually care about, it is decoration, and decoration costs money and attention.

Automated metrics are gates, not grades

String-overlap scores like BLEU and ROUGE, embedding similarity, exact match — these are blunt instruments. A perfectly good paraphrase can score low. A fluent, confident, wrong answer can score high. Use them where they are reliable:

• Format and schema validity: does the output parse as JSON, match the required fields, respect length limits?
• Deterministic checks: does the quoted order number appear in the source? Does the generated SQL run?
• Operational numbers: latency, tokens per request, refusal rate, cost per thousand calls.
• Regression tracking: has similarity to the previous version dropped sharply on your frozen eval set?

None of these tell you whether an answer is good. They tell you whether something broke. That is a valuable, narrow job — let them do it and nothing more.

LLM-as-judge: volume with caveats

A judge model reads the input, the output and the rubric, then returns a score with a reason. It scales. It is also opinionated in ways you can measure and partly control.

Prefer pairwise comparisons when you can. "Which of these two responses better satisfies the instruction?" produces far more stable results than "rate this from 1 to 5," because absolute scales drift between runs and prompts. Reserve pointwise scoring for large sweeps where you need a rough signal.

Four pitfalls are worth designing around. Position bias: swap A and B and check whether the winner flips; if it does, run both orders and average. Verbosity bias: judges lean toward longer answers, so address length explicitly in the rubric. Self-preference: a model tends to favour text that resembles its own, so use a different judge than the model under test where practical. Score before reason: ask the judge to write its reasoning first, then commit to a score — the reasoning changes what it notices.

Then read thirty or forty judgments yourself. If you disagree with the judge more than occasionally, your rubric is usually the problem, not the judge.

Rubrics a person can actually apply

A workable rubric has four to six dimensions, each with three or four levels, and every level anchored with a real example drawn from your own outputs.

• Correctness: are the claims supported by the source or independently verifiable?
• Completeness: did it do everything the instruction asked, including the boring parts?
• Format: structure, length, required fields, valid syntax.
• Tone and register: appropriate to the audience and the channel.
• Safety and refusals: did it decline when it should, and help when it could?

Two rules save a lot of pain. Ban "quality" as a dimension — it means everything and nothing. And include a "can't tell" option, because without one reviewers guess, and their guesses look like data.

Human review that survives contact with a schedule

You cannot review everything, so sample deliberately: a random slice for the baseline, every item the judge scored near a decision boundary, and the cases your team already knows are hard.

• Blind the review. Reviewers should not know which model or prompt version produced the text.
• Double-rate a slice. Have two people score the same twenty items. If they diverge often, the rubric is ambiguous — fix it before collecting more ratings.
• Calibrate out loud. Score the same ten examples together, compare, and argue about the disagreements. This is the highest-value hour in the whole process.
• Cap the session. Thirty to forty items is roughly the limit before attention collapses.
• Keep the comments. Notes reveal the failure no score captures: "answered the wrong question entirely."

A workflow you can run this week

1. Build an eval set of 50 to 200 items from real traffic, including edge cases and past failures. Version it alongside your prompts.
2. Write the rubric and the list of expected failure modes.
3. Run the automated gates first — they are cheapest and catch breakage immediately.
4. Judge everything, ideally pairwise against the current production version.
5. Send a stratified sample to human reviewers: boundary cases plus a random slice.
6. Promote disagreements into new eval items so the set gets harder over time.
7. Re-run the whole set whenever the prompt, model or retrieval changes.

Keep it small enough to maintain

Fifty items you genuinely re-run beat five thousand you ran once and never opened again. Revisit the set every few months; traffic drifts, and yesterday's edge case becomes today's ordinary request. Write down why you chose each metric and each rubric level, so the next person does not have to reconstruct your reasoning from a spreadsheet.

Evaluation is not a report you produce once. It is a habit you keep while the product changes underneath you.

Photo: RDNE Stock project / Pexels