Every serious LLM deployment eventually discovers the same lesson: leaderboard scores are not predictive of the metrics that matter to your product. Building your own evaluation is not optional.
What to measure
Effective evaluation measures user-facing outcomes, not proxies for model quality. Task completion rate. Time to resolution. Error rates on specific known failure modes. User overrides and edits. Downstream conversion. If a metric would not directly inform a product decision, it does not belong in your primary evaluation.
How to build the dataset
Start with real traffic. Sample a representative slice of production requests, annotate them with expected outcomes, and treat the annotated set as your golden dataset. Rotate the golden set every few months to prevent overfitting. Synthetic evaluations are useful for stress-testing but should never be your primary quality signal.
Include adversarial cases from day one. Ambiguous requests, requests that should be refused, edge-case formats — the model will encounter them in production, and you need to know how it handles them.
- Sample from real traffic, not from imagined use cases.
- Annotate with the outcomes your product actually cares about.
- Rotate golden datasets to avoid overfitting your evaluation to a snapshot.
- Include refusal cases and adversarial inputs from the start.
LLM-as-judge, carefully
Using a strong model to grade the output of another model is now standard practice for scale. It works when done carefully and fails silently when done casually. Grade the grader: sample its outputs against human judgments and calibrate. Use pairwise comparisons where possible — they are more reliable than single-answer ratings. Change the grader model only when you can quantify the impact.
“An LLM judge that has not been calibrated against human ratings is a random-number generator with expensive latency.”
Continuous vs milestone evaluation
Milestone evaluations run at release boundaries and catch regressions. Continuous evaluation samples production traffic and flags drifts as they emerge. Mature teams run both. Milestone evaluations are cheaper to reason about; continuous evaluations catch problems milestones miss.
The operational surface
An evaluation system in production needs alerting, dashboards, and easy re-runs on new models. It should be as first-class as your logging and monitoring infrastructure. Treat it as such: budget for it, staff it, and iterate on it as the model landscape shifts.
Above all, be honest with yourself about what the evaluation is telling you. Teams that regularly override their own evaluation to ship a favored model learn nothing from the exercise.
Key Topics
Extended Knowledge
- Pairwise comparison grading is generally more reliable than single-answer rating for LLM-as-judge.
- Golden datasets require periodic rotation to avoid Goodhart's-law overfitting.
- Production sampling and shadow deployments are the most reliable ways to catch model regressions before they impact users.
Frequently Asked
No. Leading models fail in different ways on your specific workload. Without evaluation you will not notice until users do.
Larger is better but the marginal value drops fast. Most production teams see clear signal from 200–1000 well-annotated examples covering representative cases.
Both, in a pipeline. Use humans to calibrate the judge, and the judge to scale. Never trust the judge blindly.
