Why every AI feature we build starts with a test set, and how that keeps quality steady as models change.
An LLM feature that works in a demo can still fail one in ten real users. The only way to know is to measure it, so we write the evaluation before we write the feature.
Build the test set from real cases
We collect fifty to two hundred real examples — support tickets, search queries, documents — and agree with the client on what a good answer looks like for each one.
Score every change
- Run the full set on every prompt or model change
- Track accuracy, refusals, cost and latency together
- Add every production failure back into the set
When a new model comes out, the eval tells us in minutes whether it's an upgrade for this product, not just on a public benchmark.
Guardrails for the rest
Evals catch most problems before launch. Input checks, output filters and clear ways for users to report a bad answer catch the rest in production.