A retrieval-augmented system looks impressive in a demo because the demo questions are the ones it was tuned on. In production, people ask things you didn't anticipate, in words you wouldn't choose. The only way to know how the system behaves is to test it on questions it hasn't been tuned on, scored against answers you trust. That takes about a day, not a quarter.
Build the set from reality, not imagination
- Collect 50 real questions — from support tickets, sales emails, internal chat or search logs. Real phrasing matters: misspellings, half-sentences and jargon included.
- Write the expected answer and the source document for each — in one or two sentences, with a pointer to where the answer lives. If you can't point to a source, the question doesn't belong in the set.
- Include deliberate traps — about ten questions whose correct response is 'we don't have that information', and five whose answers changed recently. These are where systems fail quietly.
- Tag each question — by topic and difficulty, so you can see which areas are weak instead of reading one blended score.
Score three things separately
A single accuracy number hides the cause of every failure. Score retrieval, grounding and behaviour independently.
- Retrieval — did the right source passage appear in what was fetched? If not, the problem is chunking, indexing or the query, and no amount of prompt tuning will fix it.
- Grounding — does the answer say only what the retrieved passages support, with a citation to them? An answer that is correct but unsupported is a lucky guess waiting to become a wrong one.
- Behaviour — on the trap questions, does it admit it doesn't know instead of inventing something plausible? For a business audience this is often the most important score.
A RAG system that sometimes says 'I don't know' and always shows its source is more valuable than one that is always fluent. Fluency is cheap; trustworthiness is the product.
Make it a gate, not a one-off
Rerun the set whenever anything changes: the embedding model, the chunking, the prompt, the underlying language model or the document set. Treat a drop on the trap questions or on a topic tag as a failed build. The set will grow as real failures arrive — every wrong answer in production becomes a new test case, which is how the system gets better instead of merely different.