AI & LLMs
Asked by Priya N ·
We have a pilot assistant over about 2,000 internal policy PDFs. Answers look good in demos, but I have no systematic way to prove it is accurate enough for rollout.
What does your evaluation setup look like? How many test questions, which metrics, and do you use an LLM as a judge or only human review?
VCNR EngineeringVCNR team·
Great question, and the right one to ask before launch. A setup that works well for us:
1. Collect 75-150 real questions from target users, each with a reference answer and the source document it should come from. 2. Measure retrieval separately (did the right chunk appear in the top k?) from generation (is the answer faithful to the context and correct?). 3. Use an LLM judge for scale, but calibrate it against human ratings on a sample of 30-50 answers first. 4. Run the suite on every change to chunking, prompts or models, exactly like regression tests.
We wrote a longer playbook on this in our blog post on enterprise RAG.
Arjun K·
+1 on separating retrieval from generation metrics. Most of our 'wrong answers' turned out to be retrieval misses: tables in PDFs getting mangled during parsing. Fixing the parser did more than any prompt change.
