Evaluate the path from question to answer
Retrieval augmented generation combines document search with a generated answer. A polished demonstration on a handful of questions says little about an outdated policy, conflicting versions, restricted material or a question with no answer in the corpus. Fluency alone is therefore a poor measure. Microsoft's RAG evaluator documentation separates retrieval quality, grounding in context and response relevance. This is a useful framework, but the acceptance rules must reflect the decisions employees will make with the answers.
First define the service's purpose. A helper locating internal procedures has a different risk profile from a tool drafting a legally significant position. Decide what the assistant may recommend, when it must show a source and when it must refer the user to a human. Assign an owner to each document class and its currency. An evaluation without this context can produce scores without showing whether the system is safe for the intended use. Write down the cost of a plausible but incorrect answer before setting any threshold.
Build tests around real work
Collect questions from actual information needs, rather than merely turning document titles into prompts. Include straightforward facts, multi-document questions, synonyms, misspellings, ambiguous names and the languages people really use. For each question, specify the expected kind of answer, acceptable sources and whether the answer is absent. Add time-sensitive cases: an old procedure beside a new one, a policy after its effective date and a withdrawn rule. Without these, a system may look accurate while citing obsolete content.
Test permission boundaries separately. Ask the same question as users with different roles and verify that restricted text does not leak through retrieved context, citations or the answer. Include off-topic queries, misleading wording and documents that contain instructions aimed at the model. Retrieved content is data, not authority over the assistant. OWASP describes this class of attack as prompt injection. It matters particularly in enterprise RAG, where an apparently ordinary document can carry hostile instructions. Capture expected refusal and safe escalation behavior for these cases.
Separate retrieval from generation
For each run, save the question, test role, index version, retrieved document identifiers, order and document versions. Review whether the correct document appears and whether the other results are relevant. When an answer fails, first ask whether the required evidence reached the generator at all. This boundary prevents a team from repeatedly editing the prompt when the problem lies in ingestion, chunk boundaries, search filters or access control.
Useful retrieval measures include the proportion of cases with relevant evidence in the first results, misleading-result rate and coverage across query types. A single aggregate score hides weak groups. Break results down by language, department, subject and document format. Company slang and product names may degrade search for only a subset of employees. Include tables, attachments and scans in the test corpus. If their structure disappears during ingestion, the generator has no reliable evidence to use. Repair the extraction pipeline before blaming the language model.
Check claims against their actual evidence
When the right context arrives, assess factual support, completeness and usefulness. Compare each material claim with the cited passage. A link to a whole document is weak evidence when the sentence does not follow from it. Distinguish direct facts, summaries and inferences. An inference should be labeled as an interpretation and accompanied by the underlying material. RAGAS describes automated evaluation of the relationships among question, context and answer. Such a metric can triage many cases, but a human must review sensitive claims and disputes.
Define expected behavior when sources conflict. The correct answer may identify the conflict, show effective dates and link both versions instead of offering one confident instruction. Calibration matters too. If the corpus lacks an answer, acknowledging the gap is better than inventing a likely explanation. Measure appropriate abstention and false refusals together: one catches unsupported answers, while the other reveals a tool that unnecessarily withholds useful information. Make these failure types visible independently in reporting.
Treat permissions and privacy as independent gates
Enforce document permissions before text reaches the model. Hiding a citation in the interface does not help if the content already entered the prompt or logs. Test roles, groups, shared folders and permission changes after indexing. Check that revoked access disappears from results within an agreed interval. Document titles and metadata can themselves be confidential. Decide separately whether a user may see that a document exists and whether they may read its contents.
Map what is retained in query logs, generated answers and feedback. These records can contain personal, contractual or commercially sensitive information. Set retention, reader permissions and deletion procedures. If an external model evaluates answers, establish exactly which data leaves the permitted environment and under which agreement. A smaller human-labeled sample may be more appropriate for a sensitive corpus than cheap-looking bulk evaluation that sends internal documents elsewhere. The evaluation process needs its own privacy review.
Automatic scores need calibration
A model judge can flag likely unsupported statements quickly, but it can also miss errors or mark valid answers wrong. Calibrate it against human-labeled examples and inspect agreement in each error category. Do not make a model using the same prompt the sole authority over its own outputs. Give evaluators a precise rubric: whether every material claim is supported, whether the answer adds an unauthorized conclusion and whether limitations are made clear. Version the rubric and evaluator; otherwise trend lines compare different measurement instruments.
Separate critical failures from routine quality improvement. A single leak of restricted text can block release despite excellent average relevance. A high-impact workflow may have zero tolerance for recommendations based on superseded policy. Ordinary internal search may use pilot-derived thresholds for coverage, relevance and speed. Set thresholds from the consequences of failure and name the person responsible for accepting risk. A dashboard number without a release decision is decoration. Preserve individual failing examples so a change can be diagnosed.
Run regression whenever the system or corpus changes
Run the same suite after model, prompt, indexing, chunking, embedding and permission changes. Keep a corpus version or document identifiers and checksums to distinguish system change from source change. Add production cases continuously: unanswered questions, disputed outputs and citations that pointed to obsolete pages. Verify each fix against previously successful cases too. Improving one question can degrade a different class of requests.
For latency, measure retrieval and generation separately. A slow response may come from a large context, an overloaded index or model queueing. Track cost per query without silently trading away evidence quality. Frequently asked questions may use approved prepared answers, provided their currency and access rules are checked. A test outcome should be reproducible with its exact inputs and versions. Otherwise the regression chart mixes real product movement with ordinary changes in company documents.
Define the release and incident process
Before a pilot, ask document owners to confirm the most important cases and the security team to inspect access boundaries. Release to a limited group and explain when users should open a source or ask a human. Allow people to flag an answer, but do not assume all feedback is correct; the reporter may lack context or authority. For critical incidents, provide a way to disable a collection, retrieval path or generator and fall back to direct document access.
After launch, monitor error categories and impact as well as scores. Have humans relabel a sample regularly and compare those decisions with automated judgments. EXCOIN can help design the test suite, instrumentation and operating rules, but the result depends on the real corpus, access model and risk tolerance of the organization. A good RAG system is one whose answers can be traced to evidence and whose silence can be explained when evidence is missing.