Choose questions with an evidence-based answer
Begin with a specific user group and set of questions, such as a procedure for a service exception or an explanation of an internal rule. For each question identify the document that should contain the right answer and the person responsible for it. If knowledge lives only in people's heads or rules contradict each other, a model cannot repair the source. Fix the content first. The original research on retrieval-augmented generation combines finding relevant material with generating a response; the quality of selected evidence therefore directly affects credibility. Some tasks need only good search and no generated answer at all.
Set the consequence of a wrong answer. An internal orientation tool may tolerate more uncertainty than instructions for a process involving money or personal data. Specify when the system should say that it did not find an answer and where a user should go next. Decide whether it should return a passage citation, a link to the whole document or a version number. The aim is not to answer everything but to find the correct and current source quickly. Measure cases where abstaining is better than a plausible guess. Name the specialist who can resolve disputed source material.
Put document ownership in order first
Map repositories, file types, language versions and access structures. For each source identify an owner, effective date, version and retirement rule. An old document with a higher search score can be worse than a short current instruction. Check that text is machine-readable and that tables and appendices retain meaning during conversion. The aim is not to ingest everything; choose a collection whose correctness can be maintained. A document without an accountable owner becomes a risk over time even with an excellent model. Record when and how each item was last checked for current use.
Prepare a procedure for conflicts between sources. When two policies state different values, the generator should not silently invent a compromise. Show the disagreement and refer it to content owners. Preserve source identifiers, original location and metadata needed for filtering. Deletion and permission changes must reach the index within a known time. Decide whether historical versions should be searchable by ordinary users at all. That decision belongs to the data owner rather than only the search administrator. Test a changed document and its old version to see which one the system actually retrieves.
Treat indexing and chunking as product decisions
Long documents are commonly divided into smaller passages that can be retrieved and provided to a model. Chunks that are too short lose context; overly long chunks make selection harder and increase cost. Respect headings, tables and the boundaries of rules. Keep a link from every passage to its complete source and effective date. Test whether a question about an exception returns the exception rather than only the general introduction. Choose size from samples of your documents and questions, not a universal number. Also test whether exact wording, semantic similarity or a combination works best for your users.
Vector search can find similar meanings but may perform poorly for an agreement number, acronym or exact product name. Conventional search may serve those requests better. Consider a hybrid method and a later ranking step. Apply team, language, publication status and validity filters before passages reach the model. Record which passages were actually used in each answer. Without that evidence, it is difficult to distinguish an indexing problem from an unsupported generated claim. Every change to retrieval strategy needs a check that other query types have not worsened. Keep representative exact-match and semantic questions in the same evaluation set.
Enforce access before generation
Users should retrieve only documents they are authorized to read. Filtering after an answer is generated is too late because the model may already have used confidential content. Carry access rules from the source repository or create a reliable permissions map, and test how updates propagate. Try direct questions about another project, similar document titles and removal of a user's access during work. Shared team documents and metadata that reveal sensitive facts deserve special attention. The model must not be allowed to widen a user's permissions. A correct answer to an unauthorized question is still a security failure.
Retrieved text can include hostile instructions, such as a direction to ignore rules or send content elsewhere. OWASP describes prompt injection as a distinct risk. A source document is data, not a trusted application instruction. Restrict tools available to the model, separate system rules from retrieval results and do not give it actions it does not need. Test attacks inserted into documents and comments. Good prompt wording does not replace server-side access checks and human approval of sensitive actions. Record the origin of each retrieved passage so an untrusted document can be investigated and removed.
Make citations enable real verification
An answer should point to the particular passage used and open the original document, rather than display only a filename. Show version or date where rules change. A citation cannot rescue a claim that its source does not support. Test whether each important statement can be checked in the source and whether the user can actually open that link. For calculations or decisions affecting customers it may be safer to show the evidence and let a person form the final answer. The interface should distinguish a literal quote, a summary and a model inference. Avoid giving an impression of certainty through citation formatting alone.
When several possible sources are found, show their differences and effective dates. Do not force the model to choose one because its text is shorter. With no adequate evidence, it should state uncertainty and suggest the next step. Users need a simple way to report a wrong answer or outdated document. Connect the feedback to the question, selected passages and index version without keeping unnecessary sensitive input. That lets the team correct the cause and verify later that the problem has not returned. Let content owners see the reports that concern their documents.
Evaluate retrieval separately from the answer
Build questions from real work with expected documents and outcomes. Include ordinary cases, exact names, similar rules, old versions, questions without answers and requests across an access boundary. Measure whether the correct passage was retrieved, whether the answer is supported by it, whether it is complete and whether citations are right. Microsoft documentation on RAG evaluation distinguishes retrieval quality from groundedness of the result; the separation is useful even without its specific tools. A poor score then points toward data, index, prompt or model rather than a generic instruction to make AI better.
Reserve part of the set for an independent check or the system will learn only the test questions. Break results down by department, language and document type; an average may hide a serious error in one process. Use human review and a clear rubric for critical answers. Alongside quality track response time, cost and requests handed to a person. Set minimum acceptable outcomes before the pilot. Add newly found production errors after release while preserving a stable core for version comparison. Record when source changes invalidate an old reference answer.
Design updates and operations from the start
A document change should reach the index within a known period; deletion or permission removal may need a faster path. Watch import success, rejected files, index age and conversion errors for tables. Verify several key questions after an update. If ingestion fails, the system should not silently present an old rule as current. Document owners need to know how to confirm their change was indexed. A runbook should explain index recovery, reverting a configuration and who resolves a dispute about content accuracy. Test these actions before staff depend on the tool for daily work.
Version chunking, retrieval, prompt, model and index. If quality declines, find what changed and which configuration produced a particular answer. Monitor availability and load while protecting question content in logs. Set retention periods, roles and a route to remove recorded personal data under the actual rules. A generation outage should not cut users off from original documents. RAG should accelerate access to knowledge rather than become the only way the organization can find it. Keep a simple search or direct document route available when the generated layer fails.
Know when RAG is the wrong approach
If an answer depends on the current state of a transaction, document search may not suffice. It needs an authorized call to the system that owns that state. If the collection is small and well organized, conventional search or navigation may be simpler and more dependable. If the outcome must be deterministic, such as calculating an amount by an approved rule, generation may add needless risk. For conflicting sources, establish content ownership before building retrieval. RAG does not replace data governance or specialist judgment. Decide on the smallest method that gives users a verifiable answer.
Limit a first pilot to one managed collection and representative questions. Set thresholds for retrieval, supported answers, correct citations, latency and stopping. Compare it with ordinary search and the current user process. EXCOIN can design and implement a solution, but the actual benefit cannot be promised without inspecting your documents and permissions. Success means a user safely finds a current source, can open it and knows when to confirm an answer with a person. Document the pilot result so later expansion does not hide unresolved weaknesses.