Start with the source, then choose a model
Enterprise knowledge is spread across documents, wikis, contracts, databases and people. RAG can help when someone needs a grounded answer from those materials and opening a list of files does not complete the task. A language model can summarise retrieved evidence in the context of a question. It cannot guarantee that the evidence is correct, current or available to the right person. Begin with an inventory of sources and responsibilities rather than a model selected from a public demonstration.
For each collection, record its owner, update frequency, access rules and the meaning of archived versions. An approved procedure may have different authority from an old chat note. Combining both in a single undifferentiated index can produce a fluent but incorrect answer. The quality of the document catalogue is part of the quality of the answer. Sometimes the first valuable step is to clean up content and naming. No retrieval technique can fix contradictory, ownerless knowledge by itself.
Trace the journey from question to answer
A typical flow receives a question, retrieves relevant passages, gives them to a model with instructions to use the evidence and presents an answer with links. Each step can fail in a different way. The query may not match document vocabulary, retrieval may choose an old version, a passage may lose important context and the model may combine two incompatible statements. Microsoft describes RAG as retrieval, context assembly and generation, and notes that model context is limited. Sending every document to the model is neither useful nor economical.
A worked design must say which parts of the question drive search, how permissions filter results and what is shown to the person asking. The response may include direct sources, effective dates and a warning when evidence is missing. For a complex question, showing conflicting evidence may be more honest than generating one confident conclusion. Distinguish a claim found in a document from advice inferred by the model. That distinction improves review and gives users a clearer reason to trust or challenge the result.
Indexing is not a one-time import
A document must be ingested, split into useful units, enriched with metadata and represented for retrieval. Large chunks can add noise; very small ones lose context. Tables, attachments and scans need different treatment from clean text. Preserve the link from each passage to its original file, version, location and access rules. Without that connection, a reliable citation is impossible and removing content after a permission change becomes risky. Plan for deletion and replacement before the first import.
Compare full-text search, semantic vectors and their combination on real questions. An exact contract number may benefit from keyword matching, while a differently phrased question may benefit from semantic similarity. Reranking can improve the order of candidates but costs latency and money. Choose chunk sizes and result counts from measurements, not a universal recipe. Keep versions of the indexing process so a sudden quality change can be traced to the ingestion logic rather than blamed automatically on the model.
Permissions must survive indexing
A person must not receive an answer based on a document they cannot open directly. An indirect disclosure in a summary still matters. A secure system applies access rules during retrieval or before selected text reaches the model; hiding a link in the final display is insufficient. A role change must affect searches without a dangerous delay. Removing a document from access also needs a defined path through indexes and caches. These are product requirements, not merely settings on a search server.
Microsoft describes document-level security trimming for RAG, but implementation depends on the identity source and product used. A custom system needs tests with several real roles and genuinely different access. Check caches, shared conversations, analytics logs and exported answers as well. A permission check has little value if it works for the first query while a sensitive passage remains visible through conversation history to another group. Test access over the entire lifecycle, including revocation.
Evaluate retrieval separately from generation
A strong model cannot ground an answer if search did not return the necessary document. Build a set of representative questions and label the expected sources, acceptable answer and cases where declining to conclude is correct. Measure whether the right source appears among results, whether an obsolete version outranks it and whether the needed passage is present. Microsoft Foundry distinguishes retrieval evaluation from answer evaluation. That separation shows which component a proposed change can actually improve.
Next assess factual correctness, completeness, citations, unsupported claims and readability. Automated scores can compare versions, but important workflows need human review too. Include ordinary, ambiguous and adversarial questions in every language that will be used. Do not tune exclusively against the same set used to announce success. Hold some examples aside for independent checks after a model, index or instruction change. Record versions with results so the team can explain both improvements and regressions.
Make citations and uncertainty useful
A citation should lead to the actual passage supporting a claim. A link to a whole document is weak evidence when that document spans many pages or contains conflicting versions. Users should be able to open the source and understand when it was valid. If retrieved text does not answer the question, the system should say so and offer the available material or a route to a knowledgeable person. Confident language without evidence raises the chance of acting on an incorrect result.
RAG assists knowledge work; it does not automatically approve decisions. For legal, financial, security or employment questions, define what the system may suggest and what an authorised person must confirm. The interface should communicate this boundary in the task itself. An answer may accurately summarise an approved internal procedure and still fail to settle a new case. Give users a way to report incorrect sources, and allow operators to withdraw a problematic collection while it is corrected.
Defend against instructions inside documents
Retrieved text is data, not an administrator of the application. A document can contain wording that tries to override the model's instructions, solicit extra information or trigger an unintended action. OWASP notes that RAG alone does not remove prompt injection. The application must separate source content from system rules, limit available tools and validate sensitive actions outside the model. Adding a sentence that says “ignore malicious text” is not an adequate control on its own.
Put suspicious instructions into test documents during the pilot and verify that they cannot gain authority or cause a tool action. Consider source poisoning as well: who may create, alter and approve a document? Index history and backup should make it possible to trace a change that suddenly harms answer quality. Security controls therefore depend on editorial ownership as much as model configuration. Clear roles and review paths matter more than a sophisticated chat interface.
Pilot carefully and operate deliberately
Limit the first pilot to one useful collection and a group who know the correct answers. Measure time to find information, source accuracy, unsupported responses, latency and cost per genuinely resolved question. Classify failures by cause: stale document, poor retrieval, wrong synthesis or broken permission. That breakdown shows whether to invest in source content, index tuning, model behaviour or the user interface. A single overall satisfaction score cannot explain what to fix.
After launch, plan reindexing, deletion, permission updates, failure review and regular checks against a stable question set. Users need to know where to report a mistake and how a source gets corrected. For some tasks, strong full-text search with clear navigation can be cheaper and more accurate. That is a valid outcome of the comparison. EXCOIN can help design and test an approach against real documents and operational boundaries; this page makes no unsupported promise of precision or return on investment.