Find out where knowledge actually lives
The first step is an inventory of the documents, wikis, tickets and databases people consult today, not a model selection. Record each source's owner, format, language, update process and access rules. Distinguish an approved current procedure from an old note that nobody maintains. If these are mixed without labels, a search engine can return a persuasive but obsolete instruction. Ask staff how they recognise the right version now and when they prefer to call a colleague.
Define the project boundary too. It might cover public product manuals, internal operating procedures or one narrow business process. Every additional category brings different owners and rules. For a pilot, select a smaller set of important, maintained documents. Do not manufacture content that looks authoritative when the answer is missing. Record the gap and name the person who can fill it. Search cannot repair the absence of ownership and decisions about company knowledge.
Keyword search, hybrid retrieval and RAG serve different needs
An exact product name, ticket number or policy code is often best found by keyword matching. A question phrased differently from the source may benefit from semantic vector retrieval. Microsoft describes a hybrid query as combining keyword and vector search in one request. Do not choose an approach from a handful of attractive demos. Compare results on real questions, including short, incomplete and multilingual phrasing. People also need a way to understand why a document ranked highly.
RAG adds a generated answer to retrieved passages. It may save time when a reader needs to combine several sources, but creates another place for errors. For a simple navigation request, offering three relevant documents may be better than generating a paragraph. For a complex question, distinguish facts from documents from the model's own summary. In a pilot, measure passage retrieval and answer quality separately. A polished final sentence does not prove that the right source was found.
Prepare sources for traceable results
Split documents into passages that retain useful context and a stable reference to the original. Segments that are too short lose conditions and exceptions; overly long ones obscure the answer. Keep title, version, modification date, owner and permissions with each segment. Tables, attachments and images may contain vital information that plain text extraction misses. Decide whether they can be processed faithfully. If not, show a link to the original and explain the index's limitation.
A citation should be useful to a reader. A link to the front page of a long manual is often insufficient; a specific section or passage identifier is better. For sensitive procedures, show the effective date and document status. When a source disappears or is superseded, update the index and any cache. An answer should not keep pointing to a missing or forbidden reference. Connect indexing to the ordinary document lifecycle instead of relying on an occasional manual import.
Permissions apply before and after retrieval
An internal index often combines material with different access rules. A user must not see even the title, snippet or semantically similar neighbour of a document they cannot read. Microsoft describes security filters and other document-level controls in Azure AI Search. The specific technology can differ, but the principle remains: verify identity and group membership at query time, synchronise original permissions into the index, and filter results before passing them to a model or screen.
Test role changes and employee departure. When a document moves into a restricted category, a stale index may still return an older result for a while. Define an acceptable synchronisation delay and consider a fresh access check on especially sensitive reads. Cached answers must respect user context, not just query text. Audit access to sensitive sources while minimising personal data in search logs. Permissions are an architectural requirement, not an optional filter added after RAG is complete.
Evaluation needs questions and expected sources
Collect real queries from the people who use the documents. For each, identify which sources should be found, whether a definite answer exists and when the correct response is to admit missing evidence. Amazon Bedrock's knowledge base evaluation guidance uses expected texts and answers as reference data. The same idea can be applied locally without that service. Include known names, paraphrases, similar documents, old versions and questions outside the chosen scope.
Measure relevant passage retrieval, result order, faithfulness of the answer to cited text, correct refusal and time needed to verify. Do not depend on one average score. A single leak of an unauthorised document or recommendation of a revoked procedure may matter more than dozens of minor style flaws. Run the same cases after changing indexing, chunking, model or permissions. Give users a way to report a wrong source and route that correction to its owner.
A pilot shows whether search is enough
Begin with one process and a small group of people. Record current search time, questions that end in asking a colleague and documents most often missing. Compare keyword search, hybrid retrieval and possibly RAG against the same queries. For answers, observe whether users open citations, understand them and trust the result to the right degree. Query volume alone is not evidence of value; it can also mean the system keeps answering unclearly.
Before expanding, assign who updates sources, who resolves bad results and how generation can be switched off when unsafe. This page does not claim EXCOIN has a proprietary search product or measured client deployment outcome. Useful initial outputs are a source and permission map, a test set, a comparison of approaches and a maintenance plan. If reliable keyword search solves the central problem, adding a model merely because it looks impressive in a demo adds cost without clear benefit.
Include deliberately misleading questions in the pilot. The user may assume a rule is still active when it was revoked. The system should find supporting evidence or admit that none is available. Check how quickly a corrected source changes the index and who can report a wrong answer.