Value begins with a specific task
AI discussions often begin too broadly: “we need a chatbot” or “we want agents.” Such statements do not say who the system helps, what input it receives or what happens after a wrong answer. Start with repeatable work, an owner and a measure: finding the right internal procedure, classifying requests or drafting a response that a person reviews. Measure current time, error rate and the proportion of cases where essential information is missing.
Then compare ordinary search, rules-based automation and a model. Generated text can help when a task involves variable language or combining information from several documents. When a decision follows precise rules, a deterministic process may be more appropriate. A pilot should show the difference on representative cases rather than a few impressive demonstrations. Success means an improvement in the whole workflow, including human review and correction.
Data and permissions before the model
The quality of internal answers depends on the quality of source material. Documents may be old, contradictory, duplicated or unavailable to the people who need them. Before building a RAG system, establish document ownership, update rules, access rights and a way to cite a specific passage. The index must not create a new path to information a user is not permitted to see. Evaluation must also include questions the system should decline to answer.
Sensitive information requires a map of its flow: what is sent to the model provider, what enters logs, how long conversations are kept and who may read outputs. A local model can give more infrastructure control while creating responsibility for performance, updates and environment security. Choose where the model runs based on real data and operating needs rather than the label “private.” Legal requirements depend on the particular use and must be assessed separately.
RAG and answer quality
RAG typically combines retrieval of relevant material with answer generation. Failure can occur at every step: a document is not indexed, search returns an old version, an excerpt loses important context or the model draws a conclusion the source does not support. Measure separately whether the right document is found, whether citations are accurate and whether the final answer is useful. Changing prompts without a test set often means changing behavior blindly.
A test set should include common questions, edge cases, similarly worded requests with different meaning and questions with no supported answer. People who perform the work can judge whether an output helps their actual decision. A change to documents, model or retrieval may change quality, so repeat the evaluation. The NIST AI Risk Management Framework offers a general approach to managing risk across the life cycle; it is not a certification of a particular application.
Automation with a human boundary
Not every AI task should end in an automatic action. A draft response, document summary or suggested handoff can save time while preserving human responsibility. If a system changes records or sends messages, specify allowed operations, approval steps, reversal and an audit trail. Tool permissions should be no wider than the task requires. An exception must reach a person in a form they can actually resolve.
Text from documents, email and websites is untrusted input even when it appears to contain instructions. OWASP identifies prompt injection as a risk in language-model applications. A system prompt is not the only defense: narrow tool permissions, separation of data and instructions, output checks and confirmation of consequential actions matter. A pilot should include malicious and accidentally confusing inputs. Without those tests, it is easy to mistake a demonstration for a dependable process.
A pilot and an operating decision
A pilot starts with a small set of tasks, known inputs, expected outcomes and an explicit cost of error. Decide in advance how to compare it with current work: time saved after review, share of correct answers, cases returned to people and operating cost. Averages can hide failures in rare but important situations. High-impact cases need a separate gate or should remain outside automation entirely.
Production adds model and prompt versioning, quality monitoring, a spending limit, incident response and a way to turn the feature off. Every change should be traceable to the evaluation set. If quality cannot be measured reliably or no one owns the outcome, assisted mode is the safer choice. This remains proportionate for a small project: fewer components, with clear responsibility and an outcome that can be checked.