Inventory tasks and accountable people
List which systems use AI, for what purpose and for whom. Distinguish drafting, recommendation, automated decision and action in another system. For each task name the process owner, data owner, technical maintainer and person who approves changes. A single AI box in an architecture diagram is insufficient: a model might classify email or have permission to send a customer message. Those cases carry different risks. The NIST AI RMF organizes work around governing, mapping, measuring and managing risks; it provides a shared language without requiring a heavy process. Keep the inventory useful for the people making release decisions.
Give each workflow a purpose and boundary. What input may it take, what output may it produce and what must be handed to a person? State whether it uses personal data, contractually restricted material or information with financial consequences. Specify what happens when the model is unavailable and whether a manual route exists. Document the decision in terms operations staff can understand. Without ownership, an error can disappear between the model provider, engineers and users. Accountability should cover the result of the process, not merely API uptime. Identify who may suspend the feature when evidence of harm appears.
Classify risk by the consequence of an action
Assess the probability and effect of error for each automation. An internal meeting summary, a customer reply draft and a change to payment details do not belong in one class. Consider whether a mistake is visible, reversible and likely to be caught by a person in time. Mark actions involving a person, money, access or disclosure of data. Verify requirements for the particular product and jurisdiction separately; a general technical guide does not make a legal classification. Greater potential impact calls for stronger testing, narrower permission and more meaningful human control. Record who accepts the residual risk after those controls.
State the conditions for increasing autonomy. A draft might eventually gain permission to save an internal label, but not to send a client message. Do not widen authority because a set of attractive demonstrations succeeded; use representative measurement and a plan for exceptions. The risk class changes when a new data source or tool is connected. Reassess it whenever the workflow changes materially. A small table of tasks, impacts and controls helps more than a long policy nobody consults during a release. Revisit the classification after incidents and as volume grows.
Keep authorization outside the model
A model should not decide for itself whether someone may read a document, change a customer record or initiate a payment. The server application must enforce those checks. Give tools the smallest scope: selected data read, a draft and perhaps a limited action requiring approval. Separate test and production accounts and keys. When roles change, check that permissions propagate into indexes, saved conversations and caches. Being signed in is not authority over every object. The access policy should be testable with concrete examples of different users. A valid model answer based on another user's confidential data remains a breach.
Input texts, email and web pages can contain instructions intended to redirect automation. OWASP describes prompt injection as a risk arising from such content. Treat untrusted text as data rather than commands. Validate action parameters, destinations and limits outside the model before execution. Watch particularly for requests to send data elsewhere or retrieve material outside the original task. Test attacks embedded in sources and ordinary user mistakes. A prompt cannot replace restricted permissions and an audit record. Review each newly connected tool as a potential expansion of what an attacker could induce the system to do.
Design human approval as a real control
An approver needs to see the proposal, supporting material, reason and possible consequences. An approve button without time and context is a formal barrier only. Decide which actions require a second person, which the initiating user may confirm and which are safe to run automatically. Approval applies to an exact output version; if the model changes the proposal afterward, a new decision is needed. Offer repair and escalation. For bulk approvals set limits because reviewing hundreds of items can turn oversight into a fiction. Measure whether the interface lets people spot a bad suggestion before it takes effect.
During the pilot observe how many suggestions people correct, reject or approve without reading. A high approval rate alone does not prove quality if checking is exhausting or evidence is hidden. Ask reviewers when they feel uncertain and what information would help them. If responsibility remains with a person, the interface must enable reasonable judgment. Automation should save routine work, not transfer risk to someone who cannot see what the model did. Let a reviewer record why they overrode a suggestion and feed that case back into testing.
Preserve traceability without collecting everything
For each consequential action record its initiator, workflow and model version, relevant input identifiers, proposal, approval and final effect. The record should explain why something happened and how to correct it. It need not copy complete sensitive documents into logs. Separate operational diagnostics from audit evidence and define who may inspect each. When an error occurs, link the output to its prompt, tool and data-source versions. Changes to parameters and permissions may matter as much as code changes. Check that a person outside the development team can reconstruct a sample decision using the retained evidence.
Set retention by purpose and actual organizational rules. Users should know when text reaches an external provider and who may see it. If the workflow uses several models or services, document the actual data path. When a source is deleted or corrected, consider whether old content remains in saved output or conversation. Incident records should be available to the responsible team rather than every administrator. Traceability is no reason to store everything forever; useful evidence selects the information needed to explain and repair an outcome. Test whether the deletion process covers derived records where required.
Measure quality on realistic scenarios
Build ordinary, difficult and prohibited cases. Measure correct outcome, unsupported claim, unsuitable action, human handoff and time to repair. For classification examine errors by class; for drafted text, factual accuracy and language; for an autonomous action, also the target and parameters. An average score can conceal a small group of serious failures. Keep tests separate from tuning examples and add new failures regularly. The NIST generative AI profile emphasizes assessing risk throughout the lifecycle, not only in an initial demonstration. Agree on a failure threshold and owner before launch so a disappointing result can stop expansion.
After release track accepted suggestions, corrections and harmful outcomes. Record missing sources, changed model behavior and failures of connected services. Each measure needs an owner and a threshold for action. If quality falls, revert the configuration or restrict the feature rather than merely highlighting a dashboard warning. Also look for user workarounds that increase risk, such as copying sensitive information into an unapproved tool. Effective governance should appear in ordinary daily work. Ask whether the system still serves its original task rather than just generating a growing number of requests.
Prepare incident response and shutdown
Before production rehearse a wrong action, exposure of sensitive content and loss of model availability. Decide who stops the automation, assesses impact and informs affected people. It should be possible to disable the AI feature separately from the main application so the business process can continue manually. Preserve evidence for investigation and prevent a repeated action after restart. For important operations prepare a compensating procedure. Redeploying an old container is insufficient if AI has already changed data in another system. Test the shutoff with the people who will actually operate it.
After an incident distinguish model error, poor source, inadequate permission, ambiguous task and operator mistake. Each cause needs a different remedy. Add a regression case, update controls and approve the autonomy level again. An incident report should explain what happened rather than claim that a similar error is now impossible. If impact cannot be established reliably, that uncertainty is an important finding. Keep the response plan concise and accessible to the people on call. Include a criterion for resuming the feature once repairs and tests have been completed.
Introduce change through small decisions
A new model, prompt, connector or data source can alter the result without changing the user interface. Version them and run the control set before release. Approve wider authority separately. For each change record expected benefit, new risk and return condition. Start at low volume, watch corrections and expand only after observation. If the feature saves time only for some cases, let people choose the appropriate path. The aim is predictable outcomes at a reasonable workload, not the maximum possible autonomy. Keep temporary restrictions visible so they do not silently become permanent policy.
EXCOIN can help design and implement a specific workflow, but cannot promise a safe autonomy level without knowledge of the data, process and applicable rules. At each milestone compare time saved, result quality, human correction volume and operating cost. If benefit does not outweigh checking work, narrow the task or do not deploy automation. Good governance allows that decision without losing the original process. A clear owner, measurement and shutoff often help more than a complex library of theoretical policies. Document the evidence that supports continuation or withdrawal.
A checklist before enabling automation
Confirm purpose, owner, data flow, risk level, permission and the person accountable for outcomes. Test incorrect and hostile input, rejection of a proposal, model outage, repeated action and manual continuation. Verify that approvers see the relevant evidence and records explain a particular result. Review input storage, log access, cost and service limits. State who can turn the feature off and how an already completed action is repaired. Only then consider wider autonomy or marketing claims about savings. Make the checklist specific to the actual workflow rather than a blanket sign-off for all AI use.
After launch schedule regular review with process, security and operations owners. When data or rules change, update tests and reassess permissions. User error reports need a simple route to repair. Distinguish a successful pilot from readiness for routine operation at higher volume. The real outcome is a traceable process in which people know when to use the model, when to verify an output and who makes the final decision. Such automation can grow without quietly transferring risk between teams. Keep the review date and results in a place future maintainers can find.