Define what must be private
“Private AI” can mean several different things. One organisation needs documents to stay on its own network. Another wants limits on training from inputs, control over logs or the ability to leave a provider. These goals cannot be replaced by a single label. Draw the data path: where a question arrives, what documents are retrieved, where history is stored and who can inspect diagnostic information. Only then can concrete deployment options be compared.
Owning a server does not automatically create a secure system. If the application passes documents to a model without checking permissions, information can leak inside the organisation. An external API may offer contractual and technical safeguards that must be checked for the particular provider; there is no universal privacy claim about all APIs. For each option, ask who processes data, how long it is retained, how it is removed and who can change the settings. Evidence should determine the decision rather than product naming.
Specify the work the model will perform
Comparing models without a task resembles selecting a database by its console colour. Assemble representative cases: questions over internal documents, request classification, a proposed response or extraction from a form. Each case needs an expected result, an error tolerance and a decision on human review. Customer-facing text has a different risk from an internal draft. Include cases where the right answer is “I do not know” and where there is no relevant source.
The evaluation set should include the languages actually used by the product. A model that answers convincingly in English may mishandle Czech or Spanish terminology. Avoid placing secrets in an unverified test environment; use anonymised cases. The starting set need not be huge, but it should include both routine and difficult work. Separate examples used to choose settings from a final set used to assess results. Otherwise the team can optimise for the very questions on which it later claims success.
Compare options against the same bar
Make a simple matrix with three columns: owned infrastructure, isolated hosting and an external API. For each, record the data boundary, quality on the same evaluation set, latency under normal and peak demand, operating work, update control and total cost. Do not let one option appear better merely because it received stronger instructions or fresher documents. Change one meaningful variable at a time and retain the model, prompt, source and compute versions with every result.
The matrix need not produce one winner for every task. Summarising sensitive internal documents may require a stricter boundary than drafting public marketing copy. Some tasks do not require a model at all: good search or rule-based automation can be cheaper and more predictable. That is not a failed pilot. The comparison should show where a model brings measurable value, where another method is better and which assumptions still need evidence.
Performance is not one demonstration answer
Self hosted inference requires memory for model weights, working data and concurrent requests. Input and output length affect latency and consumption. Quantisation can reduce memory needs but is not a free speed boost; a particular method may change quality or throughput. Hugging Face documents different approaches and behaviour. Test them against your own tasks rather than choosing from a model name or a single published benchmark.
Measure normal traffic, peak periods and long documents. Track time to first response, total duration, failed requests and cost per completed task. A slower answer may be acceptable if it is substantially better; in another workflow delay is the main obstacle. Include restart, an unavailable model server and a full queue in the trial. A laboratory figure for one query on an otherwise idle machine does not describe service for real users.
Cost includes unused capacity and people
For self hosting, include owned or rented compute, power, storage, network, backup, upgrades, monitoring and staff time. Model expected traffic and a realistic peak. If demand is intermittent, an idle server can cost more than individual API calls. At a stable high volume, dedicated capacity may make economic sense. Neither conclusion can be asserted without a specific workload and measurements.
For an API, include input and output token prices, any supporting services, limits and integration costs. Pricing changes, so verify it with the provider at decision time and leave a reserve. Compare cost per successfully resolved task, not only cost per thousand tokens. A more expensive call could be cheaper overall if it needs less repair, and the reverse can also happen. Add the cost of human review for consequential work. An impressive unit price does not equal a viable operating budget.
Security does not end at the model's location
OWASP highlights the risk of sensitive information disclosure in model output. Permissions must therefore apply during document retrieval, context assembly and answer display. Do not automatically record entire conversations in logs without a clear purpose and controlled access. Diagnostic storage, backups and test-data copies can alter the actual data boundary. An owned model server cannot protect a document that the application shows to the wrong person.
Instructions hidden in sources create another risk. A retrieved document or incoming email may contain words intended to manipulate the model. OWASP describes prompt injection as a problem that RAG alone does not solve. If AI can call tools, the application must check the permissions and parameters of every action independently of model text. Include faulty and deliberately hostile inputs in a pilot. Protection applies to the whole workflow, not to one sentence in a system prompt.
Model updates are product changes
A new model version can change style, accuracy and refusal behaviour. So can an inference server update, new source documents or revised instructions. Test each material change against a stable regression set and compare it with the prior version. Record who approved the switch and how to return. With self hosting, track model licences and provenance. With an API, monitor availability of a particular model version and the provider's change process.
Operational monitoring should include latency, errors, capacity and the quality of real outcomes. The NIST AI Risk Management Framework treats risk management as continuing work, not one prelaunch test. Give users a simple way to report a wrong answer and have it assessed. When a pattern repeats, determine whether the cause is a weak source, model, permission rule or task design. Switching to a larger model is not automatically a cure.
Plan failure and a return path
What happens if the model does not respond? Some work can wait in a queue; other work needs an immediate manual path. The interface should distinguish processing, failed attempts and results requiring review. Self hosting adds restoration of servers and model files; an external API adds network and supplier dependence. Both need timeouts, bounded retries and a rule for when a person takes over.
Before the pilot, write down conditions for stopping or changing direction: unacceptable error rate, latency, cost or an unverified data control, for example. Without a decision boundary, a team can tune the system indefinitely. End the trial with a comparison of measured outcomes, open risks and next steps. Private AI is useful when a specific task gains the required control and quality at a proportionate total cost. It is not an end in itself.