Start with the task, not a model leaderboard
Describe the specific work the model should support: finding an answer in internal rules, suggesting an email classification, summarizing a document or drafting a first response. For each task name the user, sensitivity of inputs, expected output and the decision a person makes afterward. A model that performs well on public tests may struggle with your languages, terminology and exceptions. Define when it must abstain or hand work to a person. First ask whether a model is needed at all; search, deterministic rules or a template may be more accurate and less expensive. Name a baseline process against which any improvement can be measured.
Private can mean running on your server, using a dedicated supplier instance or using a service under restricted data-processing terms. Clarify which requirement matters. It may concern data location, prohibition of training on content, access control or operation without an external network. Document the threats you aim to reduce and those that remain. A locally running model can still reveal information to a user with overly broad access. The hosting decision therefore follows the data flow and responsibilities, not merely the server address. Make sure the meaning of private is shared by legal, security and product owners before comparing prices.
Build an evaluation set from actual work
Collect representative questions and documents with the data owners' permission. Include ordinary, difficult and ambiguous cases, several languages and situations where the right answer is that the information is unavailable. Define reference expectations and scoring before running tests. Extraction may be checked against particular fields; answers over a knowledge base require factual support, completeness and useful citations. Use human review with a clear rubric and multiple reviewers for critical cases. A single aggregate score can hide a dangerous weakness in a small but important group. Record the type of each failure so the team knows what to fix.
Separate development examples from an independent check set. Repeated tuning on the same test gives a result that may not recur in normal work. Record the size, source, date and limitations of every task set. Add new pilot failures regularly while keeping a stable core for version comparison. The NIST profile for generative AI emphasizes risk management across the lifecycle; a one-time benchmark cannot replace ongoing observation. Set an acceptable error boundary and identify who approves it. Protect the test data because it may be as sensitive as the production material the system will later read.
Compare quality against the simplest alternative
Test several candidates on the same data, instructions and scoring rules. Alongside a large model include a smaller model, ordinary search and perhaps an external service if data policies permit. Look beyond the mean score to errors that might lead a user to a wrong decision. For document-based answers, separate failures to retrieve the right evidence from failures to generate a faithful response. If a model cannot acknowledge missing evidence, more training may not be the right remedy. Improve the task boundary, prompt and context first. Keep examples of both good and bad results for a review with domain specialists.
Measure each language and input length separately. A Czech or Spanish document may produce a different outcome from an English one even for the same task. Check stability under small changes in wording and among similar documents. Count unsupported claims, wrong citations, incomplete results and inappropriate refusal. In a cost comparison, do not rely only on a short laboratory prompt; real work includes long inputs, retries and human checking. The benefit must hold against the current process, not only against another model. State how much human time is still required to accept an output.
Assess security at the tool boundaries
An input document can contain instructions intended to redirect the assistant. OWASP describes prompt injection as malicious instructions in content overriding the intended behavior. Separate trusted rules from retrieved data and restrict the tools the model may call. Do not grant an entire database merely because the user asks one knowledge question. Test attempts to request another person's document, bypass a filter or trigger an unintended action. Enforce authorization outside the model; a language model is not a security boundary. Include the case where a retrieved source looks authoritative but is controlled by an untrusted person.
Review what is stored in prompts, logs, caches and backups. Internal hosting alone does not solve employee access to diagnostics or retention of sensitive inputs. Define encryption, roles, deletion, model updates and incident response. Verify the licence and provenance of model weights; public availability does not mean unrestricted commercial use. Record which systems provide context and where output returns. Security approval must cover the entire workflow rather than just the generation process. Reassess permissions whenever a new tool or data source is connected to the assistant.
Measure hardware and latency on your workload
Performance depends on model size, context length, concurrent demand, hardware and serving method. Measure time to first response, complete output time, memory use and behavior at peak demand. For an operator's live task, a slow answer may force a manual workaround; for a nightly summary, predictable completion may matter more than latency. MLPerf provides publicly defined inference benchmarks, but an investment decision still needs a test of your task and configuration. A technical number without its workload is not transferable. Include the longest realistic document and the number of people who may ask at once.
Count more than GPU purchase or rental: spare capacity, updates, monitoring, backups, energy, access administration and staff time. Compare cost per accepted result rather than only price per million tokens. A model that needs frequent human repair may cost more even when serving is cheap. Decide whether demand truly requires a dedicated server all day. A pilot may show that batch processing or a smaller model meets the goal with simpler infrastructure. Write cost assumptions about volume and concurrency so they can be recalculated. Consider the work of maintaining the application around the model as well as the model itself.
Make the pilot show refusal and escalation
Limit the first pilot to one task and a small group. Set the case count, quality measures, maximum time, cost and stopping conditions in advance. Test ordinary requests, a document without an answer, conflicting sources and intentionally malicious content. Users should know when they read a draft and who is responsible for the final decision. Retain failure evaluations without unnecessarily copying sensitive inputs. After the pilot separate model errors from problems in data, retrieval or task wording. Each cause calls for a different correction. Give reviewers a simple way to mark an output as unsafe or unsupported.
Compare the pilot with how the task was handled before. Measure human time, corrections and outcome quality. Faster work without acceptable correctness is not success; excellent correctness at an unaffordable operating cost may also fail the case. Ask users to describe situations in which they ignored the output. An unsuccessful pilot has value if it shows the task should be narrowed or solved differently. Document the next decision with evidence and uncertainty rather than impressions from attractive demonstrations. Decide whether another trial can answer the remaining question before committing to permanent hosting.
Manage model changes as product changes
A new weight file, prompt, retrieval process or index can change behavior without altering the visible screen. Version these parts and run the check set before release. If quality falls, be able to restore the previous configuration and find which version produced a particular output. Monitor errors, abstentions, latency and question types while respecting privacy. Periodically review samples of safely anonymized results with specialists. This is an operating system with accountability, not a one-time model installation. Set a release owner and document how a change is approved or rejected.
Name owners for data sources, model, infrastructure and user support. When an internal rule changes, someone must update its source and verify that an old answer is no longer offered. Prepare a response to harmful output: correction, notice to affected people and perhaps temporary suspension of the feature. Decide where human approval is required and where sample review suffices. The tool should have a simple shutoff that does not destroy original documents or work processes. Recheck costs and quality as usage grows; a pilot's economics may not represent an organization-wide rollout.
A checklist for the operating decision
Before investing, confirm the task, data flow, access rules, evaluation set, baseline, target measures and owners. Measure performance on target hardware with realistic input lengths. Test hostile instructions in documents, missing answers, loss of availability and return to the old process. Verify the model and data-source licences. Build a cost model for higher volume. If the task affects decisions about a person or money, define the boundary of human review. Without these elements, a request for private AI is too vague to compare solutions or approve a live deployment.
EXCOIN can prepare a pilot and operating design, but cannot promise that a particular model meets quality or cost targets without measuring on your data. The best result might be a smaller model, search without generation, an external service with suitable contractual terms, or a decision to defer the project. Private deployment makes sense when it brings demonstrable control or economics and a team can safely maintain the whole workflow. Decide from accepted results, cost and risk rather than a model name. Keep the decision record so new evidence can change it later.