An AI proof of concept that works is a good result, and it is a narrower result than it looks. It shows that the technology can do the job on a sample that was chosen for it, for a few users, usually with wide permissions and nobody on call. That is appropriate for a test.
Production then asks six further questions. A team that answers them before the build starts knows what the build is. A team that meets them one at a time, after a launch date has been promised, usually finds the costly ones last, when they cost the most to fix.
1. Is it right on your real data?
A proof of concept is judged on a sample, and the sample is rarely the real distribution of cases. Accuracy has to be measured on cases drawn from your own work, with a correct answer agreed for each case before the test starts. The system also has to read that data from the systems of record that hold it, which is often harder than the model.
Check first: ask two people in the business to mark the same few dozen real cases. If they cannot agree on the right answers, accuracy cannot be measured yet, and that is the first piece of work.
2. Does it hold at peak load, and what does it cost there?
Response time has to be measured at the concurrency you expect at peak, with the mix of requests you expect, and then beyond that point until it degrades, so that the margin is known. A test with one user at a time shows neither figure.
Cost follows from the same test. The cost of each type of request is measured at that load, and the monthly cost is calculated from it and from the volume you expect. Where the model runs on capacity you rent or own, the cost of a request depends on how fully that capacity is used. Where the model is priced for each token, the unit price is known from the first request, and what has to be established is the volume, the supplier's rate limits and the licence terms at that volume.
Check first: write down the peak concurrency and the response time the use case needs. Without those two numbers no sizing is possible.
3. Who and what can reach it, and what can it reach?
A test is often run with wide permissions. Production needs an answer for each component: who and what can reach it, what it is permitted to reach and do, what data it handles and stores, where its secrets are held, and what leaves your environment.
For a model there is one more question, which is what untrusted input reaches it. Text inside a document or a request can be treated by the model as an instruction, so a model that can call tools needs its permissions drawn as narrowly as any other service account.
Check first: list every outside service the proof of concept sends data to, and confirm that each one has been approved to receive it.
4. What can you show an auditor?
An auditor asks what the system did, on which data, and who approved each change. The first two come from logs that the system has to be built to keep, with a retention period and protection from alteration that you specify. The third comes from your own change approval, which every change should pass through.
Both have to be designed in from the start, because neither can be reconstructed afterwards. A log added after an incident does not describe the incident.
Check first: ask your auditors or your risk team what they would want to see for this system, before the design is drawn.
5. How does it fail?
Any component can fail. Production needs to know, for each failure that can be foreseen, its cause, its effect, how it is detected and how the system recovers. A recovery time is a target until the recovery has been rehearsed, and it is a measurement afterwards.
For an AI system the failure that matters most leaves every health check green. A wrong answer is a failure, and so is an answer that changes after the supplier updates the model. The evaluation cases from the first question are how that failure is detected, so they have to be run again whenever the model or its supplier changes.
Check first: pick the one failure that would cost the most, and ask how anyone would find out that it had happened.
6. Who owns it and who runs it?
The team that built the proof of concept is usually not the team that will run it. Your own engineers have to be able to release it, roll it back and operate it, and someone has to be on call. A named owner in the business has to accept it. In a regulated business your risk and compliance functions also have to approve it, and that approval is often the longest item in the plan.
Check first: name the owner and the operating team now. A system with neither tends to stay a proof of concept.
What to do this week
The readiness check asks these six questions about your own proof of concept and lists what is open. It takes about two minutes, and nothing is stored.
If several of the six are open, that is normal, and it is the work our AI inference systems practice exists to do. The Why Udhyam page sets out how we take a proof of concept through each question and put the answers in writing.
