Limited-time dealSupermicro HGX B200, 8 GPUs8-GPU NVIDIA HGX B200 server in stock
View the specification Call +91 92058 93902innovations@udhyamlabs.tech
AI inference systems
This practice takes an AI use case from a working pilot to a service in daily use. In the assessment we agree with you what a correct answer is, on a set of your own cases, and what accuracy, response time and cost for each request the service has to reach.
Taking an AI pilot into daily use
A model that answers well in a demonstration has not yet been asked to serve every user at once. Doing that at a response time, and at a cost for each request, that the business can accept is a separate engineering problem, and it is the one we work on.
What the work is planned to deliver
- The service in production, available to the people it was built for
- Evaluation cases drawn from your own work, which show whether its answers are right
- A measured cost, in your currency, for each type of request
- PyTorch
- TensorFlow
- vLLM
- TensorRT
- Retrieval
- Quantisation
- KV cache sizing
- Evaluation cases
The drawing follows one request through the system and shows where the waiting time and the cost arise.
What the assessment examines, what you provide, and where the risk is
The assessment has a fixed fee and a fixed length, and it ends with written findings that are yours. You can stop there.
The assessment
We run the pilot against a sample of your real requests, in an account you control, and measure accuracy, response time and cost. You receive those figures, the evaluation cases, and a statement of whether the use case can reach its targets and what that would take.
What you provide
You provide the pilot or a description of the use case, a sample of real requests, and people who can say which answers are right.
Where the risk is
Data and cost carry the risk. Data can reach a provider you have not approved, and neither the response time nor the cost at full load can be predicted from a test with one user at a time. You choose and approve the model provider, and your data stays in an account you control unless you approve a provider to receive it. We measure response time and cost at the concurrency you expect at peak.
How the work is done
Serving every user at an acceptable cost is a different problem from getting one good answer from a model. The work has four parts. The first is how requests are batched under concurrent load. The second is how much memory is reserved for the KV cache, the stored attention state for every token already processed, which lets the model produce each new token without reprocessing the prompt and the answer so far. The third is where quantisation, which runs the model at lower numeric precision, saves enough to justify the accuracy it loses. The fourth is retrieval, which is tested at the collection size you expect to reach and not only at the size you have today.
This practice builds the use case itself and the system that serves it. It is intended for document understanding, support and operations assistants, extraction and classification over internal document collections, and domain assistants. Whether the model is open weight and run on your own capacity, or a proprietary model reached through an API, depends on what your residency rules and the economics permit, and the choice is yours to approve.
Where nothing is in production yet, we run the pilot on capacity rented in an account you control, in a region you choose, with a sample of your real requests. The capacity is rented in your account and billed to you by the provider, and the quotation for the assessment states our estimate of that cost beside the fee. Where there is no pilot, we work from stated inputs, such as the number of users, the type of document and the expected volume, and we tell you how uncertain the result is.
The work produces a set of evaluation cases, so that the next model release can be assessed against your data, and a cost model for each type of request, defined by input length, output length and concurrency. You are shown the cost of one request of each type in your currency. Your engineers also see it as a cost per million input tokens and per million output tokens, which are the units suppliers price in. A token is a word or a fragment of a word.
Questions about AI inference systems
The Working with us page answers the rest. If yours is not there, ask it on the contact page and an engineer will reply.
Who chooses the AI model, and where does our data go?
You choose. We recommend a model and show the measurements behind the recommendation, and you approve it. Your data and the evaluation cases built from it stay in an account you control. Nothing is sent to a model provider or to any other service unless you have approved it.
Nothing is running yet. What do you measure?
For an AI use case we run the pilot with a sample of your real requests, on capacity rented in an account you control. Where there is no pilot yet, we work from stated inputs and say so. For a hardware purchase we run a representative workload, built from the model, the context lengths and the number of users you expect, on capacity rented in the same way. In both cases we state how uncertain the result is and what would reduce the uncertainty.
How is the work priced?
The assessment has a fixed fee and a fixed length, and both are quoted before it starts. The price of the build is set out in the proposal that follows the assessment. We do not publish rates, because the fee follows the scope.
Articles on AI inference systems
Written by the engineers who do this work.
One team runs all five
Describe the problem, and an engineer will reply within one working day
Tell us in general terms what you are trying to do and what has stopped it. If it is not ours to take on, we will say so.