A quotation for GPU servers arrives with a part list and a validity date. The part list answers the question the supplier was asked. Whether it answers the question your business has depends on four things that the quotation does not contain: the model you intend to run, the length of the requests, the number of users at peak, and the response time the use case needs.
This article sets out how we would check a quotation, in the order we would do it.
Start from the workload
Sizing starts from what will run on the hardware. Before looking at the part list, write down:
- the model, its parameter count and its numeric precision
- the distribution of context lengths, which is how long the requests are
- the ratio of input tokens to output tokens
- the arrival rate at peak
- the response time target
The last item is the one most often missing. Without a response time target there is no way to choose how many requests the server processes together, and without that there is no way to say which limit of the hardware is reached first.
Find the limit that binds
A GPU server has several limits. It can calculate only so fast, it can move data through its memory only so fast, and it holds only so much memory. The limit that your workload reaches first is called the binding constraint, and it is the one to buy against.
Which limit binds depends on the workload:
- At low batch width, the decode stage of a language model is limited by memory bandwidth.
- As batch width rises, the limit moves towards compute.
- For long contexts, the memory taken by the KV cache usually caps the batch width first. The KV cache is the stored attention state for every token already processed.
- Multi-node training is frequently limited by the network between the nodes.
This matters for the comparison between two parts. When memory bandwidth is the limit, a part with a higher compute figure and the same bandwidth adds almost no throughput at that operating point. A quotation built on the headline compute figure can therefore be the wrong purchase for a serving workload.
The way to find the binding limit is to measure. We run a representative workload on rented capacity of the same accelerator, in an account you control. That measurement characterises the accelerator. The quoted host and network are then checked by calculation, and the findings say which figures were measured and which were calculated.
Check what surrounds the accelerator
The host, the network, and the power and cooling envelope are decided together with the accelerator. A quotation can be correct on the accelerator and still not be something your facility can run. These are the questions to put to the supplier in writing:
- What power does one fully loaded server draw, and what does a full rack need?
- What cooling must the data centre supply for that rack?
- Is the network sized for serving or for multi-node training? The two need different designs.
- What does the support contract cover, and for how long?
Put buying, colocating and renting on one basis
The three routes are priced in different units, so they have to be converted to one. The unit that works is the cost per million tokens, because it is the unit that suppliers of rented capacity price in.
For hardware you own, the figure is built like this:
The capital cost depreciated over a stated period, plus financing, power, space, support contracts and the staff to run the hardware, divided by the tokens served at the utilisation you expect.
Dividing by the tokens served at the utilisation you expect is how idle capacity enters the figure. Servers that stand idle for half the day cost close to twice as much for each token as the same servers kept busy, because the fixed costs are spread over half the tokens.
| Route | What you pay for | What moves the answer most |
|---|---|---|
| Buy and house it yourself | Capital, financing, power, space, support and staff | How fully the servers are used |
| Buy and colocate | Capital, financing, the colocation fee, support and staff | Utilisation, and the length of the contract |
| Rent | A price for each hour or for each token | How steady the demand is |
Suppliers price input tokens and output tokens separately, so the figure should be given for each, or as one figure at your measured mix.
Before committing capital it can be cheaper to rent the same capacity for a period, observe how heavily it is used, and buy only if use stays high. The useful output of the exercise is the level of sustained use at which buying becomes cheaper than renting for your workload.
Before the validity date
A quotation check has four possible answers: buy as quoted, buy less or more, buy a different part, or rent first. Any of the four is a good result if the arithmetic behind it is shown.
If you hold a quotation now, the two things that decide whether a check is possible are the validity date and a description of what you intend to run. Our AI hardware advisory practice does this work, and no supplier pays us, whichever answer we give. The method is set out in more detail under how we size compute.
