InferenceLatency budgets

Latency budgets, and what providers publish

How to set an inference latency budget: TTFT, inter-token latency, batch size, and what nine European inference providers publish.

, AI Researcher5 min read
In this article6
  1. What a latency budget has to cover: TTFT, inter-token latency, total time
  2. Allocating the budget across the stack
  3. Where published benchmarks do not transfer
  4. Setting a budget providers can be held to
  5. Frequently asked questions
  6. Sources
Diagram of an inference latency budget splitting total response time into time to first token and inter-token latency, shown against a batch of concurrent requests
Diagram of an inference latency budget splitting total response time into time to first token and inter-token latency, shown against a batch of concurrent requests

What a latency budget has to cover: TTFT, inter-token latency, total time

A budget is defensible only when each of three figures carries its unit and the batch size or concurrency it was recorded at. Time to first token (TTFT) is the wait, in milliseconds or seconds, before the first streamed token arrives. Inter-token latency is the time per output token during decode, usually quoted as output tokens per second. Total response time is TTFT plus the decode time for the full output. Artificial Analysis's methodology page (retrieved 25 September 2026) defines output speed as "the average number of tokens received per second, after the first token is recei[1]ved". vLLM's documentation (retrieved 25 September 2026) lists continuous batching of incoming requests, chunked prefill and prefix caching among the features behind what it describes as state-of-the-art serving throughput[2], so how requests are batched shapes both figures.

ProviderPublished TTFTOutput tokens per secondBatch or concurrency basisDated public page
Exoscale––––
Hetzner––––
IONOS––––
Nebius––––
Open Telekom Cloud––––
OVHcloud––––
Scaleway––––
STACKIT––––
TensorX––––

A "–" means the provider's own public page, retrieved 23 September 2026, lists no value for that column. For what these providers do publish, such as processing regions, pricing units and data processing agreement terms, the comparison of European serverless inference providers covers the same row set.

Allocating the budget across the stack

A latency budget splits along the two phases of generation. TTFT is dominated by prompt length and prefill compute: a longer prompt means more tokens to process before the first output. Inter-token latency is dominated by decode speed and concurrency: the more requests share a GPU, the slower each user's stream runs.

  1. Fix the application's ceiling: the maximum TTFT and the minimum output tokens per second the product tolerates.
  2. State the concurrency the figure must hold at, not a single-user ideal case.
  3. Attribute the remainder: prompt length, model size, quantisation and batching each move one phase, not both.
  4. Record source, date and basis beside every number.

Where published benchmarks do not transfer

The recurring error on this topic is a benchmark run at a batch size no reader will ever use. NVIDIA's trtllm-bench concurrency sweeps (developer blog, retrieved 25 September 2026) show the trade-off directly: per-GPU output throughput rises as concurrency grows while per-user output speed falls, and an FP8 quantised checkpoint serves more concurrent users within the same latency budget than an FP16 baseline[3]. Argonne National Laboratory's LLM-Inference-Bench, published on 31 October 2024, spans 7B and 70B parameter models across GPUs from NVIDIA and AMD and accelerators from Intel Habana and SambaNova, with several inference frameworks, and its results shift by hardware and engine[4]. A figure measured on one engine, one quantisation and one batch size does not transfer to another. The cost side of the same engine choices is covered in the analysis of what drives NVIDIA inference platform pricing.

Setting a budget providers can be held to

Four criteria make a latency number auditable:

  • Demand the measurement basis in full: TTFT or tokens per second, the unit, the batch size or concurrency, the date and the page it came from.
  • Treat a provider's own page as first-party for that provider's figures only; it is never evidence about a rival.
  • Accept no latency, SLA or throughput figure without a dated page stating it, and no datacentre location without one.
  • Distinguish a forecast from an observed measurement before either enters the plan.

The regulatory frame sets no number to lean on. The EU AI Act, Regulation (EU) 2024/1689, was published in the Official Journal of the European Union on 12 July 2024[5], and its text contains no latency thresholds[6]. The context in which latency budgets get written is the Commission's cloud policy, whose page (retrieved 25 September 2026) cites Eurostat recording 45.2% of EU businesses using cloud services in 2023, up 4.2 percentage points from 2021, against the European Commission's Digital Decade target of 75% of European businesses using cloud-edge technologies by 2030[7]. For the data protection side of the same procurement decision, the verification steps for a GDPR compliant inference platform apply the same dated-evidence test.

Frequently asked questions

How is inference latency measured, and what TTFT should I expect from European providers?

Latency is measured as three numbers: time to first token (TTFT), inter-token latency during decode, and total response time, each stated with its unit and the batch size or concurrency behind it. Independent instruments such as Artificial Analysis publish output tokens per second for model APIs, but none of the nine European inference providers compared here publishes a dated TTFT value on its own pages (retrieved 23 September 2026). The honest expectation is a range you verify yourself at your own batch size, not a single published number.

What is the difference between TTFT and tokens per second?

TTFT is the wait until the first streamed token, driven mainly by prompt length during prefill. Tokens per second measures decode speed after that first token. A long prompt raises TTFT, while concurrency and model size drive tokens per second, so a budget needs both.

Why does batch size change benchmark results so much?

Because the more requests share a GPU, the slower each stream runs. A figure quoted without its batch size or concurrency says nothing about what a single user will experience.

Do published inference benchmarks transfer to my deployment?

Only if hardware, framework, model and batch size all match. Argonne's LLM-Inference-Bench found results shift with each, so treat any benchmark as evidence about its own configuration, not about yours.

What should a latency commitment from a provider specify?

The metric (TTFT, inter-token latency or total time), its unit, the batch size or concurrency it was measured at, and the dated page it came from.

Does EU regulation set latency requirements for AI inference?

No. The AI Act (Regulation (EU) 2024/1689) sets obligations around risk and transparency, not response-time thresholds, so latency budgets are contractual and internal.

Sources

  1. artificialanalysis.ai
  2. docs.vllm.ai
  3. developer.nvidia.com
  4. arxiv.org
  5. artificialintelligenceact.eu
  6. eur-lex.europa.eu
  7. digital-strategy.ec.europa.eu

Rankings and scores on Optinest cannot be bought — no commercial relationship changes how a provider is assessed. How we score

Related reading

All Inference →