InferenceLatency budgets

P99 latency for LLM inference: what it reports and omits

P99 latency for LLM inference explained: what the percentile reports, why batch size distorts published figures, and how to measure your own tail latency.

, AI Researcher5 min readUpdated 30 September 2026
In this article6
  1. What P99 latency actually reports
  2. How percentile latency is computed
  3. Why the tail matters more than the mean
  4. What a published P99 omits
  5. Measuring your own tail latency
  6. Sources
Chart of a request latency distribution with the median marked left of centre and the P99 tail shaded to the right, labelled with time-to-first-token and inter-token latency regions.
Chart of a request latency distribution with the median marked left of centre and the P99 tail shaded to the right, labelled with time-to-first-token and inter-token latency regions.

What P99 latency actually reports

P99 latency is the value that almost all requests complete faster than; only the slowest tail of requests takes longer. For LLM inference, the number depends on which of three measurement bases it sits on, the same three timed in Argonne National Laboratory's LLM-Inference-Bench (2024): time-to-first-token (TTFT), the time to produce the first output token; inter-token latency (ITL), the average interval between consecutive output tokens; and end-to-end request latency, the time from prompt to final token.

The three bases differ by roughly an order of magnitude for the same model and hardware, so a P99 figure without its basis cannot be compared against anything. The table below reports what each of nine providers a European buyer would shortlist publishes on its own public page.

ProviderPublished P99 latency figureStated measurement basis (TTFT / ITL / end-to-end)Batch size stated
Exoscale–––
Hetzner–––
IONOS–––
Nebius–––
Open Telekom Cloud–––
OVHcloud–––
Scaleway–TTFT only, no percentile: "response time below 200ms to get the first tokens streamed" (Generative APIs, read 30 Sep 2026)–
STACKIT–––
TensorX–––

A dash means the provider's own public page lists no value for that column. None of the nine publishes a P99 figure or a batch size. The one latency figure found is Scaleway's: its Generative APIs page promises a "response time below 200ms to get the first tokens streamed"[1] (read 30 September 2026), which is a time-to-first-token claim with no percentile and no batch size. The number a latency budget actually breaks on is therefore not available from vendor documentation for any of the nine.

How percentile latency is computed

Percentile latency is computed from the sorted set of individual request latencies collected in a measurement window. Every request is timed on the same measurement basis, the latencies are ordered from fastest to slowest, and the percentile is the latency at a given position in that ordered list. A P50 (the median) is the request in the middle; a P99 is the position near the end, so only the slowest tail of requests falls beyond it. Because the value comes from the position in a sorted list, not from an average, a single slow request cannot be averaged away: it either sits inside the tail or it does not.

Why the tail matters more than the mean

The mean hides the requests that define user experience. A worked sample from NVIDIA's trtllm-bench documentation, published on 7 July 2025, shows a Llama-3.1 8B run at 128-token input and output lengths in which the P99 time-to-first-token (232.4 ms) sits well above the P50 median (159.8 ms), read 30 September 2026:[2] the tail request waited substantially longer than the typical request. The post states explicitly that the sample output is not a performance claim, and it is presented here only as an illustration of percentile spread.

What a published P99 omits

The batch size behind the number

The recurring error on this topic is a benchmark run at a batch size no reader will ever use. LLM-Inference-Bench (Argonne National Laboratory, arXiv 2411.00136, October 2024; read 30 September 2026) measured throughput 26.6 times higher at batch size 64 than at batch size 1 for LLaMA-3-8B at a 2048-token length on an Nvidia A100.[3] NVIDIA's trtllm-bench concurrency sweep (NVIDIA Technical Blog, 7 July 2025; read 30 September 2026) shows the same effect from the user's side: an FP8-quantized Llama-3.1 8B checkpoint served 512 concurrent users at about 66 tokens per second per user, while the FP16 baseline served about 256 users at about 72 tokens per second per user.[2] A published figure rarely states which point on that curve it came from, and a latency number on a vendor page cannot be verified as a measured result rather than a marketing figure.

  • Batch size and concurrency at which the percentile was measured
  • Input and output sequence lengths
  • Quantization of the checkpoint (FP8, FP16 or other)
  • Whether the basis is TTFT, inter-token latency or end-to-end request latency

Independent trackers do not close this gap. Artificial Analysis publishes output tokens per second as a median across its model evaluations, not a P99.[4] Its benchmarking methodology states that performance measurements are represented as the median (P50) over the past 72 hours.[4]

Measuring your own tail latency

A defensible P99 comes from a benchmark the reader runs under a stated configuration. Two open-source utilities do this directly: trtllm-bench from the TensorRT-LLM project, and the benchmarking tools shipped with vLLM, whose documentation (read 30 September 2026) lists support for more than 200 model architectures, continuous batching and PagedAttention, the last two of which shape the tail.[5] How the resulting percentiles feed a service-level commitment is set out in the companion piece on the latency budget.

  • Batch size and concurrency level of the run
  • Input and output sequence lengths, in tokens
  • Quantization and serving framework version
  • The retrieval date, recorded beside every percentile

The EU boundary is clear: the AI Act (Regulation (EU) 2024/1689, published in the Official Journal on 12 July 2024) sets no latency threshold, so P99 remains a procurement criterion rather than a regulatory one.[6] European buyers specify it anyway because compute is becoming a procurement question: Eurostat reports that 45.2% of EU businesses used cloud services in 2023, and the European Commission's Cloud and AI Development Act proposal of June 2026 aims to at least triple EU data centre capacity within five to seven years.[7] Verification obligations for a provider's data processing terms are covered in the GDPR verification guide, and region and pricing disclosures for European serverless inference providers are compared separately. The cost side of self-hosting a benchmark on Nvidia platforms has its own cost drivers.

  1. Cite a P99 only with its measurement basis, batch size and the date it was retrieved.
  2. Treat any published latency figure that omits its batch size as a median at an unknown operating point, not a tail.
  3. Prefer a self-run percentile under production-shaped concurrency to any vendor-published number.

Sources

  1. scaleway.com
  2. developer.nvidia.com
  3. arxiv.org
  4. artificialanalysis.ai
  5. docs.vllm.ai
  6. artificialintelligenceact.eu
  7. digital-strategy.ec.europa.eu

Rankings and scores on Optinest cannot be bought — no commercial relationship changes how a provider is assessed. How we score

Related reading

All Inference →