P99 latency for LLM inference: what it reports and omits
P99 latency for LLM inference explained: what the percentile reports, why batch size distorts published figures, and how to measure your own tail latency.
In this article6

What P99 latency actually reports
P99 latency is the value that almost all requests complete faster than; only the slowest tail of requests takes longer. For LLM inference, the number depends on which of three measurement bases it sits on, the same three timed in Argonne National Laboratory's LLM-Inference-Bench (2024): time-to-first-token (TTFT), the time to produce the first output token; inter-token latency (ITL), the average interval between consecutive output tokens; and end-to-end request latency, the time from prompt to final token.
The three bases differ by roughly an order of magnitude for the same model and hardware, so a P99 figure without its basis cannot be compared against anything. The table below reports what each of nine providers a European buyer would shortlist publishes on its own public page.
| Provider | Published P99 latency figure | Stated measurement basis (TTFT / ITL / end-to-end) | Batch size stated |
|---|---|---|---|
| Exoscale | – | – | – |
| Hetzner | – | – | – |
| IONOS | – | – | – |
| Nebius | – | – | – |
| Open Telekom Cloud | – | – | – |
| OVHcloud | – | – | – |
| Scaleway | – | TTFT only, no percentile: "response time below 200ms to get the first tokens streamed" (Generative APIs, read 30 Sep 2026) | – |
| STACKIT | – | – | – |
| TensorX | – | – | – |
A dash means the provider's own public page lists no value for that column. None of the nine publishes a P99 figure or a batch size. The one latency figure found is Scaleway's: its Generative APIs page promises a "response time below 200ms to get the first tokens streamed"[1] (read 30 September 2026), which is a time-to-first-token claim with no percentile and no batch size. The number a latency budget actually breaks on is therefore not available from vendor documentation for any of the nine.
How percentile latency is computed
Percentile latency is computed from the sorted set of individual request latencies collected in a measurement window. Every request is timed on the same measurement basis, the latencies are ordered from fastest to slowest, and the percentile is the latency at a given position in that ordered list. A P50 (the median) is the request in the middle; a P99 is the position near the end, so only the slowest tail of requests falls beyond it. Because the value comes from the position in a sorted list, not from an average, a single slow request cannot be averaged away: it either sits inside the tail or it does not.
Why the tail matters more than the mean
The mean hides the requests that define user experience. A worked sample from NVIDIA's trtllm-bench documentation, published on 7 July 2025, shows a Llama-3.1 8B run at 128-token input and output lengths in which the P99 time-to-first-token (232.4 ms) sits well above the P50 median (159.8 ms), read 30 September 2026:[2] the tail request waited substantially longer than the typical request. The post states explicitly that the sample output is not a performance claim, and it is presented here only as an illustration of percentile spread.
What a published P99 omits
The batch size behind the number
The recurring error on this topic is a benchmark run at a batch size no reader will ever use. LLM-Inference-Bench (Argonne National Laboratory, arXiv 2411.00136, October 2024; read 30 September 2026) measured throughput 26.6 times higher at batch size 64 than at batch size 1 for LLaMA-3-8B at a 2048-token length on an Nvidia A100.[3] NVIDIA's trtllm-bench concurrency sweep (NVIDIA Technical Blog, 7 July 2025; read 30 September 2026) shows the same effect from the user's side: an FP8-quantized Llama-3.1 8B checkpoint served 512 concurrent users at about 66 tokens per second per user, while the FP16 baseline served about 256 users at about 72 tokens per second per user.[2] A published figure rarely states which point on that curve it came from, and a latency number on a vendor page cannot be verified as a measured result rather than a marketing figure.
- Batch size and concurrency at which the percentile was measured
- Input and output sequence lengths
- Quantization of the checkpoint (FP8, FP16 or other)
- Whether the basis is TTFT, inter-token latency or end-to-end request latency
Independent trackers do not close this gap. Artificial Analysis publishes output tokens per second as a median across its model evaluations, not a P99.[4] Its benchmarking methodology states that performance measurements are represented as the median (P50) over the past 72 hours.[4]
Measuring your own tail latency
A defensible P99 comes from a benchmark the reader runs under a stated configuration. Two open-source utilities do this directly: trtllm-bench from the TensorRT-LLM project, and the benchmarking tools shipped with vLLM, whose documentation (read 30 September 2026) lists support for more than 200 model architectures, continuous batching and PagedAttention, the last two of which shape the tail.[5] How the resulting percentiles feed a service-level commitment is set out in the companion piece on the latency budget.
- Batch size and concurrency level of the run
- Input and output sequence lengths, in tokens
- Quantization and serving framework version
- The retrieval date, recorded beside every percentile
The EU boundary is clear: the AI Act (Regulation (EU) 2024/1689, published in the Official Journal on 12 July 2024) sets no latency threshold, so P99 remains a procurement criterion rather than a regulatory one.[6] European buyers specify it anyway because compute is becoming a procurement question: Eurostat reports that 45.2% of EU businesses used cloud services in 2023, and the European Commission's Cloud and AI Development Act proposal of June 2026 aims to at least triple EU data centre capacity within five to seven years.[7] Verification obligations for a provider's data processing terms are covered in the GDPR verification guide, and region and pricing disclosures for European serverless inference providers are compared separately. The cost side of self-hosting a benchmark on Nvidia platforms has its own cost drivers.
- Cite a P99 only with its measurement basis, batch size and the date it was retrieved.
- Treat any published latency figure that omits its batch size as a median at an unknown operating point, not a tail.
- Prefer a self-run percentile under production-shaped concurrency to any vendor-published number.
Sources
Rankings and scores on Optinest cannot be bought — no commercial relationship changes how a provider is assessed. How we score
Related reading
All Inference →Latency budgets, and what providers publish
How to set an inference latency budget: TTFT, inter-token latency, batch size, and what nine European inference providers publish.

Inference engines compared: what eight serving engines are for
vLLM, TensorRT-LLM, TGI, SGLang and four more inference engines compared on hardware, licence and maturity, with every benchmark's batch size named.

The 70/20/10 and 30% rules for AI budgets: what they claim
What the 70/20/10 and 30% rules for AI budgets actually say, who published them, and which dated spending figures can replace them in a plan.
