Latency in AI: How to Reduce TTFT and Network Delays
Learn how to reduce AI inference latency by optimizing Time to First Token (TTFT), implementing continuous batching, and selecting regional EU providers.
In this article8
- The anatomy of AI inference latency: TTFT versus TPOT
- The physics of network delays and propagation limits
- Model quantization for VRAM reduction
- Continuous batching and attention caching
- Speculative decoding for long-form workloads
- Criteria for selecting low-latency EU providers
- Frequently asked questions
- Sources

The anatomy of AI inference latency: TTFT versus TPOT
Latency in AI model inference splits into two phases with different hardware constraints: compute-bound prefill and memory-bandwidth-bound decode. Total response latency equals the Time to First Token (TTFT) plus the product of generated tokens and Time Per Output Token (TPOT). Evaluating a platform means isolating both metrics rather than relying on aggregate end-to-end times.
The compute-bound prefill phase
The prefill phase ingests the entire prompt simultaneously to construct the Key-Value (KV) cache. Because matrix multiplications across all input tokens execute in parallel, this step fully saturates GPU tensor cores and scales primarily with sequence length. Long prompts therefore consume dedicated compute time before the first output token appears. Under heavy batch loads, queueing delays compound this baseline, directly inflating TTFT.
The memory-bound decode phase
Once prefill completes, the model enters autoregressive decoding, emitting one token per step. Each step requires loading all model weights and the accumulated KV cache from High Bandwidth Memory (HBM) into the compute cores. Because compute intensity per token is low, TPOT is bounded by memory bandwidth rather than raw FLOPS. A stream delivering 20 to 30 tokens per second corresponds to an inter-token latency of 33 to 50 milliseconds per token.
| Phase | Dominant Bottleneck | Primary Metric | Key Influencing Factors |
|---|---|---|---|
| Prefill | Compute (FLOPS) & KV initialisation | Time to First Token (TTFT) | Prompt length, queue time, network transit |
| Decode | Memory Bandwidth (HBM) | Time Per Output Token (TPOT) | Model parameter size, quantization, batch size |
| End-to-End | Composite pipeline | Total generation duration | Output token count, generation stopping criteria |
Perceived responsiveness in interactive applications depends almost entirely on TTFT: once streaming starts, throughput above roughly 20 tokens per second already outpaces what a reader consumes. In agentic workflows, TPOT and total execution time dominate instead.
The physics of network delays and propagation limits
Before server-side computation begins, physical distance establishes an immutable lower bound on latency. Light travels roughly 30% slower through the silica glass core of an optical fiber than through a vacuum, because of the core's group refractive index[1] (about 1.4682 for standard single-mode telecom fiber at 1550 nm)[2]. Standard telecom fiber therefore introduces approximately 4.9 microseconds of propagation delay per kilometre[1], which works out to roughly 5 milliseconds for every 1,000 kilometres of one-way travel[2].
In real networks, routing, optical switching and border hops add overhead on top of that floor. A Frankfurt-to-US-East round trip therefore costs far more transit latency than an intra-European path before compute begins, and new TLS handshakes add further round trips.
- Physical fiber propagation: ~5 ms per 1,000 km one-way transit delay
- Network routing and peering hops: typically 10 to 30 ms of additional transit overhead
- TLS 1.3 handshake negotiation: 1 full round-trip time (RTT) on initial connection
- Inference API gateway processing: typically 10 to 50 ms for routing, authentication, and rate limits
A fundamental architectural contrast separates training from inference within AI infrastructure networking. Distributed model training requires strict microsecond-level synchronization across thousands of GPUs, confining training clusters to a single facility or metro campus. In contrast, inference requests are stateless and decoupled, permitting regional distribution across local points of presence to minimize the user-to-accelerator optical distance.
Model quantization for VRAM reduction
Reducing the precision of model weights from 16-bit brain floating point (BF16) to 8-bit (FP8) or 4-bit formats directly alleviates the memory-bandwidth bottleneck during decoding. Because each decode step moves the whole parameter set through GPU memory, cutting weight size cuts memory traffic per token proportionally.
Precision trade-offs and memory footprint
Activation-aware Weight Quantization (AWQ) works from the finding that protecting only the top 1% of salient weight channels, identified from the activation distribution rather than the weights themselves, greatly reduces quantization error[3]. The remaining channels compress to 4-bit integers, sharply shrinking the VRAM footprint relative to BF16. A 70B model needing 140 GB of VRAM in FP16 fits in roughly 40 GB at 4-bit, running on far fewer accelerators.
- Weight size: 4-bit compression cuts weight data moved per decode step versus FP16
- Memory bandwidth efficiency: Lower data transfer volume accelerates TPOT across batch sizes
- Hardware accessibility: Large models fit onto standard dual-GPU nodes instead of high-density 8-GPU servers
- Regional hosting: compact footprints fit low-latency regional hubs
Quantization also enables hosting on cost-effective inference hardware arrays closer to end users. By deploying quantized models in local European facilities rather than massive centralized compute farms, engineering teams simultaneously reduce both decode memory latency and wide-area network propagation delays.
Continuous batching and attention caching
Modern inference engines eliminate GPU idle states through scheduling and memory management techniques designed specifically for dynamic sequence lengths. Standard static batching pads sequences to match the longest request, wasting substantial compute capacity on pad tokens.
Continuous batching mechanisms
Continuous batching, pioneered by systems like Orca and vLLM, schedules requests at token-level granularity. As soon as a request finishes, the engine ejects its slot and admits a new one from the queue without waiting for the rest of the batch. This keeps GPU compute utilization high under steady load, substantially curtailing queueing delay and TTFT inflation.
PagedAttention and prefix reuse
PagedAttention manages the KV cache in non-contiguous memory pages, an approach inspired by classical virtual memory and paging in operating systems, which achieves near-zero waste in KV cache memory[4]. Built on top of it, the vLLM serving system improves the throughput of popular LLMs by 2 to 4 times at the same level of latency compared with prior state-of-the-art systems such as FasterTransformer and Orca[4].
RadixAttention, the KV cache reuse optimization in the SGLang runtime, advances memory reuse further by organizing cached KV tensors into a radix tree structure[5]. When prompts share system instructions, few-shot examples or document context, the engine matches the prefix against existing tree nodes and reuses the cached attention states instead of recomputing prefill. Across agent control, few-shot benchmarks, JSON decoding, retrieval-augmented pipelines, and multi-turn chat, SGLang reports up to 6.4 times higher throughput than prior state-of-the-art inference systems[5].
| Optimization Method | Target Bottleneck | Mechanism | Observed Performance Impact |
|---|---|---|---|
| Continuous Batching | Queue wait time & idle compute | Token-level dynamic request scheduling | Keeps GPU utilization high under sustained load |
| PagedAttention | Memory fragmentation | Paged KV cache allocation in non-contiguous VRAM | 2 to 4 times higher throughput at the same latency |
| RadixAttention | Prefill compute on shared prompts | Tree-structured KV cache reuse across requests | Up to 6.4 times higher throughput in SGLang benchmarks |
Speculative decoding for long-form workloads
Speculative decoding accelerates the memory-bound decode phase without modifying model weights or compromising output mathematical exactness[6]. The architecture pairs a large target model with a lightweight draft model operating over the same vocabulary.
The draft-and-verify workflow
During each generation step, the draft model rapidly generates a speculative sequence of candidate tokens (typically 3 to 5 tokens) using minimal compute overhead. The primary target model then verifies the full speculative candidate block in a single forward pass, confirming the tokens it agrees with and correcting the first one it does not, so only the validated prefix is emitted[6].
- Draft generation: Compact model proposes K candidate tokens autoregressively
- Parallel verification: Target model evaluates all K candidates concurrently in one forward execution
- Acceptance verification: Valid tokens are emitted simultaneously; the first mismatched token is corrected
- Throughput acceleration: Multiple tokens generated per target model execution pass
When draft and target models are well matched, a high share of drafted tokens survives verification on structured tasks such as code generation and summarization. In low queries-per-second testing, vLLM measured up to a 1.5x speedup in token generation with draft-model speculative decoding on ShareGPT, and up to 2.8x with prompt lookup decoding on summarization data[6]. In high queries-per-second environments, however, the extra compute needed to propose and verify tokens can slow a system that is already compute-bound; vLLM is developing dynamic speculative decoding that shortens the proposed length under load[6].
Criteria for selecting low-latency EU providers
Selecting LLM inference providers for production applications requires establishing clear criteria that account for geographical routing, specialized model architectures, and verifiable service level agreements.
Regional data residency and infrastructure placement
European workloads should prioritize providers hosting compute inside continental data center hubs such as Frankfurt, Paris or the Nordics. Local routing removes the transatlantic leg: at roughly 5 ms per 1,000 km of fiber one way, an intra-European path spends a fraction of what a Frankfurt-to-US-East round trip costs before compute begins.
Latency-optimized model variants
Inference catalogues increasingly separate compact fast-tier models from their flagship counterparts, and the fast tier is consistently priced well below the flagship on the same catalogue. The latency argument for such fast tiers is structural rather than benchmarked here: fewer parameters mean less weight data crossing the memory bus per decode step.
| Evaluation Criterion | Scoring Focus | Benchmark Threshold |
|---|---|---|
| Geographic Placement | Proximity to user base to cut optical transit | < 25 ms network RTT to primary EU regions |
| Prefix Caching Support | Radix or prefix KV cache reuse | Sub-100 ms TTFT on repeated context |
| Throughput Scaling | Continuous batching and speculative decoding support | Stable TPOT under 95th percentile load |
| SLA Transparency | Independent p95 TTFT and jitter telemetry | Published p95 latency figures, or an explicit statement that none exist |
Measure TTFT and TPOT across representative prompt sizes and concurrency levels, and audit p95 tail metrics rather than vendor-published averages.
Frequently asked questions
What is Time to First Token (TTFT) in AI inference?
Time to First Token (TTFT) is the delay between submitting a prompt and receiving the first generated token. It is driven by the compute-bound prefill phase and by network routing delays.
How much latency do optical networks add to AI inference?
Physical distance imposes a strict propagation limit of roughly 5 ms per 1,000 km in optical fiber. A transatlantic round trip therefore spends tens of milliseconds in propagation alone, before routing hops, TLS handshakes or compute are added.
What causes high Time Per Output Token (TPOT)?
Time Per Output Token (TPOT), or inter-token latency, is set by the memory-bound decode phase: the GPU reloads the model weights and the KV cache for every token, so memory bandwidth is the bottleneck, not raw compute.
How does speculative decoding reduce latency?
Speculative decoding uses a smaller, faster draft model to generate several candidate tokens at once. The large target model then verifies these tokens in a single parallel pass. In low queries-per-second testing, vLLM measured up to a 1.5x speedup with a draft model and up to 2.8x with prompt lookup decoding on summarization data.
Why is continuous batching better than static batching?
Continuous batching injects new requests the moment a compute slot opens, instead of waiting for a whole batch to finish. That keeps the GPU busy under sustained load, improving throughput without sacrificing responsiveness.
Sources
- Optical Fiber Latency & Delay Calculator ephotonics.com
- The 5-Microsecond Rule: Fiber Propagation Latency per Kilometer mapyourtech.com
- proceedings.mlsys.org
- Efficient Memory Management for Large Language Model Serving with PagedAttention ar5iv.labs.arxiv.org
- SGLang: Efficient Execution of Structured Language Model Programs arxiv.org
- How Speculative Decoding Boosts vLLM Performance by up to 2.8x vllm.ai
Rankings and scores on Optinest cannot be bought — no commercial relationship changes how a provider is assessed. How we score
Related reading
All Datacenters →Latency budgets, and what providers publish
How to set an inference latency budget: TTFT, inter-token latency, batch size, and what nine European inference providers publish.

Where to run Kimi K3 in Europe, and what it costs
Compare published Kimi K3 token prices across European inference providers, including cached rates, and learn what rate cards do not cover.

Best serverless AI for European teams
Compare ten European serverless AI inference providers based on data residency, certification, latency, and per-token pricing to make an auditable choice.
