Fastest LLM inference provider: Speed and benchmarks
Compare the fastest LLM inference providers and benchmarks, including hardware performance for H100 GPUs and engine comparisons between vLLM and TensorRT.
In this article10
- Fastest LLM inference provider: Speed and benchmarks
- Defining LLM inference speed: TTFT and throughput
- What defines inference speed: TTFT vs decode throughput
- Open inference frameworks: vLLM and TensorRT-LLM
- Specialised silicon versus GPU clustering
- Speed benchmark: Groq, Together AI and Fireworks AI
- The impact of concurrency on latency variation
- Jurisdiction and data sovereignty for EU inference
- Frequently asked questions
- Sources

Fastest LLM inference provider: Speed and benchmarks
Defining LLM inference speed: TTFT and throughput
Evaluating Large Language Model (LLM) inference speed requires distinguishing between two primary performance metrics: Time-to-First-Token (TTFT) and output throughput. TTFT measures the latency between submitting a request and receiving the initial output token from the server, capturing prompt processing and pre-fill execution speed. Output throughput measures the rate at which the model streams subsequent generated tokens, expressed in tokens per second per stream or total system output.
| Performance Metric | Definition | Primary Architectural Bottleneck | Representative Benchmark Target |
|---|---|---|---|
| Time to First Token (TTFT) | Duration from HTTP request dispatch to initial token receipt | Memory bandwidth and prompt pre-fill compute | Under 100 ms per Groq's own LPU claims; measured medians are higher |
| Output Throughput | Rate of output token generation per second | Tensor core compute and continuous batching efficiency | 200 to 300+ tok/s on optimized Llama 3 architectures |
Specialized hardware architectures such as Groq's Language Processing Unit (LPU) focus on minimizing TTFT by utilizing custom SRAM memory routing, achieving sub-100 millisecond initial token responses for real-time applications[1]. However, for high-concurrency enterprise workloads, evaluating LLM cost per token requires balancing low initial latency against sustained generation throughput across full context windows.
What defines inference speed: TTFT vs decode throughput
Large language model (LLM) inference performance is governed by two distinct operational phases: the prefill phase and the autoregressive decode phase. Time-to-first-token (TTFT) measures the time from query submission to the first received token[3]. TTFT is dominated by the prefill phase, where the inference engine ingests and computes prompt key-value (KV) matrices in parallel. Because prompt evaluation is compute-bound, matrix multiplication across large input contexts scales primarily with raw floating-point operations per second (FLOPs) and effective tensor parallel processing across GPU clusters.
In contrast, decode throughput - measured in output tokens per second per user or aggregate tokens per second across concurrent streams - dictates performance during sequential generation. The vLLM team identifies memory, not compute, as the bottleneck in autoregressive decoding, because every generated token requires the attention key and value tensors of all preceding tokens to be held and read from GPU memory[4]. For every generated token, the inference processor must load all model weights from memory into active compute registers alongside the historical KV cache. Consequently, systems designed for high decode throughput prioritize memory bandwidth and efficient KV cache management over raw compute density.
Open inference frameworks: vLLM and TensorRT-LLM
Organizations seeking to prevent engine lock-in rely on open-source inference frameworks. The two dominant open execution stacks in enterprise production are vLLM, driven by community-developed PagedAttention, and NVIDIA TensorRT-LLM, built around optimized C++ runtimes[5].
| Evaluation Dimension | vLLM Engine | NVIDIA TensorRT-LLM Engine |
|---|---|---|
| Cold Start Prep Time | Immediate (weights load directly into memory) | 25 to 40 minutes required for engine compilation on 70B models |
| Optimization Architecture | Dynamic PagedAttention KV-cache management | Static C++ execution graph with ahead-of-time optimization |
| Deployment Suitability | Rapid iteration and multi-model routing | Fixed production models running under steady high concurrency |
To resolve memory bottlenecks during the decode phase, modern serving engines employ specialized virtual memory abstractions. PagedAttention, introduced in the vLLM architecture, partitions the KV cache of each sequence into blocks that need not be contiguous in memory space[4]. The vLLM team reports that this leaves memory waste of under 4%, letting the system batch more sequences together, raise GPU utilization and significantly increase throughput without model architecture changes[4].
Specialised silicon versus GPU clustering
Evaluating hardware platforms for AI inference requires assessing memory bandwidth, memory capacity, and interconnect topologies. Standard enterprise GPUs, such as the NVIDIA H100 and the Blackwell-based B200, rely on High Bandwidth Memory (HBM). The NVIDIA B200 features up to 192 GB of HBM3e delivering up to 8 TB/s of memory bandwidth[7]. While HBM offers high memory density suited for hosting parameter-heavy foundation models on a single node, its bandwidth remains a physical ceiling for single-user decode speed.
To bypass HBM bandwidth limits, specialized processor architectures employ Static Random-Access Memory (SRAM) and wafer-scale integration. Language Processing Units (LPUs), such as Groq silicon, eliminate DRAM by storing model weights entirely within high-speed SRAM distributed across interconnected chips[8]. Groq states that its on-chip SRAM reaches memory bandwidth upwards of 80 terabytes per second, against about eight terabytes per second for off-chip GPU HBM[9]. Similarly, wafer-scale engines such as the Cerebras CS-3 integrate on-wafer SRAM; Cerebras claims the WSE-3 processor carries 7,000 times more memory bandwidth than an NVIDIA H100[10].
| Hardware Class | Memory Technology | Peak Bandwidth Profile | Memory Capacity per Device |
|---|---|---|---|
| Enterprise GPU (NVIDIA B200) | HBM3e | Up to 8 TB/s per GPU | Up to 192 GB HBM3e |
| Custom LPU (Groq silicon) | On-chip SRAM | Upwards of 80 TB/s on-chip | Hundreds of megabytes of on-chip SRAM used as primary weight storage rather than cache |
| Wafer-Scale Engine (Cerebras CS-3) | On-wafer SRAM | 7,000x the memory bandwidth of an NVIDIA H100, per Cerebras | Single wafer-scale processor per CS-3 system |
Speed benchmark: Groq, Together AI and Fireworks AI
On Artificial Analysis's provider benchmark for Llama 3.3 70B, Groq's LPU infrastructure is the fastest endpoint in the set at 289.5 output tokens per second, ahead of every GPU-hosted endpoint in the same comparison. Groq's own architecture materials attribute that lead to compiler-scheduled, deterministic execution on on-chip memory, which keeps per-step timing predictable and lets tokens start streaming almost immediately in interactive sessions[9].
| Architecture | Primary Silicon | Key Optimisation Strategy | Primary Advantage |
|---|---|---|---|
| Custom ASIC | Groq LPU | Compiler-scheduled execution on on-chip memory | Fastest measured output speed on Llama 3.3 70B at 289.5 tokens per second |
| Optimised GPU Cluster | NVIDIA H100 / B200 | Speculative decoding, KV cache reuse and FP8/NVFP4 quantization | Broad model coverage across 15 benchmarked providers, with speculative decoding speedups of about 1.5x to 4x |
By comparison, GPU-based providers optimize for balance across concurrency and multi-model hosting flexibility. Artificial Analysis benchmarks 15 providers for Llama 3.3 70B, and the GPU-hosted endpoints from Fireworks, Together AI and the hyperscalers cluster far below the custom-silicon leader on output speed while offering broader model coverage and features such as JSON mode. Fireworks narrows the latency gap with adaptive speculative decoding, in which a smaller draft model proposes tokens that the target model verifies in parallel. To understand how these operational trade-offs impact overall inference API costs, consult our analysis on token economics.
The impact of concurrency on latency variation
Single-stream latency benchmarks do not capture how serverless inference endpoints perform under heavy concurrent user traffic. When multiple simultaneous API requests hit an inference cluster, hardware resource contention affects both prefill and decode stages.
Median (p50) latency figures often obscure severe tail latency degradations (p95 and p99). Under elevated concurrency levels, custom silicon configurations with constrained SRAM footprints can experience scheduling backpressure or queueing bottlenecks, causing tail response times to increase substantially.
GPU clusters utilize dynamic continuous batching and high-capacity High Bandwidth Memory (HBM) pools, allowing them to absorb incoming query spikes with more predictable latency scaling. While single-user TTFT may be higher than specialized ASICs, sustained aggregate throughput under concurrent load remains more stable on high-memory GPU architectures.
Jurisdiction and data sovereignty for EU inference
| Model Class | Per-Token Pricing (Input / Output) | Output Price vs Proprietary Frontier Baseline | Context Window |
|---|---|---|---|
| Open-Weight Lightweight | $0.15 / $0.20 per 1M tokens | $0.20 vs $25.00 per 1M output tokens | 256K tokens |
| Open-Weight Frontier | $1.75 / $3.50 per 1M tokens | $3.50 vs $25.00 per 1M output tokens | 1M tokens |
| Proprietary Frontier Baseline (Claude Opus 4.6) | $5.00 / $25.00 per 1M tokens | Baseline: $25.00 per 1M output tokens | 200K tokens |
When evaluating global inference providers, speed and cost metrics must be weighed against legal control and compliance obligations. Routing sensitive corporate prompts or European customer personal data to globally hosted inference endpoints introduces regulatory risk under General Data Protection Regulation (GDPR) mandates.
United States headquartered cloud and API providers fall under the jurisdiction of the Clarifying Lawful Overseas Use of Data (CLOUD) Act. Under this statutory framework, US law enforcement authorities can compel US parent entities to provide access to data stored on overseas servers, irrespective of local physical data residency claims. Consequently, executing inference on European edge nodes owned or operated by US parent organizations does not grant total legal immunity from extraterritorial warrant requests.
Frequently asked questions
What is the fastest LLM inference API?
Speed depends on the metric. On Artificial Analysis's Llama 3.3 70B provider benchmark, custom silicon leads output speed, with Groq at 289.5 output tokens per second, while the lowest time-to-first-token in that set belongs to a GPU-hosted endpoint at 0.69 s. For batch processing, aggregate throughput on GPU clusters from providers such as Together AI or Fireworks AI becomes the primary factor.
How does Together AI compare to Groq for speed?
Groq is faster for single-stream generation because of its Language Processing Unit architecture: 289.5 output tokens per second and 0.93 s median first-chunk latency on Llama 3.3 70B, against roughly 102 tokens per second and 1.63 s for Together AI's endpoint. Together AI's high-capacity HBM pools and continuous batching, however, scale more predictably under high concurrency.
Why is time-to-first-token important for voice agents?
Voice applications cannot begin text-to-speech processing until the LLM returns its first text output. With speech-to-text endpointing and text-to-speech each consuming roughly 150 ms of a one-second turn budget, an LLM time-to-first-token above about 600 ms makes the agent feel slow regardless of model quality.
Sources
- Blog | Groq is the premier neocloud for fast inference groq.com
- Metrics docs.nvidia.com
- vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention blog.vllm.ai
- vLLM vs TensorRT-LLM vs SGLang: Which Is Fastest? (H100 Benchmarks, 2026) spheron.network
- NVIDIA B200 Specs and Price: 192GB Blackwell GPU for AI (2026) | AI FAQ jarvislabs.ai
- Cerebras Delivers Record-Breaking Performance with Meta's Llama 3.1-405B Model cerebras.ai
Rankings and scores on Optinest cannot be bought — no commercial relationship changes how a provider is assessed. How we score
Related reading
All Accelerators →The H100 is no longer the top of the stack
Compare the successors to the NVIDIA H100. Evaluate the H200, B200, and B300 GPUs based on memory bandwidth, capacity, and inference performance.

Inference engines compared: what eight serving engines are for
vLLM, TensorRT-LLM, TGI, SGLang and four more inference engines compared on hardware, licence and maturity, with every benchmark's batch size named.

LLM APIs with zero data retention: A provider comparison
Compare 7 LLM inference providers that offer zero data retention. Learn what each commitment covers and how to get API logging restrictions in writing.
