Will inference become cheaper?
Discover why AI inference costs are dropping rapidly and how enterprises can leverage open-weight models and falling per-token prices to scale production.
In this article12
- Will inference become cheaper?
- The trajectory of AI inference prices
- Drivers behind the cost reduction per token
- Algorithmic progress and model quantization
- The impact of open-weight models on token pricing
- Agentic workflows and the rising volume of tokens
- Will AI inference get cheaper across every task?
- Strategies for optimizing production inference costs
- Total cost of ownership: Hosted versus API
- What open-weight inference costs per token today
- Frequently asked questions
- Sources

Will inference become cheaper?
The trajectory of AI inference prices
Across artificial intelligence infrastructure, the unit cost of serving trained models has fallen rapidly. Andreessen Horowitz's LLMflation analysis reports an order-of-magnitude decline in the cost of inference every year for models of equivalent benchmark performance[1]. Between late 2021 and late 2024, the price to reach GPT-3-level quality (roughly 42[1] on the Massive Multitask Language Understanding benchmark) fell about 1,000x, from $60.00 per million tokens to around $0.06[1] per million tokens, a pace of roughly 10x per year[2].
API prices for large language model (LLM) inference have dropped rapidly since 2021. In evaluating this trajectory, it is necessary to distinguish inference from model training. Training represents a fixed capital expenditure to optimize model parameters, whereas inference is an ongoing operational expenditure required to evaluate tokens for end users. Front-loading initial price milestones illustrates the shift: GPT-4 cost about $36 per million tokens on a blended basis at its initial release in March 2023, and by August 2024 GPT-4o tokens cost $4 per million on the same blended basis[3]. That reduction over 17 months corresponds to about a 79% drop in price per year[3].
Long-term industry forecasts indicate that baseline inference efficiency will continue to advance through the end of the decade. Research firm Gartner forecasts that by 2030, performing inference on a large language model with one trillion parameters will cost GenAI providers over 90 percent less than in 2025[4]. This ongoing cost decline is driven by specialized inference ASICs, improved chip utilization, and architectural innovations[4].
Drivers behind the cost reduction per token
The sustained decline in per-token inference prices stems from compounding technical efficiency gains across hardware architectures, model compilation, and competitive market dynamics. On the hardware layer, successive GPU generations deliver better cost per operation, a result of both rising transistor counts and structural design improvements[1].
- Model quantization: inference was initially done at 16-bit precision, while 4-bit precision is expected to become common on Blackwell-generation GPUs, a net performance increase of at least 4x because less data movement is required and arithmetic units are simpler[1].
- Architectural efficiency: a 1-billion-parameter model now exceeds the performance of a 175-billion-parameter model from three years earlier, largely because models are trained on far more tokens than Chinchilla scaling laws considered optimal[1].
- Post-training optimization: techniques such as reinforcement learning from human feedback (RLHF) and direct preference optimization (DPO) improve models after pre-training, raising capability per parameter[1].
- Open-source margin compression: open models from Meta, Mistral and others can be hosted by competing low-cost model-as-a-service providers, which reduced profit margin across the value chain and lowered prices[1].
Algorithmic progress and model quantization
Reductions in API list prices stem directly from progress in accelerator hardware and serving software. LLM inference during the autoregressive generation phase is memory-bandwidth bound. Every output token requires passing model weights through accelerator memory. Increasing memory bandwidth and optimizing precision formats directly increases the volume of tokens generated per accelerator-hour.
Quantization addresses this memory bandwidth bottleneck by converting 16-bit floating-point weights (FP16) into lower-precision representations, such as 8-bit integers (INT8) or 4-bit integers (INT4 and AWQ). Moving from FP16 to INT4 reduces memory footprint and data transfer volumes by 75%, allowing a 70-billion parameter model that required 140 GB of VRAM to fit into 35 GB[7]. In production deployments, combining 4-bit AWQ quantization with paged attention memory management (vLLM) and continuous batching increased serving throughput from 22 tokens per second to over 410 tokens per second per server instance, driving a 5-fold serving cost reduction[7].
| Quantization Precision | VRAM Footprint (70B Model) | Memory Bandwidth Savings | Serving Throughput (tokens/sec per instance) |
|---|---|---|---|
| FP16 (16-bit float) | 140 GB | 0% (baseline) | 22 tokens/sec |
| INT8 (8-bit integer) | 70 GB | 50% bandwidth reduction | Between the FP16 and INT4 figures |
| INT4 / AWQ (4-bit integer) | 35 GB | 75% bandwidth reduction | 410+ tokens/sec, a 5.0x serving cost reduction |
The impact of open-weight models on token pricing
The expansion of open-weight model releases has systematically dismantled proprietary pricing premiums across the API inference ecosystem. When OpenAI launched GPT-4 in March 2023, access was priced at $30.00[10] per million input tokens and $60.00[10] per million output tokens[2].
The subsequent release of competitive open-weight families, including Meta Llama, Alibaba Qwen, and Mistral, enabled third-party specialized hosting providers to offer equivalent reasoning capabilities at a fraction of closed-source rates. By mid-2026, inference costs for GPT-4-class performance collapsed by approximately 95%[10], bringing commercial token pricing down to under $0.50 per million tokens[2]. Enterprise procurement teams analyzing now evaluate open-weight models as direct substitutes for proprietary API endpoints.
| Model & Architecture | Provider rate card | Input Price / 1M Tokens | Output Price / 1M Tokens |
|---|---|---|---|
| gpt-oss-120b (open weight) | OVHcloud AI Endpoints | €0.08 | €0.40 |
| gpt-oss-120b (open weight) | IONOS AI Model Hub | $0.17 | $0.71 |
| gpt-oss-120b (open weight) | STACKIT AI Model Serving, LLM-Plus tier | €0.45 | €0.65 |
| Proprietary Frontier Tier (e.g. Claude Opus equivalent) | Closed API Tier | $5.00 | $25.00 |
Agentic workflows and the rising volume of tokens
However, falling unit token prices do not automatically lead to reduced total enterprise expenditure. Gartner notes that agentic models require between 5 and 30 times more tokens per task than standard conversational chatbots due to multi-step reasoning, tool invocations, and iterative verification loops[4]. Consequently, expanding token volume offsets unit cost drops.
Single-turn queries process a prompt and a single response, so token consumption per interaction stays small. Autonomous agents instead operate in multi-step execution loops, conducting search, tool execution, output verification, and replanning[8]. A single user task assigned to an agentic system can therefore trigger dozens of sequential model calls, multiplying the tokens consumed before the task completes.
Will AI inference get cheaper across every task?
While aggregate inference costs are dropping, deflation rates vary significantly depending on task complexity and reasoning requirements. Research institute Epoch AI analysed the cheapest models above each performance threshold on six benchmarks and fitted a log-linear price trend at each threshold; across all of those benchmarks and thresholds it found prices declining between 9x per year and 900x[9] per year, with a median of 50x per year[9]. Epoch AI's underlying price dataset was last updated on 20 November 2025[9], and the figures below were retrieved and verified against that dataset on 30 July 2026, with the next review scheduled for 30 January 2027.
Standard text generation, sentiment analysis, and basic code synthesis tasks have experienced the fastest cost erosion. Conversely, specialized reasoning models that utilize test-time compute scaling (such as chain-of-thought verification loops) command a price premium. Reasoning models generate multiple hidden intermediate reflection tokens per output token, maintaining higher operational costs even as base silicon efficiency improves.
Strategies for optimizing production inference costs
Bain & Company's token economics analysis indicates that while average per-token costs fell by 50 percent from December 2024 to December 2025, token consumption grew by 4.5 times over the same period[6]. To prevent total expenditures from rising alongside usage, organizations implement model-routing strategies. Enterprise deployments demonstrate that reorchestrating simple requests away from frontier models to smaller worker models yields up to a 90 percent reduction in operational cost[6].
- Prompt Caching: Storing static prompt headers and reference documents yields 50% to 90% cost reductions on repeated input tokens[8].
- Model Routing: Sending routine classification and formatting requests to lightweight models while reserving frontier models for reasoning reduces total API spend by up to 85%[8].
- Output Control: Output tokens cost 4x to 6x more than input tokens[8]; enforcing structured JSON schemas and tight output limits prevents unnecessary generation.
- Batch Pricing: Non-real-time work submitted to batch API endpoints is billed at half the standard list price[8].
Total cost of ownership: Hosted versus API
Enterprise decision-makers face a strategic choice between acquiring dedicated GPU infrastructure (self-hosted or leased VMs) and consuming serverless managed API endpoints. Evaluating this decision requires factoring in the 50x to 200x annual token deflation curve against fixed hardware amortization periods.
Procuring dedicated GPU clusters commits an enterprise to fixed capital expenditure (CapEx) or multi-year leased operational expenditure (OpEx). Given a standard 3-year GPU hardware depreciation cycle, a cluster purchased today risks economic obsolescence within 12 to 18 months as newer accelerator architectures and software runtimes reduce per-token serving costs for competitors. Organizations must also manage GPU hardware costs, idle capacity waste during non-peak hours, electricity tariffs, and specialized MLOps staffing.
| Deployment Model | Cost Structure | Deflation Capture | Best Suited Workload |
|---|---|---|---|
| Dedicated GPU Cluster | Fixed CapEx / OpEx (3-year cycle) | Locked into procurement hardware efficiency | Predictable 24/7 baseline demand at maximum utilization |
| Serverless API | Variable Pay-Per-Token | Automatically captures 50x-200x annual deflation | Fluctuating traffic, agentic loops, enterprise applications |
Consuming serverless inference APIs shifts infrastructure risk to specialized providers. Under a serverless pay-per-token structure, enterprise buyers automatically absorb ongoing market price declines without writing off physical assets. Self-hosting remains economically advantageous only when an organization sustains near-continuous, round-the-clock cluster utilization or faces strict regulatory isolation constraints.
What open-weight inference costs per token today
Published rate cards make the current floor concrete. The same open-weight model, gpt-oss-120b, is listed at €0.08 input / €0.40 output per million tokens on OVHcloud's AI Endpoints catalogue, $0.17 / $0.71 on the IONOS AI Model Hub and €0.45 / €0.65 in STACKIT's LLM-Plus tier, each read 14 September 2026 and quoted in the currency its provider publishes. Selecting a cost-efficient model per request then aligns inference spend with query complexity.
An open-weight model served through a European endpoint, gpt-oss-120b, is listed at €0.08 per million input tokens and €0.40 per million output tokens on OVHcloud's AI Endpoints catalogue . Set against the $30.00 [10] input and $60.00 [10] output pricing that GPT-4 carried at its March 2023 launch, that is the same order-of-magnitude collapse the benchmark data describes, now visible in a live rate card [2] . The per-token rates in the table below were retrieved from published price lists on 14 September 2026; because list prices move, each carries the date it was read.
| Model Name | Provider rate card | Input Price (per 1M) | Output Price (per 1M) |
|---|---|---|---|
| gpt-oss-120b | OVHcloud AI Endpoints | €0.08 | €0.40 |
| gpt-oss-120b | IONOS AI Model Hub | $0.17 | $0.71 |
| gpt-oss-120b | STACKIT AI Model Serving, LLM-Plus tier | €0.45 | €0.65 |
| Meta-Llama-3.3-70B-Instruct | OVHcloud AI Endpoints | €0.67 | €0.67 |
Falling list prices are only half of the cost equation: what a workload actually spends depends on which model each request lands on. Smart-routing endpoints route requests across the provider's own capacity to keep availability high; choosing a cheaper model for straightforward calls and a larger target for complex queries remains the caller's decision. For a cost-driven buyer, that mix effect can move a token bill as much as the underlying market deflation does.
Frequently asked questions
How much have LLM inference prices fallen since 2023?
Prices have dropped exponentially. According to analysis by Andreessen Horowitz, the cost of running a large language model at a fixed capability level has fallen by a factor of 1,000 over three years, far outpacing historical computing cost deflation rates.
Are open-weight models cheaper to run than proprietary ones?
Yes. Open-weight models from developers like Meta and Mistral are hosted by multiple competing providers, which compresses profit margins and drives down the per-token price. Consequently, GPT-4-class inference now costs under $0.50 per million tokens through open-weight providers.
Should enterprises self-host models or use serverless APIs?
For most enterprises, serverless APIs offer a better total cost of ownership. Because inference prices at a fixed capability level have been falling at a median rate of 50x per year according to Epoch AI, locking into a multi-year hardware depreciation cycle for on-premise GPUs forces an organisation to pay a premium for fixed capabilities.
How does reasoning affect per-token inference pricing?
While straightforward chat completion costs have plummeted, complex reasoning models that run long chains of thought consume significantly more compute. The market is stratifying, meaning that simple generation tasks ride the deflation curve while frontier reasoning capabilities still demand a premium.
Sources
- Welcome to LLMflation - LLM inference cost is going down fast ⬇️ | Andreessen Horowitz a16z.com
- AI Inference Cost Reduction 2026: Down 95% in Two Years valueaddvc.com
- Falling LLM Token Prices and What They Mean for AI Companies deeplearning.ai
- Gartner Forecasts 90% Drop in LLM Inference Costs by 2030 hpcwire.com
- How Token Economics Will Change Opex bain.com
- AI Inference Cost Optimization Math & Efficiency Guide branch8.com
- Cut LLM API Costs 60% Without Changing Models tokencost.app
- LLM inference prices have fallen rapidly but unequally across tasks epoch.ai
Rankings and scores on Optinest cannot be bought — no commercial relationship changes how a provider is assessed. How we score
Related reading
All Inference →What proprietary tokens cost over open weights
Compare LLM cost per token across proprietary and open-weight models in 2026. Discover how caching, batch processing, and routing cut inference API prices.

The token price war has a floor: three cents
Compare the cheapest LLM API token pricing, from budget models at $0.03 per million tokens to open-weight frontier options, and optimize inference costs.

How much is 1 million tokens in an LLM? Pricing guide
Discover how much 1 million AI tokens cost in 2026. Compare LLM API pricing for ChatGPT, Claude, and open-weight models to optimise your inference bill.
