Deploying a Hugging Face model as an API endpoint
Evaluate the routes to deploy a Hugging Face model as an API endpoint. Compare managed endpoints, serverless platforms, and VMs across European providers.
In this article7

The three routes and who owns what
A Hugging Face model becomes a production API when something runs its weights behind an HTTP endpoint. Three routes do this, and they differ in one thing: who owns the runtime, the scaling and the upgrade path.
- Managed endpoint: the provider runs the serving stack, autoscaling and upgrades. The team ships weights or selects from a catalogue and writes no serving code. Operational burden is lowest, but deployment is limited to what the provider accepts.
- Container on a serverless GPU platform: the team owns the serving code and its container image; the platform owns scale-to-zero and per-second billing. Cold starts and image size become the team's problem.
- Self-managed GPU VM: the team owns everything, including OS patching, CUDA driver versions, the serving engine, TLS termination and health checks. Full control, full burden.
A root-access VM leaves the entire stack to the team, while a serverless GPU route that runs a Python script or a Docker image keeps the serving code and its container with the team and moves only the machine underneath.
Which serving engine runs inside the chosen route is a separate decision from the route itself; for that decision, see vLLM, SGLang or Ray, where execution engines and orchestration layers are compared.
Sizing VRAM at the precision you will serve
The model card states a parameter count, not the memory a deployment needs. Weight memory is arithmetic: parameters multiplied by the bytes each parameter occupies at the serving precision. A 70-billion-parameter model at FP16, two bytes per parameter, needs about 140 GB for weights alone; the same weights at INT8 need about 70 GB, and at INT4 about 35 GB. Quantization trades memory for accuracy, so the precision the team will actually serve, not the checkpoint's native format, sets the requirement.
| Precision | Bytes per parameter | Weights, 70B model (calculation) | Example hardware that fits |
|---|---|---|---|
| FP16 / BF16 | 2 | ~140 GB | Two 80 GB GPUs or one 141 GB GPU; almost no headroom |
| INT8 | 1 | ~70 GB | One 80 GB GPU |
| INT4 | 0.5 | ~35 GB | One 48 GB GPU |
Weights are the floor, not the total. The KV cache, the memory that stores intermediate attention state for each in-flight sequence, grows with context length and with the number of concurrent requests. A deployment sized to weights alone will queue or fail requests once concurrency rises. The working rule: weights, plus KV cache at the target maximum concurrency and context length, plus activation memory, must fit in VRAM with headroom. Worked sizing tables for 7B to 70B models, including quantization formats, are published separately: see 70B VRAM footprints.
Which platforms take any model and which take a catalogue
A team with custom or fine-tuned weights needs a platform that accepts any Hugging Face model; a team serving a well-known open model can accept a fixed catalogue in exchange for less operations work. A reader who wants to serve the weights themselves can take a Lyceum on-demand GPU VM with full root access, H100 at $2.79/hr, billed per second with no minimum (read 22 September 2026)[1], and run their own serving stack on it. The table below reads each provider's own public page on that date; entries marked 30 September 2026 were re-read that day.
| Provider | Routes on the provider's pages | Model restriction on the page read | Example GPU price |
|---|---|---|---|
| Lyceum | On-demand GPU VMs with full root access; serverless GPU execution | None stated (self-managed VM) | H100 $2.79/hr |
| Exoscale | GPU instances (A30 and further GPU instance families); Dedicated Inference: any Hugging Face model (public, gated or private) served as an API on dedicated GPUs (read 30 Sep 2026) | None stated (self-managed VM); Dedicated Inference accepts any Hugging Face model | A30 €1.2265/hr (GPU instance); RTX Pro 6000 €2.15/hr on Dedicated Inference (read 30 Sep 2026) |
| Hetzner | Dedicated GPU servers, GEX line (RTX PRO 4000, 24 GB; RTX PRO 6000, 96 GB) | None stated (self-managed server) | GEX45 (RTX PRO 4000) €0.3429/hr, max €214.00/month, excl. VAT (read 30 Sep 2026) |
| IONOS | AI Model Hub, per-token API; Cloud GPU VMs (H200), self-managed (read 30 Sep 2026) | Fixed catalogue (named models, e.g. Llama 3.3 70B Instruct); none stated on GPU VMs (self-managed) | H200 GPU VM (H200 – S) 3 €/hr pay-as-you-go (read 30 Sep 2026) |
| Nebius | GPU cloud instances, on-demand; Serverless AI endpoints running the team's own container image, per-second billing (read 30 Sep 2026) | None stated (self-managed VM or own container image) | H100 $3.85/hr |
| OVHcloud | Public Cloud GPU instances (H100, L40S, L4, A10 and others); AI Deploy: the team's Docker image on GPU nodes, fixed or automatically scaled, billed per minute (read 30 Sep 2026) | None stated (self-managed VM or own Docker image) | H100 $2.99/hr excl. VAT (read 30 Sep 2026); AI Deploy h100-1-gpu $3.39/hr |
| Scaleway | GPU instances; Managed Inference: a model library or the team's own model (e.g. Hugging Face) on dedicated GPUs (read 30 Sep 2026) | None stated (self-managed VM); Managed Inference accepts the team's own model | L4 €0.79/hr |
| STACKIT | Compute Engine GPU VMs, hourly billing | None stated (self-managed VM) | GPU Server-n1.14d.g1-EU01 at €2.658/hr |
Several providers also run a managed or container route beside their VMs, read on 30 September 2026: OVHcloud AI Deploy[13] and Nebius Serverless AI[12] run a team's own container image on GPUs, while Exoscale Dedicated Inference[10] and Scaleway Managed Inference[14] serve a team's own Hugging Face model on dedicated GPUs. IONOS lists H200 GPU VMs at 3 € per hour[11], and Hetzner's GEX45 at €0.3429 per hour before VAT[4]. The split that matters for a custom model is the third column: catalogue platforms such as IONOS's AI Model Hub serve the models they list, while self-managed VMs and the container and own-model routes above run whatever weights the team supplies, including open-weight models such as DeepSeek on European infrastructure. A broader scored comparison of European AI infrastructure providers, covering sovereignty and compliance criteria, is published separately: see European providers, compared.
Scaling, upgrades and the health check
Scaling behaviour follows the route. A managed endpoint scales replicas on the provider's terms, and the team's lever is the request rate the endpoint is configured for. A serverless container platform scales instances with traffic and back to zero, which trades idle cost for cold-start latency. On a self-managed VM, scaling is the team's work: more replicas mean more VMs to provision, monitor and patch, and steady traffic is what makes that fixed capacity pay off. Upgrades follow the same split. On a VM, a CUDA driver or serving-engine update is a change the team must test against its own weights and roll out itself; on a managed endpoint the provider rolls the upgrade, and the team's job is to catch regressions in its evaluation suite. Whatever the route, the deployment needs a health check: an HTTP endpoint the platform or the load balancer polls to confirm the model is loaded and answering, so a failed instance is replaced instead of silently serving errors.
Choosing a route for a first deployment
For a first production deployment, the decision reduces to two questions: do the weights fit the platform, and who carries the operations. If the model is in a provider's catalogue and the per-token price is acceptable, a managed endpoint removes runtime and upgrade work entirely. If the weights are custom, the realistic routes are a container on a serverless GPU platform or a self-managed VM, and the choice turns on traffic shape: idle-heavy or spiky traffic favours scale-to-zero billing, while steady traffic on a VM avoids cold starts and simplifies capacity planning. Size VRAM at the serving precision plus peak KV cache before comparing prices, because an hourly rate only compares across GPUs that hold the model at the serving precision. Whichever route is chosen, record the model ID, the precision, the serving engine version and the retrieval dates of every figure the decision rested on, so the deployment can be re-evaluated when prices or hardware change.
Provider figures in this article were read from each provider's own public page on 22 September 2026; the Exoscale, Hetzner, IONOS and OVHcloud rates and the managed and container routes of Exoscale, Nebius, OVHcloud and Scaleway were read on 30 September 2026.
Frequently asked questions
How do I deploy a Hugging Face model?
Engineering teams can use a managed service that handles the runtime, run a containerized inference engine on a serverless GPU platform, or provision a self-managed GPU virtual machine to build their own HTTP API.
What VRAM does a 70B model need?
A 70-billion parameter model requires roughly 140 gigabytes of VRAM to load its weights in FP16 precision. Using INT8 or INT4 quantization reduces this footprint to about 70 GB or 35 GB, though additional memory must be allocated for the KV cache to handle concurrent requests.
Can I deploy any model or only a catalogue?
A self-managed GPU virtual machine or a serverless container platform allows you to deploy any custom model. Fully managed inference APIs often restrict deployments to a fixed catalogue of pre-validated models.
Managed endpoint or my own VM?
A managed endpoint suits teams wanting to avoid infrastructure management and scaling logic. A self-managed GPU virtual machine, such as a Lyceum on-demand H100 VM at $2.79/hr, is needed for a custom inference engine or full root access; proprietary weights can also run on managed services that accept a team's own model, such as Exoscale Dedicated Inference or Scaleway Managed Inference, or as a container on OVHcloud AI Deploy or Nebius Serverless AI.
Sources
Rankings and scores on Optinest cannot be bought — no commercial relationship changes how a provider is assessed. How we score
Related reading
All Inference →Scale-to-zero platforms for autonomous agents
Compare scale-to-zero platforms for autonomous agents across European cloud providers, and calculate the cost of cold starts versus idle capacity.

Choosing a model backend for AI agents in the EU
Compare European inference providers for AI agents on tool calling, rate limits and pricing. See which EU endpoints support parallel and forced tool calls.
Who pays for AI tokens: Cost structures and provider pricing
Understand who pays for AI tokens and how rate cards work. Compare European serverless inference providers to model your exact generative AI costs.
