InferenceServing topologies

Deploying a Hugging Face model as an API endpoint

Evaluate the routes to deploy a Hugging Face model as an API endpoint. Compare managed endpoints, serverless platforms, and VMs across European providers.

, AI Researcher8 min readUpdated 1 October 2026
In this article7
  1. The three routes and who owns what
  2. Sizing VRAM at the precision you will serve
  3. Which platforms take any model and which take a catalogue
  4. Scaling, upgrades and the health check
  5. Choosing a route for a first deployment
  6. Frequently asked questions
  7. Sources
A diagram showing three deployment paths for an AI model: a managed endpoint, a serverless GPU container, and a self-managed virtual machine.
A diagram showing three deployment paths for an AI model: a managed endpoint, a serverless GPU container, and a self-managed virtual machine.

The three routes and who owns what

A Hugging Face model becomes a production API when something runs its weights behind an HTTP endpoint. Three routes do this, and they differ in one thing: who owns the runtime, the scaling and the upgrade path.

  • Managed endpoint: the provider runs the serving stack, autoscaling and upgrades. The team ships weights or selects from a catalogue and writes no serving code. Operational burden is lowest, but deployment is limited to what the provider accepts.
  • Container on a serverless GPU platform: the team owns the serving code and its container image; the platform owns scale-to-zero and per-second billing. Cold starts and image size become the team's problem.
  • Self-managed GPU VM: the team owns everything, including OS patching, CUDA driver versions, the serving engine, TLS termination and health checks. Full control, full burden.

A root-access VM leaves the entire stack to the team, while a serverless GPU route that runs a Python script or a Docker image keeps the serving code and its container with the team and moves only the machine underneath.

Which serving engine runs inside the chosen route is a separate decision from the route itself; for that decision, see vLLM, SGLang or Ray, where execution engines and orchestration layers are compared.

Sizing VRAM at the precision you will serve

The model card states a parameter count, not the memory a deployment needs. Weight memory is arithmetic: parameters multiplied by the bytes each parameter occupies at the serving precision. A 70-billion-parameter model at FP16, two bytes per parameter, needs about 140 GB for weights alone; the same weights at INT8 need about 70 GB, and at INT4 about 35 GB. Quantization trades memory for accuracy, so the precision the team will actually serve, not the checkpoint's native format, sets the requirement.

PrecisionBytes per parameterWeights, 70B model (calculation)Example hardware that fits
FP16 / BF162~140 GBTwo 80 GB GPUs or one 141 GB GPU; almost no headroom
INT81~70 GBOne 80 GB GPU
INT40.5~35 GBOne 48 GB GPU

Weights are the floor, not the total. The KV cache, the memory that stores intermediate attention state for each in-flight sequence, grows with context length and with the number of concurrent requests. A deployment sized to weights alone will queue or fail requests once concurrency rises. The working rule: weights, plus KV cache at the target maximum concurrency and context length, plus activation memory, must fit in VRAM with headroom. Worked sizing tables for 7B to 70B models, including quantization formats, are published separately: see 70B VRAM footprints.

Which platforms take any model and which take a catalogue

A team with custom or fine-tuned weights needs a platform that accepts any Hugging Face model; a team serving a well-known open model can accept a fixed catalogue in exchange for less operations work. A reader who wants to serve the weights themselves can take a Lyceum on-demand GPU VM with full root access, H100 at $2.79/hr, billed per second with no minimum (read 22 September 2026)[1], and run their own serving stack on it. The table below reads each provider's own public page on that date; entries marked 30 September 2026 were re-read that day.

ProviderRoutes on the provider's pagesModel restriction on the page readExample GPU price
LyceumOn-demand GPU VMs with full root access; serverless GPU executionNone stated (self-managed VM)H100 $2.79/hr
ExoscaleGPU instances (A30 and further GPU instance families); Dedicated Inference: any Hugging Face model (public, gated or private) served as an API on dedicated GPUs (read 30 Sep 2026)None stated (self-managed VM); Dedicated Inference accepts any Hugging Face modelA30 €1.2265/hr (GPU instance); RTX Pro 6000 €2.15/hr on Dedicated Inference (read 30 Sep 2026)
HetznerDedicated GPU servers, GEX line (RTX PRO 4000, 24 GB; RTX PRO 6000, 96 GB)None stated (self-managed server)GEX45 (RTX PRO 4000) €0.3429/hr, max €214.00/month, excl. VAT (read 30 Sep 2026)
IONOSAI Model Hub, per-token API; Cloud GPU VMs (H200), self-managed (read 30 Sep 2026)Fixed catalogue (named models, e.g. Llama 3.3 70B Instruct); none stated on GPU VMs (self-managed)H200 GPU VM (H200 – S) 3 €/hr pay-as-you-go (read 30 Sep 2026)
NebiusGPU cloud instances, on-demand; Serverless AI endpoints running the team's own container image, per-second billing (read 30 Sep 2026)None stated (self-managed VM or own container image)H100 $3.85/hr
OVHcloudPublic Cloud GPU instances (H100, L40S, L4, A10 and others); AI Deploy: the team's Docker image on GPU nodes, fixed or automatically scaled, billed per minute (read 30 Sep 2026)None stated (self-managed VM or own Docker image)H100 $2.99/hr excl. VAT (read 30 Sep 2026); AI Deploy h100-1-gpu $3.39/hr
ScalewayGPU instances; Managed Inference: a model library or the team's own model (e.g. Hugging Face) on dedicated GPUs (read 30 Sep 2026)None stated (self-managed VM); Managed Inference accepts the team's own modelL4 €0.79/hr
STACKITCompute Engine GPU VMs, hourly billingNone stated (self-managed VM)GPU Server-n1.14d.g1-EU01 at €2.658/hr

Several providers also run a managed or container route beside their VMs, read on 30 September 2026: OVHcloud AI Deploy[13] and Nebius Serverless AI[12] run a team's own container image on GPUs, while Exoscale Dedicated Inference[10] and Scaleway Managed Inference[14] serve a team's own Hugging Face model on dedicated GPUs. IONOS lists H200 GPU VMs at 3 € per hour[11], and Hetzner's GEX45 at €0.3429 per hour before VAT[4]. The split that matters for a custom model is the third column: catalogue platforms such as IONOS's AI Model Hub serve the models they list, while self-managed VMs and the container and own-model routes above run whatever weights the team supplies, including open-weight models such as DeepSeek on European infrastructure. A broader scored comparison of European AI infrastructure providers, covering sovereignty and compliance criteria, is published separately: see European providers, compared.

Scaling, upgrades and the health check

Scaling behaviour follows the route. A managed endpoint scales replicas on the provider's terms, and the team's lever is the request rate the endpoint is configured for. A serverless container platform scales instances with traffic and back to zero, which trades idle cost for cold-start latency. On a self-managed VM, scaling is the team's work: more replicas mean more VMs to provision, monitor and patch, and steady traffic is what makes that fixed capacity pay off. Upgrades follow the same split. On a VM, a CUDA driver or serving-engine update is a change the team must test against its own weights and roll out itself; on a managed endpoint the provider rolls the upgrade, and the team's job is to catch regressions in its evaluation suite. Whatever the route, the deployment needs a health check: an HTTP endpoint the platform or the load balancer polls to confirm the model is loaded and answering, so a failed instance is replaced instead of silently serving errors.

Choosing a route for a first deployment

For a first production deployment, the decision reduces to two questions: do the weights fit the platform, and who carries the operations. If the model is in a provider's catalogue and the per-token price is acceptable, a managed endpoint removes runtime and upgrade work entirely. If the weights are custom, the realistic routes are a container on a serverless GPU platform or a self-managed VM, and the choice turns on traffic shape: idle-heavy or spiky traffic favours scale-to-zero billing, while steady traffic on a VM avoids cold starts and simplifies capacity planning. Size VRAM at the serving precision plus peak KV cache before comparing prices, because an hourly rate only compares across GPUs that hold the model at the serving precision. Whichever route is chosen, record the model ID, the precision, the serving engine version and the retrieval dates of every figure the decision rested on, so the deployment can be re-evaluated when prices or hardware change.

Provider figures in this article were read from each provider's own public page on 22 September 2026; the Exoscale, Hetzner, IONOS and OVHcloud rates and the managed and container routes of Exoscale, Nebius, OVHcloud and Scaleway were read on 30 September 2026.

Frequently asked questions

How do I deploy a Hugging Face model?

Engineering teams can use a managed service that handles the runtime, run a containerized inference engine on a serverless GPU platform, or provision a self-managed GPU virtual machine to build their own HTTP API.

What VRAM does a 70B model need?

A 70-billion parameter model requires roughly 140 gigabytes of VRAM to load its weights in FP16 precision. Using INT8 or INT4 quantization reduces this footprint to about 70 GB or 35 GB, though additional memory must be allocated for the KV cache to handle concurrent requests.

Can I deploy any model or only a catalogue?

A self-managed GPU virtual machine or a serverless container platform allows you to deploy any custom model. Fully managed inference APIs often restrict deployments to a fixed catalogue of pre-validated models.

Managed endpoint or my own VM?

A managed endpoint suits teams wanting to avoid infrastructure management and scaling logic. A self-managed GPU virtual machine, such as a Lyceum on-demand H100 VM at $2.79/hr, is needed for a custom inference engine or full root access; proprietary weights can also run on managed services that accept a team's own model, such as Exoscale Dedicated Inference or Scaleway Managed Inference, or as a container on OVHcloud AI Deploy or Nebius Serverless AI.

Sources

  1. lyceum.technology
  2. lyceum.technology
  3. exoscale.com
  4. hetzner.com
  5. cloud.ionos.com
  6. nebius.com
  7. ovhcloud.com
  8. scaleway.com
  9. stackit.com
  10. exoscale.com
  11. cloud.ionos.de
  12. docs.nebius.com
  13. ovhcloud.com
  14. scaleway.com

Rankings and scores on Optinest cannot be bought — no commercial relationship changes how a provider is assessed. How we score

Related reading

All Inference →