AI inference¶
Serve open-weight models from infrastructure in the Netherlands, on the same clusters as
the rest of your workloads. A served model is an InferenceService — scheduled by the same
control plane, secured by the same RBAC, and monitored by the same stack as any other
Kubernetes object.
How it fits together¶
flowchart LR
CLIENT["Your application"] --> GW["LLM gateway<br/>keys · budgets · routing"]
GW --> ISVC["InferenceService<br/>KServe + vLLM"]
GW --> EXT["External providers"]
ISVC --> GPU["NVIDIA GPU nodes"]
ISVC --> S3["S3 object storage<br/>model weights"]Most teams put the LLM gateway in front of inference rather than calling endpoints directly — it centralises keys, budgets and failover across self-hosted and external models alike.
What this section will cover¶
- Serving runtimes and when to choose which
- GPU requests, partitioning and sizing
- Model storage and data connections
- Rollout, canary and rollback
- Model registry and versioning
Scaffold status
This is a placeholder. Supported runtimes, model list and per-GPU sizing tables are written here once confirmed against the platform.