Skip to content

Host your own models

By the end of this tutorial you will have an open-weight model served from NVIDIA GPUs in the Netherlands, reachable as an inference endpoint, and registered behind your gateway.

Prefer clicking through the console instead? See the console walkthrough.

Scaffold status

The step structure below is agreed and stable. Exact commands, manifests and captured output are added as this flow is verified end to end on the live platform — this page deliberately shows no invented output.

Prerequisites

Overview

flowchart LR
    S3[("S3 object storage<br/>model weights")] --> ISVC
    ISVC["InferenceService<br/>KServe + vLLM"] --> GPU["NVIDIA GPU node<br/>MIG-partitioned"]
    GW["LLM gateway"] --> ISVC

Steps

1. Size the deployment

GPU memory is usually the binding constraint: model weights, plus KV cache, plus activation overhead. Establish this before you request hardware — an undersized request fails at load time, an oversized one strands capacity other workloads could use.

2. Stage the weights

Put weights in an S3 bucket in your project, and create the data connection the serving runtime reads them through.

3. Choose a serving runtime

vLLM and SGLang are both available. The choice affects throughput characteristics and which sampling and batching features you get.

4. Create the InferenceService

A KServe InferenceService naming the runtime, the storage URI for the weights, and the GPU resource request.

5. Wait for it to become ready

First start includes pulling weights and loading them onto the GPU, so it is slower than a normal pod start.

6. Register it with the gateway

Add the endpoint as a provider — see Add a model provider — so clients reach it by model name.

Verify

Call the endpoint directly first, to separate model problems from gateway problems:

oc get inferenceservice -n <project>
oc logs -l serving.kserve.io/inferenceservice=<name> -n <project> --tail=50

Then call the same model through the gateway and confirm both paths return a response.

Check readiness before blaming throughput

A model that has not finished loading will look slow rather than broken. Confirm the InferenceService reports ready before you measure anything.

Clean up

Remove everything this tutorial created, in reverse order:

oc delete -k . -n <project>          # or delete the project outright
oc delete project <project>

Next

Deploy an MCP server →