Add a model provider¶
By the end of this tutorial the gateway will have a new model backend registered, with routing and failover configured, and clients will be able to reach it by model name without knowing where it runs.
Prefer clicking through the console instead? See the console walkthrough.
Scaffold status
The step structure below is agreed and stable. Exact commands, manifests and captured output are added as this flow is verified end to end on the live platform — this page deliberately shows no invented output.
Prerequisites¶
- A running gateway — see Deploy an LLM gateway
- Credentials for the provider you are adding, or a self-hosted endpoint URL
What a provider is¶
To the gateway, a provider is a backend that speaks a known API and a set of models it serves. That can be an external commercial API, or an inference endpoint you host yourself on GRN GPUs. Clients see model names; they never see which is which.
Steps¶
1. Store the credential¶
Add the provider key to the gateway's Secret. Never place it in a ConfigMap or a manifest committed to Git.
2. Register the provider¶
Declare the provider type, its base URL and the credential reference.
3. Map model names¶
Give each backend model a stable public name. Clients depend on this name, so choose it once and keep it — changing it later breaks every caller.
4. Set routing and failover¶
Decide the order backends are tried and when the gateway falls back to a secondary. This is where a self-hosted primary with a commercial fallback is configured.
5. Reload¶
Apply the configuration and confirm the gateway picks it up.
Verify¶
Call the new model name through the gateway and confirm the request is attributed to the new provider in the logs:
curl -sS https://<your-gateway-host>/v1/chat/completions \
-H "Authorization: Bearer <your-virtual-key>" \
-H "Content-Type: application/json" \
-d "{\"model\":\"<model-name>\",\"messages\":[{\"role\":\"user\",\"content\":\"ping\"}]}"
Then test failover by making the primary unavailable and confirming traffic moves to the secondary without the client changing anything.
Clean up¶
Remove everything this tutorial created, in reverse order: