Models (LiteLLM)
https://litellm.k8s.semanticscience.org — OpenAI-compatible. You talk to models only through this gateway; never to a model server directly.
POST /v1/chat/completions
POST /v1/embeddings
GET /v1/models
Authorization: Bearer sk-...What is served
model | what | where |
|---|---|---|
gemma4-g1 | gemma-4-26B-A4B-it, 4-bit AWQ, 32k context, chat | vLLM on one A40 |
nomic-embed-text | nomic-embed-text-v1.5, 768-dim embeddings, 8k context | llama.cpp on a Mac mini (Metal) |
claude-sonnet claude-opus claude-haiku claude-fable | Anthropic, metered spend | Anthropic API |
The model field is the gateway name from the left column — not a Hugging Face id, not a backend path.
Calling it
curl -s https://litellm.k8s.semanticscience.org/v1/chat/completions \
-H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
-d '{"model":"gemma4-g1","messages":[{"role":"user","content":"Summarise FAIR in three sentences."}],"max_tokens":300}'from openai import OpenAI
client = OpenAI(base_url="https://litellm.k8s.semanticscience.org/v1", api_key="sk-...")
r = client.chat.completions.create(model="gemma4-g1", messages=[{"role":"user","content":"hi"}])
e = client.embeddings.create(model="nomic-embed-text", input="search_query: knowledge graphs")Anything that takes an OpenAI-compatible base_url — LangChain, LlamaIndex, DSPy, your own loop — works the same way.
nomic needs a prefix. search_query: for queries, search_document: for what you index. Without them retrieval quality drops.
Getting a key
Ask an admin, or if you have been invited, sign in at https://litellm.k8s.semanticscience.org/ui with email + password and create one under Virtual Keys. One key per project, so spend is attributable and a leaked key can be revoked without breaking the rest.
Keys can be scoped to specific models. If a request returns 403 key_model_access_denied, the key was not issued for that model — that is intended, not broken.
Requesting a model
Two steps, and for now you ask rather than do it yourself:
- It is served — deployed via LLMKube on g1 (vLLM) or the mini (llama.cpp). Each served model is one file in git.
- It gets a gateway name and appears in
/v1/models.
Send: Hugging Face repo id · quantization · context length you need · chat or embedding · g1 or mini.
There is no scale-to-zero. A served model holds its hardware until removed. With three A40s and one taken by gemma4-g1, a new large model is a swap, not an add.