# Models (LiteLLM)

https://litellm.k8s.semanticscience.org — OpenAI-compatible. You talk to models **only** through this gateway; never to a model server directly.

```
POST /v1/chat/completions
POST /v1/embeddings
GET  /v1/models
Authorization: Bearer sk-...
```

## What is served

| `model` | what | where |
|---|---|---|
| `gemma4-g1` | gemma-4-26B-A4B-it, 4-bit AWQ, **32k context**, chat | vLLM on one A40 |
| `nomic-embed-text` | nomic-embed-text-v1.5, 768-dim embeddings, 8k context | llama.cpp on a Mac mini (Metal) |
| `claude-sonnet` `claude-opus` `claude-haiku` `claude-fable` | Anthropic, metered spend | Anthropic API |

The `model` field is the **gateway name** from the left column — not a Hugging Face id, not a backend path.

## Calling it

```bash
curl -s https://litellm.k8s.semanticscience.org/v1/chat/completions \
  -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d '{"model":"gemma4-g1","messages":[{"role":"user","content":"Summarise FAIR in three sentences."}],"max_tokens":300}'
```

```python
from openai import OpenAI
client = OpenAI(base_url="https://litellm.k8s.semanticscience.org/v1", api_key="sk-...")
r = client.chat.completions.create(model="gemma4-g1", messages=[{"role":"user","content":"hi"}])
e = client.embeddings.create(model="nomic-embed-text", input="search_query: knowledge graphs")
```

Anything that takes an OpenAI-compatible `base_url` — LangChain, LlamaIndex, DSPy, your own loop — works the same way.

**nomic needs a prefix.** `search_query: ` for queries, `search_document: ` for what you index. Without them retrieval quality drops.

## Getting a key

Ask an admin, or if you have been invited, sign in at https://litellm.k8s.semanticscience.org/ui with email + password and create one under **Virtual Keys**. **One key per project**, so spend is attributable and a leaked key can be revoked without breaking the rest.

Keys can be scoped to specific models. If a request returns `403 key_model_access_denied`, the key was not issued for that model — that is intended, not broken.

## Requesting a model

Two steps, and for now you ask rather than do it yourself:

1. **It is served** — deployed via LLMKube on g1 (vLLM) or the mini (llama.cpp). Each served model is one file in git.
2. **It gets a gateway name** and appears in `/v1/models`.

Send: Hugging Face repo id · quantization · context length you need · chat or embedding · g1 or mini.

**There is no scale-to-zero.** A served model holds its hardware until removed. With three A40s and one taken by `gemma4-g1`, a new large model is a **swap**, not an add.