Start here

The API in five minutes

llamay answers OpenAI's, Anthropic's and Ollama's APIs on one address. The client library you already use works once you change its base URL. Every example on this page was run against llamay 0.3.0 with qwen2.5:0.5b on a laptop CPU, and the output shown is what came back.

1 · Start a server

llamay pull qwen2.5:0.5b
llamay pull all-minilm          # an encoder, for embeddings
llamay serve -m qwen2.5:0.5b -embed all-minilm

It listens on localhost:11435. On loopback it needs no key. Check it answers, and which model it holds:

curl -s localhost:11435/healthz

{"backend":"cpu","id":"590bc10b059c7189","instance":"caacd23f45f3d8ee","model":"qwen2.5:0.5b","status":"ok","version":"0.3.0"}

model is the name to send in requests. id identifies the weights, and instance this process, so a client can tell a restart from the same server.

2 · curl

curl -s localhost:11435/v1/chat/completions -d '{
  "model": "qwen2.5:0.5b", "temperature": 0, "max_tokens": 40,
  "messages": [{"role": "user", "content": "Name the capital of France in one word."}]
}'

{"choices":[{"finish_reason":"stop","index":0,"message":{"content":"Paris","role":"assistant"}}],
 "created":1790482610,"id":"chatcmpl-llamay","model":"qwen2.5:0.5b","object":"chat.completion",
 "usage":{"completion_tokens":1,"llamay_cached_prompt_tokens":0,"llamay_decode_ms":22.767,
  "llamay_prefill_ms":289.196,"prompt_tokens":17,"prompt_tokens_details":{"cached_tokens":0},"total_tokens":18}}

The llamay_ fields in usage are extras. A client that does not know them ignores them.

3 · The OpenAI SDK

from openai import OpenAI

client = OpenAI(base_url="http://localhost:11435/v1", api_key="unused-on-loopback")

reply = client.chat.completions.create(
    model="qwen2.5:0.5b",
    temperature=0,
    messages=[{"role": "user", "content": "Say hello in French, in two words."}],
)
print(reply.choices[0].message.content)

stream = client.chat.completions.create(
    model="qwen2.5:0.5b", temperature=0, stream=True,
    messages=[{"role": "user", "content": "Count from one to five in words."}],
)
for chunk in stream:
    if chunk.choices:
        print(chunk.choices[0].delta.content or "", end="", flush=True)
print()

vec = client.embeddings.create(model="all-minilm", input=["a cat sat on the mat"])
print(len(vec.data[0].embedding), "dimensions")

Bonjour!
Sure, here are the words for counting from one to five:

- One
- Two
- Three
- Four
- Five
384 dimensions

The SDK insists on an API key. Any string works until the server has keys; then send a real one. In JavaScript the same change is new OpenAI({ baseURL: "http://localhost:11435/v1", apiKey: "…" }).

4 · The Anthropic SDK

from anthropic import Anthropic

client = Anthropic(base_url="http://localhost:11435", api_key="unused-on-loopback")

msg = client.messages.create(
    model="qwen2.5:0.5b",
    max_tokens=64,
    system="Answer in one short sentence.",
    messages=[{"role": "user", "content": "What is the boiling point of water at sea level in Celsius?"}],
)
print(msg.content[0].text)
print(msg.stop_reason, msg.usage.input_tokens, msg.usage.output_tokens)

The boiling point of water at sea level is 100°C.
end_turn 32 15

The base URL has no /v1: the SDK adds it. POST /v1/messages/count_tokens is served too, counted by the model's own tokenizer.

5 · The Ollama client

from ollama import Client

client = Client(host="http://localhost:11435")

r = client.chat(model="qwen2.5:0.5b", options={"temperature": 0},
                messages=[{"role": "user", "content": "Name a primary colour. One word."}])
print(r.message.content)
print([m.model for m in client.list().models][:3])

Red
['all-minilm:latest', 'gemma2:2b', 'gemma3-4b-mmproj:latest']

The Ollama routes answer in Ollama's own conventions: newline-delimited JSON, streaming on by default, durations in nanoseconds. Models Ollama already downloaded are listed and served without a second download.

Which model answers

• Name one in model. The server loads it on first use, keeping up to -max-loaded models resident.
• Send "" and the server's default answers: the -m it was started with.
• A name it does not have is a 404 that lists what it does have.

auto is a model name on llamay's hosted service, not on a local server. Locally it is a 404 like any other unknown name.

What else the server does

When it says no

Errors come back in the shape of the API you called, with a sentence meant to be shown to a person:

{"error":{"message":"no model named \"auto\". Available:\n  all-minilm:latest\n  gemma2:2b\n  qwen2.5:0.5b ...","type":"Not Found"}}

400 means change the request. 429 means the queue is full: wait for Retry-After and send it again. The whole list is on the errors page.

Where to go next