Start here
Quickstart
Install, pull a model, talk to it, serve it. Four commands. If you would rather not use a terminal at all, the apps do the same four steps with a window.
The four steps
1 · Install
macOS and Linux:
curl -fsSL https://llamay.com/install.sh | shWindows, in PowerShell:
irm https://llamay.com/install.ps1 | iex
The script works out your platform, downloads the matching build,
checks it against the release's SHA256SUMS,
installs the binary and sets up a service. It refuses to continue on a
machine with no SHA-256 tool, rather than skipping the check.
| Platform | Binary | Service |
|---|---|---|
| macOS | /usr/local/bin/llamay | LaunchAgent, runs as you |
| Linux | /usr/local/bin/llamay | systemd, its own llamay user |
| Windows | %LOCALAPPDATA%\Programs\llamay | A scheduled task at logon |
LLAMAY_NO_SERVICE=1 before the script installs only the command
line. Packages, the apps and the portable archives are on
the download page; every route is on
the install page.
2 · Get a model
llamay pull qwen2.5:0.5b # 400 MB, runs anywhere
llamay models # what this machine hasname size architecture store all-minilm:latest 43.8 MiB bert (embedder) llamay gemma2:2b 1.6 GiB gemma2 (decoder) llamay qwen2.5:0.5b 379.4 MiB qwen2 (decoder) llamay
Downloads resume, and a file is installed only under the SHA-256 of its finished bytes. Models Ollama already downloaded are listed too, and run without a second download. Hugging Face and plain URLs work as well; see getting models.
3 · Talk to it
llamay run -m qwen2.5:0.5b -chat -p "Name the capital of France in one word."
-chat sends the prompt through the model's own chat template, as
a conversation. Without it the prompt is continued as raw text, which is
what a base model expects.
The whole of the above, recorded on a laptop:
Transcript — the whole session as text
~/llamay $ # one static binary — this is the whole install ~/llamay $ llamay version llamay v0.1.58-1-gb14f550 build go1.26.8 darwin/arm64, cgo on, tags metal revision b14f5501578e kernels neon+i8mm gpu metal built in; metal:Apple M4 formats 23: F32, F16, Q4_0, Q4_1, Q5_0, Q5_1, Q8_0, Q8_1, Q2_K, Q3_K, Q4_K, Q5_K, Q6_ K, Q8_K, IQ2_XXS, IQ2_XS, IQ3_XXS, IQ4_NL, IQ3_S, IQ2_S, IQ4_XS, BF16, MXFP4 architectures 17 entries, 39 names: azmx (azmx-one, llama, mistral), qwen2 (qwen3), gemma, gemma2, gemma3 (gemma3_text), azmx-code (azmx_code, azmxcode), gpt-oss (gptoss, gpt_oss, ope nai-moe), phi3 (phi3.5), stablelm (stablelm2), gpt2, falcon (rw), gptneox (gpt_neox, gpt-neo x, neox), phi2, qwen3moe (qwen3_moe), qwen2moe (qwen2_moe), azmx-ocr (ocr, ctc), azmx-embed (bert, embedding) ~/llamay $ # models are named, not pathed, once they are in the store ~/llamay $ llamay models name size architecture store all-minilm:latest 43.8 MiB bert (embedder) llamay gemma2:2b 1.6 GiB gemma2 (decoder) llamay gemma3:270m 278.0 MiB gemma3 (decoder) llamay gpt2:q8_0 169.4 MiB gpt2 (decoder) llamay qwen2.5:0.5b 379.4 MiB qwen2 (decoder) llamay smollm2:135m 258.3 MiB llama (decoder) llamay azmx-code-test:latest 1.1 GiB gemma3 (decoder) ollama 6 in ~/.llamay/models, 1 in ~/.ollama/models ~/llamay $ # and that is the entire setup ~/llamay $ llamay run -m gemma2:2b -chat -n 110 -p "Explain what a KV cache is to someone wh o writes web servers. Two sentences." A KV cache is a memory-based store used by web servers to quickly retrieve frequently access ed data. It works by storing key-value pairs (like website configurations, user sessions, or static content) and using those pairs to bypass the slower database or disk access for fast er responses. prompt 25 tokens in 2.507s (10.0 tok/s) · generated 58 in 15.61s (3.7 tok/s) · stop: eog
4 · Serve it
llamay serve
With no -m, serve takes the first decoder in your
store, sorted by name so a restart serves the same one. It listens on
localhost:11435: this machine only. With no models
at all it still starts, answers /healthz and
POST /api/pull, and loads the first model that lands.
curl -s localhost:11435/v1/chat/completions -d '{
"model": "qwen2.5:0.5b",
"messages": [{"role": "user", "content": "Name the capital of France in one word."}]
}'{"choices":[{"finish_reason":"stop","index":0,"message":{"content":"Paris","role":"assistant"}}], ...}
The same server answers POST /v1/messages in Anthropic's shape
and POST /api/chat in Ollama's, so an existing client needs a
new base URL and nothing else. The API in five
minutes does this with each SDK.
It sits beside Ollama's 11434, so both can run at once. And
/healthz answers with the model, the version and an instance
id, so "is it running" is a question about identity rather than about
whether something holds a port.
The same model behind an HTTP endpoint, answering the request every client already sends:
Transcript — the whole session as text
~/llamay $ llamay serve -m qwen2.5:0.5b -addr localhost:11439
time=2026-09-18T08:49:09.326-04:00 level=INFO msg="device graph attached" max-tokens=256
time=2026-09-18T08:49:09.326-04:00 level=INFO msg="llamay serving" addr=localhost:11439 mode
l=qwen2.5:0.5b arch=qwen2 max-loaded=2 keepalive=5m0s params="0.36 GB" context=32768 threads
=7 kv=f32 lexicon=0
OpenAI POST http://localhost:11439/v1/chat/completions
Anthropic POST http://localhost:11439/v1/messages
Embeddings POST http://localhost:11439/v1/embeddings (start with -embed <encoder.gguf
>)
Rerank POST http://localhost:11439/rerank (start with -rerank <reranker.gg
uf>)
Contexts POST http://localhost:11439/v1/contexts (then /{id}/fork?n=8, /{id}/sna
pshot)
Models POST http://localhost:11439/api/pull (and /api/copy, /api/delete)
Stats GET http://localhost:11439/v1/stats
~/llamay $ # the server stays up; the rest of this is a second terminal
~/llamay $ curl -s http://localhost:11439/v1/models | jq
{
"data": [
{
"created": 1789735749,
"id": "all-minilm:latest",
"object": "model",
"owned_by": "azmx"
},
{
"created": 1789735749,
"id": "gemma2:2b",
"object": "model",
"owned_by": "azmx"
},
{
"created": 1789735749,
"id": "gemma3:270m",
"object": "model",
"owned_by": "azmx"
},
{
"created": 1789735749,
"id": "gpt2:q8_0",
"object": "model",
"owned_by": "azmx"
},
{
"created": 1789735749,
"id": "qwen2.5:0.5b",
"object": "model",
"owned_by": "azmx"
},
{
"created": 1789735749,
"id": "smollm2:135m",
"object": "model",
"owned_by": "azmx"
},
{
"created": 1789735749,
"id": "azmx-code-test:latest",
"object": "model",
"owned_by": "azmx"
}
],
"object": "list"
}
~/llamay $ curl -s http://localhost:11439/v1/chat/completions -H 'content-type: application/
json' \
> -d '{"model":"qwen2.5:0.5b","messages":[{"role":"user","content":"One sentence: why run
a model locally?"}]}' | jq
{
"choices": [
{
"finish_reason": "stop",
"index": 0,
"message": {
"content": "running a model locally can be beneficial because it allows you to optim
ize the model locally, which can be faster and more efficient compared to running the model
globally.",
"role": "assistant"
}
}
],
"created": 1789735751,
"id": "chatcmpl-llamay",
"model": "qwen2.5:0.5b",
"object": "chat.completion",
"usage": {
"completion_tokens": 31,
"llamay_cached_prompt_tokens": 0,
"llamay_decode_ms": 1528.48,
"llamay_prefill_ms": 153.028,
"prompt_tokens": 17,
"prompt_tokens_details": {
"cached_tokens": 0
},
"total_tokens": 48
}
}
~/llamay $ # the same route with stream:true — this is the wire format
~/llamay $ curl -sN http://localhost:11439/v1/chat/completions -H 'content-type: application
/json' \
> -d '{"model":"qwen2.5:0.5b","stream":true,"messages":[{"role":"user","content":"Count to
five."}]}' | head -c 520
data: {"choices":[{"delta":{"content":"Count"},"finish_reason":null,"index":0}],"created":17
89735751,"id":"chatcmpl-llamay","model":"qwen2.5:0.5b","object":"chat.completion.chunk"}
data: {"choices":[{"delta":{"content":"ing"},"finish_reason":null,"index":0}],"created":1789
735751,"id":"chatcmpl-llamay","model":"qwen2.5:0.5b","object":"chat.completion.chunk"}
data: {"choices":[{"delta":{"content":" to"},"finish_reason":null,"index":0}],"created":1789
735751,"id":"chatcmpl-llamay","model":"qwen2.5:0.5b","object":"cha~/llamay $ # and the same
stream read the way a client reads it
~/llamay $ curl -sN http://localhost:11439/v1/chat/completions -H 'content-type: application
/json' \
> -d '{"model":"qwen2.5:0.5b","stream":true,"messages":[{"role":"user","content":"Count to
five, then stop."}]}' \
> | python3 scripts/site/demo_sse.py
1
2
3
4
5
Where to go next
Build against it
curl, the OpenAI SDK, the Anthropic SDK and the Ollama client, each with real output.
API in five minutes →Fork a context
Pay for a long system prompt once, then branch it, save it, or move it to another server.
Contexts →Constrain the output
JSON Schema and tool calls enforced while sampling, and what is refused.
Structured output →