How it works

Architecture

A few dozen packages, six of which carry the weight. This page follows one GGUF file from the disk to a token on a socket, and then follows one HTTP request through the same machinery from the other end.

The package map

Package	What it owns
`cmd/llamay`	The CLI, and the `argv[0]` trick that makes `llamay-server` serve with no subcommand.
`pkg/store`	The model store: content-addressed blobs and manifests. A blob is only ever installed under the digest of its own finished bytes.
`pkg/ollama`	Reading a local Ollama store, if there is one. A courtesy, not a dependency.
`pkg/gguf`	The file format: parse, mmap, write. 23 tensor formats read; `IQ1_S` and `IQ1_M` deliberately absent, so a file carrying them is refused rather than read as garbage.
`pkg/arch`	Which block shape a file describes, and what a model identifies itself as when the architecture field names only the vehicle. 17 entries reached by 39 declared names.
`pkg/quant`	Block layouts, dequantisation, and the fused W4A8 kernels every one of them is tested against.
`pkg/tensor`	Matrices, `MatVec` for decode, `MatMul` for prefill, and the worker pool that stays off the efficiency cores.
`pkg/tok`	Byte-level BPE, SentencePiece and WordPiece, with four hand-written pre-tokenizers — hand-written because RE2 has no lookahead.
`pkg/model`	Weight binding and the forward pass. `State` holds one conversation's scratch; `Batcher` decodes several at once.
`pkg/kv`	The state engine: paging, the radix prefix tree, copy-on-write forks, versioned snapshots.
`pkg/sample`	Samplers and the constraint interface they call into.
`pkg/constrain`	JSON, JSON Schema, GBNF and lexicon constraints, compiled to byte-level pushdown automata.
`pkg/serve`	The HTTP surface, the scheduler, continuous batching and the context routes.
`pkg/vision` · `pkg/video`	Vision projectors, images and scanned PDF pages into positions, and clips into timed frames.
`pkg/whisper`	Whisper speech-to-text: whisper.cpp's model files, the mel spectrogram, encoder and greedy decoder.
`pkg/lora`	GGUF LoRA adapters, applied as low-rank corrections at the multiply.
`pkg/index` · `pkg/pii`	The document index with per-passage labels; rule-based redaction.
`pkg/audit` · `pkg/capture` · `pkg/seal`	The hash-chained audit record, encrypted content capture, and sealed snapshots.

Loading a file

A GGUF file is memory-mapped rather than read. The weights are the vast majority of the bytes, they are read once per token in a pattern the kernel's page cache is good at, and copying six gigabytes into the heap to then read it linearly buys nothing.

The audit at the end is the unusual step: llamay info -m file.gguf reports whether any tensor in the file went unread, which is how a half-wired architecture announces itself at load rather than as strange text an hour later.

The two phases, timed apart, because they are bound by different things:

Transcript — the whole session as text

~/llamay $ llamay bench -m qwen2.5:0.5b -gpu -prompt 512 -gen 128 -reps 3
llamay v0.1.58-1-gb14f550 · Qwen2.5 0.5B Instruct · qwen2 · 7 threads · kernels neon+i8mm ·
matmul metal:Apple M4 · gemm own kernels · kv f32
weights 373.71 MiB · repack on · context 32768 · prompt 512 · generate 128

phase     rate          best     note
prefill   794.3 tok/s   645ms    one command buffer per 256-token tile
decode    47.3 tok/s    2.705s   506 dispatches per token, one submission

cpu encoding 133ms · gpu execution 3.175s (4% encoding)

phase      rate   best   note
fork x8    11µs          0 pages allocated (a copy would be 64)
snapshot   35ms          12.00 MiB for 512 tokens, at f32
restore    34ms          vs 645ms to re-prefill

gpu time: staging 41ms · kernels 2.333s · copy-back 50ms

Restoring a 512-token context is 18.9x faster than recomputing it.
~/llamay $ # prefill is arithmetic-bound; decode is memory-bound. One number would hide that
.

Prefill and decode, measured. The command behind the table on the throughput page. It reports the two phases apart because they are bound by different things. Recorded 2026-09-18 · Apple M4, macOS 26.0.1 · llamay v0.1.58-1-gb14f550 · metal:Apple M4 · 0:09

The forward pass

Prefill and decode are the same code. Batched prefill is an optimisation of sequential decode, so if the two ever disagree, a benchmark of the engine is a benchmark of two different models. This is the first invariant llamay verify asserts, and it was written before any optimisation was.

The KV cache is the only thing in the block with memory across calls, which is why it is a separate package with its own invariants rather than a buffer inside the model.

Two constraints shape the loop rather than the maths. There is no allocation inside the token loop — scratch buffers are allocated once per State — and the worker count avoids efficiency cores, because a pool sized to the core count schedules a third of its work onto cores a fifth as fast and then waits for them.

And the lifecycle of one request, from a server that was not running a moment earlier:

Transcript — the whole session as text

~/llamay $ llamay serve -m qwen2.5:0.5b -addr localhost:11439
time=2026-09-18T08:49:09.326-04:00 level=INFO msg="device graph attached" max-tokens=256
time=2026-09-18T08:49:09.326-04:00 level=INFO msg="llamay serving" addr=localhost:11439 mode
l=qwen2.5:0.5b arch=qwen2 max-loaded=2 keepalive=5m0s params="0.36 GB" context=32768 threads
=7 kv=f32 lexicon=0

  OpenAI      POST http://localhost:11439/v1/chat/completions
  Anthropic   POST http://localhost:11439/v1/messages
  Embeddings  POST http://localhost:11439/v1/embeddings     (start with -embed <encoder.gguf
>)
  Rerank      POST http://localhost:11439/rerank            (start with -rerank <reranker.gg
uf>)
  Contexts    POST http://localhost:11439/v1/contexts        (then /{id}/fork?n=8, /{id}/sna
pshot)
  Models      POST http://localhost:11439/api/pull          (and /api/copy, /api/delete)
  Stats       GET  http://localhost:11439/v1/stats


~/llamay $ # the server stays up; the rest of this is a second terminal
~/llamay $ curl -s http://localhost:11439/v1/models | jq
{
  "data": [
    {
      "created": 1789735749,
      "id": "all-minilm:latest",
      "object": "model",
      "owned_by": "azmx"
    },
    {
      "created": 1789735749,
      "id": "gemma2:2b",
      "object": "model",
      "owned_by": "azmx"
    },
    {
      "created": 1789735749,
      "id": "gemma3:270m",
      "object": "model",
      "owned_by": "azmx"
    },
    {
      "created": 1789735749,
      "id": "gpt2:q8_0",
      "object": "model",
      "owned_by": "azmx"
    },
    {
      "created": 1789735749,
      "id": "qwen2.5:0.5b",
      "object": "model",
      "owned_by": "azmx"
    },
    {
      "created": 1789735749,
      "id": "smollm2:135m",
      "object": "model",
      "owned_by": "azmx"
    },
    {
      "created": 1789735749,
      "id": "azmx-code-test:latest",
      "object": "model",
      "owned_by": "azmx"
    }
  ],
  "object": "list"
}
~/llamay $ curl -s http://localhost:11439/v1/chat/completions -H 'content-type: application/
json' \
>   -d '{"model":"qwen2.5:0.5b","messages":[{"role":"user","content":"One sentence: why run
a model locally?"}]}' | jq
{
  "choices": [
    {
      "finish_reason": "stop",
      "index": 0,
      "message": {
        "content": "running a model locally can be beneficial because it allows you to optim
ize the model locally, which can be faster and more efficient compared to running the model
globally.",
        "role": "assistant"
      }
    }
  ],
  "created": 1789735751,
  "id": "chatcmpl-llamay",
  "model": "qwen2.5:0.5b",
  "object": "chat.completion",
  "usage": {
    "completion_tokens": 31,
    "llamay_cached_prompt_tokens": 0,
    "llamay_decode_ms": 1528.48,
    "llamay_prefill_ms": 153.028,
    "prompt_tokens": 17,
    "prompt_tokens_details": {
      "cached_tokens": 0
    },
    "total_tokens": 48
  }
}
~/llamay $ # the same route with stream:true — this is the wire format
~/llamay $ curl -sN http://localhost:11439/v1/chat/completions -H 'content-type: application
/json' \
>   -d '{"model":"qwen2.5:0.5b","stream":true,"messages":[{"role":"user","content":"Count to
 five."}]}' | head -c 520
data: {"choices":[{"delta":{"content":"Count"},"finish_reason":null,"index":0}],"created":17
89735751,"id":"chatcmpl-llamay","model":"qwen2.5:0.5b","object":"chat.completion.chunk"}

data: {"choices":[{"delta":{"content":"ing"},"finish_reason":null,"index":0}],"created":1789
735751,"id":"chatcmpl-llamay","model":"qwen2.5:0.5b","object":"chat.completion.chunk"}

data: {"choices":[{"delta":{"content":" to"},"finish_reason":null,"index":0}],"created":1789
735751,"id":"chatcmpl-llamay","model":"qwen2.5:0.5b","object":"cha~/llamay $ # and the same
stream read the way a client reads it
~/llamay $ curl -sN http://localhost:11439/v1/chat/completions -H 'content-type: application
/json' \
>   -d '{"model":"qwen2.5:0.5b","stream":true,"messages":[{"role":"user","content":"Count to
 five, then stop."}]}' \
>   | python3 scripts/site/demo_sse.py
1
2
3
4
5

An OpenAI endpoint on your own machine. llamay serve, then the request every client already sends — non-streaming, then streaming, with the base URL as the only thing that changed. Recorded 2026-09-18 · Apple M4, macOS 26.0.1 · llamay v0.1.58-1-gb14f550 · metal:Apple M4 · 0:29

A request, end to end

Prefill is chunked so one long prompt cannot stall everybody else; decode is batched across sequences so N generations read the weights once rather than N times.

The step that matters is where the KV context is acquired: at admission, not at submission. That is what makes the queue safe to have at all. A queued request is a channel and a few hundred bytes, so the memory ceiling is the batch width rather than the queue depth, and a burst of a thousand requests costs a thousand small structs instead of a thousand caches.

The prefix cache is on the same path

Before prefill runs, Cache.Match(tokens) walks the radix tree and returns a context holding the longest prefix already computed. Only the new suffix is prefilled, and the response reports prompt_tokens_details.cached_tokens beside the usual usage — the number that says whether the prefix cache is earning its memory. How the tree works.

Why Go

Every hot loop is destined to be assembly or a GPU shader, so the language's job is orchestration. There, Go's static single binary, its cross-compilation and its scheduler are worth more than a marginal codegen advantage over C — and they are what let one source tree build every platform in a release.

The costs are real and planned for rather than discovered: no allocation inside the token loop, a worker count that stays off efficiency cores, and hand-written assembly for all eight block formats on both architectures. The portable Go reference kernels stay in the tree and are correct — 2.8× to 7.6× slower per dot product on an M4 — and LLAMAY_BACKEND=portable runs the whole suite through them on a machine that has SIMD, because building a fallback is not the same as running it.