Capabilities

Contexts: fork, save and move

A context is the model's computed state for a prompt — the part that costs time to build. llamay lets you build it once and then fork it, save it to a file, move it to another server, and let idle ones wait on disk. Every example below ran against llamay 0.3.0. How it works underneath.

What it is for

Four workloads, one shape: an agent exploring alternatives from a shared history; best-of-N sampling; many documents processed against one long instruction; a session that has to survive a restart. Each re-reads the same prompt again and again. A context reads it once.

One prefill, eight branches, and the copy that was not made:

~/llamay $ llamay ctx build -m qwen2.5:0.5b -o build/demo/work/base.llamayctx -p "You are re
viewing a database migration. The table is users, the column is email, and the index is uniq
ue."
prefilled 23 tokens in 175ms
saved build/demo/work/base.llamayctx (552.14 KiB) in 7ms
~/llamay $ ls -lh build/demo/work/base.llamayctx | awk '{print $5, $9}'
552K build/demo/work/base.llamayctx
~/llamay $ # eight branches off that one prefill
~/llamay $ llamay ctx fork -m qwen2.5:0.5b -i build/demo/work/base.llamayctx -n 8
source                    build/demo/work/base.llamayctx, 23 tokens, 1 pages
branches                  8 in 1µs (200ns each)
pages allocated           0
pages a copy would need   8
memory held               1.50 MiB (a copy would be 12.00 MiB)
~/llamay $ # pages allocated: 0. The branches share every page until one writes.

Build and fork one over HTTP

curl -s -X POST localhost:11435/v1/contexts -d '{"messages": [{"role": "system",
  "content": "You label support tickets with one word: billing, bug, feature, account or other."}]}'

{"created":1790481689,"id":"ctx_41b093a8375cd2661abf1c6cc33e7c7b","pages":1,"parent":"","shared_pages":0,"tokens":26}

curl -s -X POST "localhost:11435/v1/contexts/ctx_41b093a8375cd2661abf1c6cc33e7c7b/fork?n=2"

{"data":[{"created":1790481689,"id":"ctx_2020c1df813426e785e476426bed6c1c","pages":1,"parent":"ctx_41b093a8375cd2661abf1c6cc33e7c7b","shared_pages":1,"tokens":26},
         {"created":1790481689,"id":"ctx_529756752580da6cce4f83ef03d19f33","pages":1,"parent":"ctx_41b093a8375cd2661abf1c6cc33e7c7b","shared_pages":1,"tokens":26}],
 "fork_ms":0.009}

Each branch holds all 26 tokens and shares its parent's page: shared_pages is 1 and nothing was copied. A branch copies a page only when it writes to one.

Use one in a request

Name the context with llamay_session. The request must start with exactly what the context holds — here, the same system message — followed by the new turn:

curl -s localhost:11435/v1/chat/completions -d '{
  "model": "qwen2.5:0.5b", "temperature": 0, "max_tokens": 4,
  "llamay_session": "ctx_2020c1df813426e785e476426bed6c1c",
  "messages": [
    {"role": "system", "content": "You label support tickets with one word: billing, bug, feature, account or other."},
    {"role": "user", "content": "my card was charged twice"}]}'

{"choices":[{"finish_reason":"stop","index":0,"message":{"content":"billing","role":"assistant"}}], ...
 "usage":{"completion_tokens":1,"llamay_cached_prompt_tokens":24,"prompt_tokens":36,"prompt_tokens_details":{"cached_tokens":24}, ...}}

24 of the 36 prompt tokens came from the context; only the new turn was computed.

The context grows with the conversation: after the request, it holds the prompt and the answer, and the next turn continues from there. usage.prompt_tokens_details.cached_tokens (OpenAI routes) and cache_read_input_tokens (Anthropic routes) say how much was reused. Ollama's /api/chat takes llamay_session too.

One request writes to a context at a time; a second gets 409. Fork it for parallel work.

The same object over HTTP, which is how a fleet of agents shares one prefill:

~/llamay $ llamay serve -m qwen2.5:0.5b -addr localhost:11439
time=2026-09-18T08:49:07.893-04:00 level=INFO msg="device graph attached" max-tokens=256
time=2026-09-18T08:49:07.894-04:00 level=INFO msg="llamay serving" addr=localhost:11439 mode
l=qwen2.5:0.5b arch=qwen2 max-loaded=2 keepalive=5m0s params="0.36 GB" context=32768 threads
=7 kv=f32 lexicon=0

  OpenAI      POST http://localhost:11439/v1/chat/completions
  Anthropic   POST http://localhost:11439/v1/messages
  Embeddings  POST http://localhost:11439/v1/embeddings     (start with -embed <encoder.gguf
>)
  Rerank      POST http://localhost:11439/rerank            (start with -rerank <reranker.gg
uf>)
  Contexts    POST http://localhost:11439/v1/contexts        (then /{id}/fork?n=8, /{id}/sna
pshot)
  Models      POST http://localhost:11439/api/pull          (and /api/copy, /api/delete)
  Stats       GET  http://localhost:11439/v1/stats


~/llamay $ # the server stays up; the rest of this is a second terminal
~/llamay $ curl -s http://localhost:11439/v1/contexts -H 'content-type: application/json' \
>   -d '{"model":"qwen2.5:0.5b","prompt":"You are a careful code reviewer."}' | jq
{
  "created": 1789735748,
  "id": "ctx_1",
  "pages": 1,
  "parent": "",
  "shared_pages": 0,
  "tokens": 7
}
~/llamay $ curl -s http://localhost:11439/v1/contexts | jq
{
  "data": [
    {
      "created": 1789735748,
      "id": "ctx_1",
      "pages": 1,
      "parent": "",
      "shared_pages": 0,
      "tokens": 7
    }
  ]
}
~/llamay $ # fork the one that exists, four ways
~/llamay $ ID=$(curl -s http://localhost:11439/v1/contexts | jq -r '.data[0].id') && \
>   curl -s http://localhost:11439/v1/contexts/$ID/fork -H 'content-type: application/json'
\
>   -d '{"n":4}' | jq
{
  "data": [
    {
      "created": 1789735748,
      "id": "ctx_2",
      "pages": 1,
      "parent": "ctx_1",
      "shared_pages": 1,
      "tokens": 7
    }
  ],
  "fork_ms": 0.009
}
~/llamay $ curl -s http://localhost:11439/v1/contexts | jq '.data | length'
2

The routes

Each takes ?model= for a model other than the default (creating one takes model in the body instead): a context belongs to the model that made it. Bodies and errors are on the API reference.

How long a context lives

• Two hours unused, then closed. The server sweeps idle contexts so a client that forgets them cannot fill memory. Anything mid-request is never taken. -session-ttl changes the interval; a negative value keeps every context until it is deleted.
• Treat the id as a cache, not a handle. A request naming one the server no longer has gets 404: make another and send again, which is what the apps do.
• A restart loses them, unless you saved a snapshot.
• They belong to their caller. On a server with named callers, another caller's context is a 404, and with classification markings so is your own at another marking.

Save one to a file

curl -s localhost:11435/v1/contexts/$ID/snapshot -o base.llamayctx             # 639,207 bytes
curl -s "localhost:11435/v1/contexts/$ID/snapshot?precision=q8" -o base-q8.llamayctx  # 199,911 bytes
curl -s -X POST localhost:11435/v1/contexts/restore --data-binary @base.llamayctx

{"created":1790481689,"id":"ctx_62d6934d40785bc44ba39f148a9045c0","pages":1,"parent":"","shared_pages":0,"tokens":26}

precision is f32 (the default, exact), vq8 or q8 — about a third of the size, and lossy. From the command line the same object is llamay ctx build, fork and restore:

llamay ctx build   -m qwen2.5:0.5b -p "$(cat long-system-prompt.txt)" -o base.llamayctx
llamay ctx fork    -m qwen2.5:0.5b -i base.llamayctx -n 8 -o branches/
llamay ctx restore -m qwen2.5:0.5b -i base.llamayctx -p "now answer:" -n 200

Without -p, ctx restore continues the snapshot where it stopped. -temp 0 is greedy, so a restored continuation can be compared token for token with llamay run -temp 0.

A context written to a file and picked up again by a different process:

~/llamay $ # build a context and put it on disk
~/llamay $ llamay ctx build -m qwen2.5:0.5b -o build/demo/work/session.llamayctx -p "Notes f
rom the incident: the queue backed up at 02:14, the consumer had been restarted at 02:09, an
d the dead-letter topic was empty."
prefilled 38 tokens in 244ms
saved build/demo/work/session.llamayctx (912.20 KiB) in 6ms
~/llamay $ ls -lh build/demo/work/session.llamayctx | awk '{print $5, $9}'
912K build/demo/work/session.llamayctx
~/llamay $ # a different process, later: restore it rather than prefill again
~/llamay $ llamay ctx restore -m qwen2.5:0.5b -i build/demo/work/session.llamayctx
restored 38 tokens in 2ms

The same weights, or a refusal

KV computed under different weights is not detectably wrong at generation time — it produces fluent, confident nonsense — so it has to fail when it is loaded. A snapshot records the model's layout and the SHA-256 of its weights file, and a restore checks both:

curl -s -X POST "localhost:11435/v1/contexts/restore?model=smollm2:135m" --data-binary @base.llamayctx

{"error":{"message":"kv: snapshot was taken with different weights: snapshot holds \"590bc10b059c7189\", this cache holds \"3920124fb79ca350\"","type":"Conflict"}}

A fine-tune shares its base's layout, which is why the SHA-256 is checked too. Snapshots from before 0.3.0 carry no SHA-256 and restore by layout alone, with a warning — except under -profile enterprise or on a request with a classification marking, where they are refused unless the server has -allow-legacy-snapshots.

The same rule from the CLI:

~/llamay $ llamay ctx build -m qwen2.5:0.5b -o build/demo/work/from-qwen.llamayctx -p "Two p
lus two is"
prefilled 4 tokens in 47ms
saved build/demo/work/from-qwen.llamayctx (96.07 KiB) in 5ms
~/llamay $ # now try to restore it under a different model
~/llamay $ llamay ctx restore -m smollm2:135m -i build/demo/work/from-qwen.llamayctx
llamay ctx: kv: snapshot was taken with different weights: snapshot holds "590bc10b059c7189"
, this cache holds "3920124fb79ca350"
~/llamay $ # refused, named, and non-zero — not a silent wrong answer

Move one to another server

llamay ctx move streams a context from one running server into another that holds the same model file, then deletes it from the first:

llamay ctx move -from http://localhost:11435 -to http://localhost:11436 -id ctx_b5d094a25688d4212869701db58dd5a0

moved ctx_b5d094a25688d4212869701db58dd5a0 (7 tokens, 168.4 KiB) to http://localhost:11436 as ctx_715e4f8b3b765485c9e5ea51ab2b4ded in 3ms — 52.3 MB/s
dropped from http://localhost:11435

• The source is deleted only after the destination has answered 201. -keep leaves it, which makes the move a copy.
• An export carries its owner and marking: it lands only for the same caller at the same marking. -key, -from-key, -to-key, -classification and the certificate flags authenticate to each side.
• The destination checks the same weights, layout and SHA-256, and answers 409 otherwise.
• -precision q8 sends a third of the bytes, lossily. The default is the cache's own precision, the only one that arrives exact.
• Underneath it is two routes, GET /v1/contexts/{id}/export and POST /v1/contexts/import, which any client can call.

On a server enforcing classification markings, a bare snapshot is refused — it carries no marking — and a context moves only as an export sealed under -context-key, so the marking cannot be edited on the way.

Encrypted snapshots

A snapshot is the conversation: it holds the model's state for every token. Give the server a key and every snapshot and export it hands out is encrypted with AES-256-GCM, in authenticated 1 MiB chunks; a truncated, reordered or edited file is refused.

openssl rand -hex 32 > context.key
llamay serve -context-key context.key
llamay ctx build -m qwen2.5:0.5b -key context.key -p "..." -o base.llamayctx   # the CLI takes -key

A sealed snapshot sent to a server without the key is refused: this snapshot is encrypted, and this server has no -context-key to open it.

More contexts than memory

-ctx-mem-budget 8G keeps at most that much context memory per model. Past it, idle contexts are written to disk, encrypted, and read back exactly when next used. Here three contexts under a 1 MB budget:

{"data":[{"id":"ctx_2a5784e068b842e2a9cd0dad0614aa8d","pages":0,"spilled":true,"tokens":206, ...},
         {"id":"ctx_82db6136779fc5a08082efeedf8d1813","pages":0,"spilled":true,"tokens":206, ...},
         {"id":"ctx_4121b1a231b6b769bcaabd0eb43406e1","pages":0,"spilled":true,"tokens":206, ...}]}

Forking the first one read it back — llamay_context_restores_total went from 0 to 1 — and the fork shared all four of its pages. See spill.

Two rules that are not negotiable