Build with llamay

Recipes

Things people actually build, each one complete enough to paste into a terminal. Every recipe runs against a local llamay serve on 127.0.0.1:11435 and sends nothing anywhere.

llamay pull hf:Qwen/Qwen2.5-7B-Instruct-GGUF/qwen2.5-7b-instruct-q4_k_m.gguf
llamay serve

Which shape is your problem?

1. Pull structured data out of unstructured text

The most common thing anyone builds, and where a schema earns its keep: response_format is enforced while sampling, so the reply parses or the request fails. It never returns prose that merely looks like JSON.

curl -s http://127.0.0.1:11435/v1/chat/completions \
  -H 'content-type: application/json' -d '{
  "model": "",
  "messages": [{"role": "user", "content":
    "Invoice 4471 from Acme Ltd, dated 3 March 2026, total 1,284.50 EUR, due in 30 days."}],
  "response_format": {
    "type": "json_schema",
    "json_schema": {"name": "invoice", "schema": {
      "type": "object",
      "properties": {
        "number":   {"type": "string"},
        "supplier": {"type": "string"},
        "date":     {"type": "string"},
        "total":    {"type": "number"},
        "currency": {"type": "string", "enum": ["EUR", "USD", "GBP"]},
        "due_days": {"type": "integer"}
      },
      "required": ["number", "supplier", "total", "currency"]
    }}
  }
}' | jq -r '.choices[0].message.content'

{"number":"4471","supplier":"Acme Ltd","date":"2026-03-03",
 "total":1284.5,"currency":"EUR","due_days":30}

enum is worth more than it looks. The sampler cannot emit a currency outside the three, so a downstream ledger never sees one — the constraint shapes the token distribution rather than being checked afterwards.

2. Classify a backlog without paying for the prompt each time

Ten thousand tickets, one instruction. Send the instruction once, fork the context, and each item pays only for its own tokens. The saving is the prefill, which on a long instruction is most of the work.

import json, urllib.request

BASE = "http://127.0.0.1:11435"

def post(path, body):
    r = urllib.request.Request(BASE + path, data=json.dumps(body).encode(),
                               headers={"content-type": "application/json"})
    return json.load(urllib.request.urlopen(r))

RULES = (
    "You label support tickets. Answer with exactly one word from: "
    "billing, bug, feature, account, other. No punctuation, no explanation."
)

# The instruction, computed once and kept on the device.
ctx = post("/v1/contexts", {"messages": [{"role": "system", "content": RULES}]})

tickets = ["card declined again", "app crashes on export",
           "can you add dark mode", "cannot reset my password"]

for t in tickets:
    fid = post("/v1/contexts/" + ctx["id"] + "/fork?n=1", {})["data"][0]["id"]
    out = post("/v1/chat/completions", {
        "model": "", "llamay_session": fid, "max_tokens": 4, "temperature": 0,
        "messages": [{"role": "user", "content": t}],
    })
    print(out["choices"][0]["message"]["content"].strip().ljust(10), t)

billing    card declined again
bug        app crashes on export
feature    can you add dark mode
account    cannot reset my password

A fork is a page-table copy, not a memory copy — see the state engine. Forking a 4,000-token instruction a thousand ways costs a thousand small page tables, not four million tokens of KV.

3. Answer from your own documents

Retrieval, reranking and generation in one process. Start the server with an encoder and a cross-encoder alongside the decoder and nothing leaves the box.

llamay serve \
  -m qwen2.5-7b-instruct-q4_k_m.gguf \
  -embed bge-small-en-v1.5-f16.gguf \
  -rerank bge-reranker-base-f16.gguf

docs = [
    "Refunds are issued to the original payment method within 5 business days.",
    "Our warranty covers manufacturing defects for 24 months from purchase.",
    "Shipping is free on orders above 50 EUR within the EU.",
]
question = "how long does a refund take"

ranked = post("/rerank", {"query": question, "documents": docs, "top_n": 2})
top = [docs[r["index"]] for r in ranked["results"]]

answer = post("/v1/chat/completions", {
    "model": "", "temperature": 0, "max_tokens": 120,
    "messages": [
        {"role": "system", "content":
         "Answer only from the passages given. If they do not say, say so."},
        {"role": "user", "content":
         "Passages:\n- " + "\n- ".join(top) + "\n\nQuestion: " + question},
    ],
})
print(answer["choices"][0]["message"]["content"])

4. Let the model call your functions

Tool arguments are generated under the tool's own JSON Schema, so a call either fits the signature or is not produced. You never parse a hallucinated argument list.

TOOLS = [{
    "type": "function",
    "function": {
        "name": "search_orders",
        "description": "Find orders for a customer.",
        "parameters": {
            "type": "object",
            "properties": {
                "email":  {"type": "string"},
                "status": {"type": "string", "enum": ["open", "shipped", "refunded"]},
            },
            "required": ["email"],
        },
    },
}]

msgs = [{"role": "user", "content": "any refunded orders for [email protected]?"}]
out = post("/v1/chat/completions", {"model": "", "tools": TOOLS, "messages": msgs})

call = out["choices"][0]["message"]["tool_calls"][0]
args = json.loads(call["function"]["arguments"])   # guaranteed to fit the schema
rows = search_orders(**args)                       # your code

msgs += [
    out["choices"][0]["message"],
    {"role": "tool", "tool_call_id": call["id"], "content": json.dumps(rows)},
]
final = post("/v1/chat/completions", {"model": "", "tools": TOOLS, "messages": msgs})
print(final["choices"][0]["message"]["content"])

5. Redact before anything else sees it

A local model is the only kind you can put in front of a hosted one: the redaction step cannot leak what it is redacting, because it never leaves the machine.

curl -s http://127.0.0.1:11435/v1/chat/completions \
  -H 'content-type: application/json' -d '{
  "model": "", "temperature": 0,
  "messages": [
    {"role": "system", "content":
     "Replace every personal identifier with a tag: [NAME], [EMAIL], [PHONE], [ID], [ADDR]. Change nothing else. Output only the rewritten text."},
    {"role": "user", "content":
     "Ada Lovelace ([email protected], +44 7700 900123) reported fault 88213 at 12 Hanover Sq."}
  ]}' | jq -r '.choices[0].message.content'

[NAME] ([EMAIL], [PHONE]) reported fault [ID] at [ADDR].

6. Stream to a UI

Server-sent events, in the framing every OpenAI client already parses. The first frame carries the role and no text, which is what lets a client render an empty bubble immediately.

const res = await fetch("http://127.0.0.1:11435/v1/chat/completions", {
  method: "POST",
  headers: { "content-type": "application/json" },
  body: JSON.stringify({
    model: "", stream: true, max_tokens: 400,
    messages: [{ role: "user", content: "explain a bloom filter" }],
  }),
});

const reader = res.body.getReader();
const dec = new TextDecoder();
let buf = "";
for (;;) {
  const { value, done } = await reader.read();
  if (done) break;
  buf += dec.decode(value, { stream: true });
  const lines = buf.split("\n");
  buf = lines.pop();                      // keep the partial line
  for (const line of lines) {
    if (!line.startsWith("data:")) continue;
    const payload = line.slice(5).trim();
    if (payload === "[DONE]") return;
    const piece = JSON.parse(payload).choices[0]?.delta?.content;
    if (piece) process.stdout.write(piece);
  }
}

Buffer the tail. A chunk boundary can land mid-line, and a client that splits on newline without keeping the remainder drops a token every few hundred — rarely enough that it looks like the model's fault.

7. Stop at a delimiter

Useful when the model fills a template and the rest of the template is yours. The stop string is not included in the answer, and a sequence split across two tokens still stops.

curl -s http://127.0.0.1:11435/v1/chat/completions \
  -H 'content-type: application/json' -d '{
  "model": "", "stop": ["\n\n"], "max_tokens": 200,
  "messages": [{"role": "user", "content": "Write one paragraph about tides."}]
}' | jq -r '.choices[0].finish_reason'

stop

8. Move a warm context to another machine

Build the expensive context once — a codebase, a contract, a manual — write it to a file, and restore it where the work happens. The snapshot carries the model identity, and a restore against different weights is refused rather than producing plausible nonsense.

# on the machine with the corpus
ID=$(curl -s -X POST localhost:11435/v1/contexts \
      -H 'content-type: application/json' \
      -d "{\"prompt\": $(jq -Rs . < manual.txt)}" | jq -r .id)

curl -s "localhost:11435/v1/contexts/$ID/snapshot" > manual.ctx

# on another machine, with the same model loaded
NEW=$(curl -s -X POST localhost:11435/v1/contexts/restore \
        --data-binary @manual.ctx | jq -r .id)

curl -s localhost:11435/v1/chat/completions -H 'content-type: application/json' \
  -d "{\"model\":\"\",\"llamay_session\":\"$NEW\",
       \"messages\":[{\"role\":\"user\",\"content\":\"what does it say about warranty?\"}]}"

9. Run a nightly job over a corpus

Concurrency is the server's business, not yours: requests queue and batch against the same weights. Sending them in parallel fills the device; sending far too many at once only grows the queue.

import asyncio, aiohttp

BASE = "http://127.0.0.1:11435/v1/chat/completions"
SEM = asyncio.Semaphore(8)          # match -conc on the server

async def one(session, text):
    async with SEM:
        async with session.post(BASE, json={
            "model": "", "temperature": 0, "max_tokens": 160,
            "messages": [
                {"role": "system", "content": "Summarise in one sentence."},
                {"role": "user", "content": text},
            ],
        }) as r:
            d = await r.json()
            return d["choices"][0]["message"]["content"]

async def main(docs):
    async with aiohttp.ClientSession() as s:
        return await asyncio.gather(*(one(s, d) for d in docs))

print(asyncio.run(main(open("corpus.txt").read().split("\n\n"))))

10. Constrain output to a grammar

When a schema is the wrong shape — a query language, a DSL, a fixed vocabulary — GBNF constrains the sampler directly.

cat > move.gbnf <<'G'
root   ::= piece square square
piece  ::= "K" | "Q" | "R" | "B" | "N" | ""
square ::= [a-h] [1-8]
G

llamay run -m model.gguf --grammar move.gbnf \
  -p "White's best opening move, in algebraic notation:"

11. Grade your own prompts

The same split llamay uses on itself: deterministic assertions that need no model to judge, and a rubric scored by a stronger one. See Correctness for why the two halves are reported separately.

CASES = [
    {"ask": "is this spam? 'WIN A FREE IPHONE NOW'",   "want": "yes"},
    {"ask": "is this spam? 'your package arrives tuesday'", "want": "no"},
]

passed = 0
for c in CASES:
    out = post("/v1/chat/completions", {
        "model": "", "temperature": 0, "max_tokens": 4,
        "messages": [
            {"role": "system", "content": "Answer yes or no. One word."},
            {"role": "user", "content": c["ask"]},
        ],
    })["choices"][0]["message"]["content"]
    ok = out.strip().lower().startswith(c["want"])
    passed += ok
    print(("ok  " if ok else "FAIL"), c["ask"][:40], "->", out.strip())
print(passed, "/", len(CASES))

Twelve more, in brief

Where to go next

Deployment patterns covers the shapes these recipes sit inside — one machine, a queue, a fleet, an air-gapped rack. By sector is the same material organised by the constraint you work under rather than the thing you are building. The HTTP API is the exhaustive reference for every field used above.