Build with llamay

Deployment patterns

Nine shapes a llamay deployment takes, what each one is good at, and the failure each one has. Pick by the constraint that actually binds you — latency, isolation, cost per token, or a network that does not exist.

Choosing one

1. Sidecar

llamay runs beside your application, on the same host, bound to loopback. The application talks to 127.0.0.1:11435 and the port is reachable from nowhere else.

# docker-compose.yml
services:
  app:
    build: .
    environment:
      OPENAI_BASE_URL: http://llamay:11435/v1
      OPENAI_API_KEY: not-used-but-sdks-insist
  llamay:
    image: ghcr.io/azmxai/llamay:latest
    command: ["serve", "-addr", "0.0.0.0:11435", "-m", "/models/qwen.gguf"]
    volumes: ["./models:/models:ro"]
    # No ports: published. Only `app` can reach it, over the compose network.

Good at: the simplest thing that works, and the easiest to reason about in a review — there is no second machine to secure. Fails when: your application scales horizontally and every replica loads its own copy of the weights. At that point the weights, not the requests, are what you are paying for.

2. A pool behind a queue

One llamay per GPU, a queue in front, workers pulling from it. The queue is what gives you back-pressure; llamay's own scheduler batches whatever arrives.

Good at: throughput, and surviving a burst without dropping it. Fails when: you put a load balancer in front instead of a queue. Round-robin sends a request to a machine whose prefix cache has never seen it, which is the difference between a 200 ms answer and an 8 s one.

3. Cache affinity

If you must balance rather than queue, balance on a key that keeps a conversation on the machine that already holds it. Hash the session, not the request.

upstream llamay {
    hash $http_x_session consistent;
    server 10.0.0.11:11435;
    server 10.0.0.12:11435;
    server 10.0.0.13:11435;
}

Good at: preserving the prefix cache, which is most of the latency on a long system prompt. Fails when: a machine dies and every conversation it held reshuffles at once. Consistent hashing limits the blast radius; it does not remove it.

4. Air-gapped

No egress at all. The binary is static with no runtime dependency, so the transfer is a file: the executable, the weights, and a checksum.

# on a connected machine
llamay pull hf:Qwen/Qwen2.5-7B-Instruct-GGUF/qwen2.5-7b-instruct-q4_k_m.gguf
sha256sum llamay ~/.llamay/models/blobs/* > MANIFEST

# carry llamay, the blobs and MANIFEST across; on the far side
sha256sum -c MANIFEST
llamay serve -m qwen2.5-7b-instruct-q4_k_m.gguf -pull=false

-pull=false removes the write routes entirely, so a deployment that is not allowed to fetch anything also cannot be asked to. Fails when: nobody planned how a model gets updated. Write the transfer procedure down before the first one, not after the model is a year old.

5. Multi-tenant, one process

Contexts are the isolation boundary. A context belongs to the model it was created on and is addressed by an opaque id; one tenant cannot name another's.

# one context per tenant, created at sign-in and kept
ctx = post("/v1/contexts", {"messages": [{"role": "system", "content": tenant_rules}]})
store.put(tenant_id, ctx["id"])

# every later turn continues it, and pays only for the new tokens
post("/v1/chat/completions", {
    "model": "", "llamay_session": store.get(tenant_id),
    "messages": [{"role": "user", "content": turn}],
})

6. Local first, hosted on overflow

The local model answers what it can; anything it declines or takes too long on goes elsewhere. The interesting part is the redaction step in between, which only works because it is local.

7. On the user's own machine

Ship the engine with your application. No account, no key, no egress, and the feature works with the network off. The cost is distribution size and the first-run download of weights.

The macOS app is this pattern: a static binary, a model store under ~/.llamay, and a UI that talks to loopback. Nothing about it requires llamay to have written the UI.

8. In CI

A job that reviews a diff, drafts release notes or checks documentation has no key to configure and no egress rule to get approved, which is usually what kills these before they ship.

- name: Review the diff
  run: |
    curl -sL https://llamay.com/install.sh | sh
    llamay pull hf:Qwen/Qwen2.5-3B-Instruct-GGUF/qwen2.5-3b-instruct-q4_k_m.gguf
    llamay serve & sleep 5
    git diff origin/main... | python3 tools/review.py

Fails when: the runner is ephemeral and re-downloads the weights every build. Cache the model directory keyed on the model name, not on the commit.

9. Pre-warmed contexts

If every request shares a large prefix — a manual, a schema, a codebase — build it once at startup and fork per request. The first request after a deploy is otherwise the slowest one anybody sees.

def boot():
    ctx = post("/v1/contexts", {"prompt": open("handbook.txt").read()})
    app.state.base = ctx["id"]          # built once, before traffic

def handle(question):
    fid = post("/v1/contexts/" + app.state.base + "/fork?n=1", {})["data"][0]["id"]
    try:
        return post("/v1/chat/completions", {
            "model": "", "llamay_session": fid,
            "messages": [{"role": "user", "content": question}],
        })
    finally:
        delete("/v1/contexts/" + fid)   # forks are cheap, but they are not free

Sizing, honestly

The weights have to fit in memory the device can reach, and the KV cache grows with context length times concurrency. A 7B at q4_k_m is roughly 4.5 GB of weights; the cache is the number that surprises people, and vq8 roughly halves it at a cost the verifier measures rather than asserts.

Where to go next

Serving is the operational detail behind these shapes — the scheduler, residency, authentication and what is logged. Recipes is what runs inside them. By sector is the same question asked from the compliance side.