Guides

Serving

llamay serve is one process holding one or more models, a scheduler in front of them, and four API families on one address. This page is what to set and why, rather than a list of flags with their help strings repeated.

Starting it

llamay serve                                  # first decoder in the store, 127.0.0.1:11435
llamay serve -m gemma3:270m -addr :8080       # a specific model, a specific address
llamay serve -batch 8 -max-loaded 3           # continuous batching, three models resident
llamay serve -m azmx-one-q4.gguf \
  -embed all-minilm.gguf -rerank ms-marco-MiniLM-L6-v2.gguf

With no -m, serve takes the first decoder in the store, sorted by name so a restart serves the same one. A decoder rather than simply the first name, because an encoder-only server refuses every generation route — and on a store holding four decoders and one all-minilm, sorting alone served the encoder. Both llamay's own store and Ollama's are searched, which is the same pair -m resolves against. The model named by -m is pinned, which is what makes a request that names nothing always have an answer.

An empty store is not an error

On a machine with no models yet, serve starts anyway and answers the routes that mean something without one: /healthz, empty /v1/models and /api/tags, and POST /api/pull. Everything else answers 503 with the reason and the command that fixes it. The moment a model lands — from that route or from llamay pull in a terminal — the same process loads it and starts serving properly, with no restart.

It matters because every packaged install starts the server with no -m, so "empty store" and "fresh install" are the same moment. This used to exit, and a supervisor that restarts on exit turned that into a loop every three seconds — while the macOS app, which pulls its first model through /api/pull, had no server to pull through.

The scheduler

ARRIVAL request holds no memory yet BOUNDED QUEUE a channel and a few hundred bytes ADMISSION — CONCURRENCY 2 KV acquired here not at submission Full → 429, immediately the ceiling is the batch width, not the queue depth EACH SCHEDULER STEP DOES ONE OF TWO THINGS Prefill — 256 tokens per step The prefix tree is matched first, so only the new suffix costs anything. What is left is cut into chunks, so a 30,000-token prompt cannot hold every other sequence still while it runs. -prefill-chunk 256 Decode — batched across sequences N generations read the weights once rather than N times, which is where the throughput of a busy server comes from. Batch composition never changes the output; verify asserts that. -batch 0 (A GOROUTINE EACH) OR -batch N
Acquiring the KV context at admission rather than at submission is the design decision that makes the queue safe: a burst of a thousand requests costs a thousand small structs, not a thousand caches.

Flags

FlagDefaultMeaning
-addr127.0.0.1:11435Listen address.
-mfirst in the storeThe pinned model. A name, or a path — a path that exists always wins.
-concurrency2Generations allowed to run at once.
-batch0Sequences decoded together. 0 runs each generation on its own goroutine.
-prefill-chunk256Prompt tokens prefilled per scheduler step, so a long prompt does not stall decodes.
-prefix-entries32How many shared prefixes the radix tree keeps. An entry holds page references, so this is a memory bound as much as a hit-rate one.
-max-loaded2Models resident at once. A request for another loads it and may evict an idle one.
-keepalive5mHow long an idle model stays loaded. 0 keeps every loaded model forever.
-kvf32Cache precision: f32, vq8 or q8. The trade.
-api-key—Required on every route but /healthz. Prefer LLAMAY_API_KEY.
-embed · -rerank—An encoder and a cross-encoder served beside the decoder, in the same process.
-pulltrueWhether POST /api/pull, /api/copy and DELETE /api/delete may write to the store.
-otlp · -traces—OTLP/HTTP collector for spans. Also read from OTEL_EXPORTER_OTLP_ENDPOINT.
-gpu · -ngloffRun the layer stack on the device. -ngl is llama.cpp's spelling of the same thing.
-voffLog every request.

Model residency

The server loads models on demand. A request naming any model this machine has loads it, evicting one that has gone idle. 

llamay serve -m qwen2.5:0.5b -max-loaded 3 -keepalive 5m
curl localhost:11435/api/chat -d '{"model":"gemma3:270m","messages":[…]}'   # loads it
curl localhost:11435/api/ps                                                 # what is resident

Two properties are worth stating because they are the ones a naive implementation gets wrong. A model is never unloaded while a request is using it — eviction counts references rather than trusting the idle timer to be slower than a generation. And concurrent requests for the same unloaded model produce one load, not one each.

Authentication

-api-key, or LLAMAY_API_KEY, requires a key on every route but /healthz — which stays open because the thing reading it is usually a container probe with no way to carry a secret.

The environment is the better of the two

A key on the command line is visible in the process list to everyone on the machine. And without a key the server is open, which is the right default for something bound to localhost and the wrong one the moment -addr 0.0.0.0 is typed — along with -pull=false, since anyone who can reach /api/pull can make the process fetch a URL and spend the machine's disk on it.

Health, metrics and traces

RouteWhat it answers
GET /healthz{"status":"ok","model":…,"id":…}. This is identity, not liveness — a 200 from a port tells you something is listening, not what.
GET /metricsPrometheus, including the scheduler's own counters: queue depth, admissions, prefill and decode tokens, evictions.
GET /v1/statsThe same numbers as JSON.
OTLPSpans for the request, the queue wait, prefill and decode, joining the caller's traceparent, on both the sequential and the -batch path.

Running it as llama-server

make server also produces bin/llamay-server, which is a link to the same binary: the program reads argv[0] and preselects serve under that name, so deployment tooling written for llama-server needs no subcommand. One artifact, two names, nothing that can drift.