Capabilities

Getting and managing models

llamay runs GGUF files. It has its own downloader and its own store, reads Ollama's store if you have one, and serves LoRA adapters as models of their own. This page is how to get a model, where it goes, and how to account for every one on the machine.

Pulling

llamay pull qwen2.5:0.5b                                          # Ollama's registry
llamay pull hf.co/Qwen/Qwen2.5-0.5B-Instruct-GGUF:q4_k_m          # Hugging Face, quantisation as the tag
llamay pull hf:Qwen/Qwen2.5-0.5B-Instruct-GGUF/qwen2.5-0.5b-instruct-q4_k_m.gguf   # one exact file
llamay pull https://example.com/models/model-q4.gguf -as mymodel:q4   # any URL serving a GGUF

Source	Means
`<name>[:<tag>]`	An Ollama-style registry, `registry.ollama.ai` by default.
`hf.co/<owner>/<repo>[:<quant>]`	A Hugging Face repository; the tag picks the quantisation.
`hf:<owner>/<repo>/<file.gguf>[@revision]`	An exact file, optionally pinned to a commit.
`https://…`	Any URL serving a GGUF: a mirror, an object store.

Downloads resume, and a file is installed only under the SHA-256 of its finished bytes.
-as name:tag names it; -force downloads again.
HF_TOKEN is sent to Hugging Face when it is set, which gated repositories need.
A server can pull too: POST /api/pull, unless it was started with -pull=false.

The store

$ llamay models
name                                     size  architecture           store
all-minilm:latest                    43.8 MiB  bert (embedder)        llamay
gemma2:2b                             1.6 GiB  gemma2 (decoder)       llamay
gemma3-4b-mmproj:latest             811.8 MiB  unsupported            llamay
gemma3-4b-vision:latest               2.3 GiB  gemma3 (decoder)       llamay
qwen2.5:0.5b                        379.4 MiB  qwen2 (decoder)        llamay
…

8 in ~/.llamay/models, 1 in ~/.ollama/models

Where: ~/.llamay/models, or LLAMAY_MODELS. The Linux service uses /var/lib/llamay/models, shared with members of the llamay group.
How: blobs named by their SHA-256, and small manifests that name them. Two names for one model are one file.
Ollama's store is read too, when there is one: models Ollama pulled appear with ollama in the last column and are served without a second download. llamay never writes there.
A path always wins. -m ./model.gguf means that file, whatever the stores hold. (A projector shows as unsupported because it is not a model on its own; it is used with -mmproj.)

llamay cp qwen2.5:0.5b small:latest     # a second name, not a second copy
llamay rm small:latest                  # removes the name; the blob goes when nothing names it
llamay info -m qwen2.5:0.5b             # architecture, shapes, quantisation, and whether any tensor went unread

Which model

Memory	Comfortable	Notes
8 GB	3B at `q4_k_m`	About 2 GB of weights; leaves room for the cache
16 GB	7B–8B at `q4_k_m`	About 4.5 GB; the common choice
32 GB	14B, or 7B at `q8_0`	A bigger model at lower precision usually beats a smaller one at higher
64 GB+	32B–70B	Watch the KV cache, not the weights

The architectures llamay runs, and the ones it refuses, are on models and formats.

LoRA adapters, as models

A LoRA adapter is a small file that specialises a base model. Name each one when the server starts, and clients pick it with the ordinary model field, as <base>+<name>:

llamay serve -m qwen2.5:7b -lora legal=legal-lora.gguf -lora claims=claims-lora.gguf
curl -s localhost:11435/v1/chat/completions -d '{"model": "qwen2.5:7b+legal", "messages": [...]}'

The adapter is a GGUF LoRA file, as llama.cpp's converter writes it.
It is checked against the base at startup — the architecture, and the name and shape of every matrix it corrects — so an adapter for another model stops the server.
The base and every adapter share one copy of the weights. Each adapter has its own cache, so a prefix computed under one is never reused under another.
Adapters appear in /v1/models and load on first use.
Adapter requests run on the per-matmul path, not a whole-model device graph, so on a GPU they are slower than the base.
Adapters are served, not trained: there is no fine-tuning in llamay.

An inventory, with evidence

llamay inventory lists every model the machine can serve: its name, the SHA-256 of the exact file, architecture, parameter count, licence, and where it was pulled from.

$ llamay inventory
model inventory · host.example · llamay 0.3.0 · 2026-09-27T04:24:21Z

name                     digest               architecture  parameters  license     source                                            validated
all-minilm:latest        sha256:797b70c4edf8  bert          23M         -           all-minilm:latest                                 -
gemma2:2b                sha256:e0aee85060f1  gemma2        2.6B        gemma       hf:bartowski/gemma-2-2b-it-GGUF/gemma-2-2b-it...  -
qwen2.5:0.5b             sha256:c5396e06af29  qwen2         494M        apache-2.0  qwen2.5:0.5b                                      -
…

The second field of the first line is the machine's host name, shown here as host.example.

-validate puts every language model through llamay verify — prefill against decode, fork and snapshot exactness, thread-count invariance, the quantised cache's bound — and records each result with the date. -json -o inventory.json writes it for a model-risk register: which file, from where, validated when, by which build. It loads every model, so it is slow.

Quantising

llamay imatrix  -m model-f16.gguf -f calibration.txt -o imatrix.dat   # which columns your text drives
llamay quantize -i model-f16.gguf -o model-q4_k.gguf -type q4_k -imatrix imatrix.dat
llamay ppl      -m model-q4_k.gguf -f held-out.txt                     # measure what it cost

quantize writes q8_0, q6_k, q5_0, q4_k, q4_0, f16 and f32. It reads the i-quants and MXFP4 but does not write them. Compare perplexity before and after rather than assuming.