Ollama vs llama.cpp vs llamay: picking a local inference server
Three ways to serve a GGUF file on your own hardware, what each one is actually optimising for, and the one capability only one of them has.
All three run the same GGUF files. They differ in what they think the hard problem is.
The short version
| Ollama | llama.cpp | llamay | |
|---|---|---|---|
| Optimises for | Getting started | Raw control | Reusing computed state |
| Install | One installer | Build, or a binary | One static binary |
| Dependencies | None visible | Depends on build | None |
| API | Its own + OpenAI | Its own + OpenAI | OpenAI, Anthropic, Ollama |
| Model store | Its own | Files you manage | Its own, and reads Ollama’s |
| Structured output | Format flag | GBNF grammars | Schema enforced while sampling |
| Fork a warm context | — | — | Yes |
| Snapshot to a file | — | — | Yes |
Ollama
The one to pick if you want a model answering in two minutes and do not want to think about any of this. ollama run llama3.2 and you are done. The model library is curated, the defaults are sensible, and it hides the file layer completely.
What you give up is that hiding. When you want a specific quantisation of a specific file, or to know why something is slow, the abstraction is in the way.
llama.cpp
The reference implementation and the thing the others are measured against. Every knob is exposed, new architectures land here first, and if you want to understand exactly what your hardware is doing this is where you look.
The cost is that it is a toolkit rather than a product. You will build it, you will read the flags, and you will keep a note of which ones you used.
llamay
Same files, same quantisations, and it reads Ollama’s store, so switching costs no downloads. The API surface is wider — it serves OpenAI, Anthropic and Ollama shapes natively, so a client written for any of them works unchanged.
What it has that the other two do not is a state engine. A conversation’s key-value cache is a first-class object:
# Build an expensive context once
ID=$(curl -s -X POST localhost:11435/v1/contexts \
-H 'content-type: application/json' \
-d "{\"prompt\": $(jq -Rs . < handbook.txt)}" | jq -r .id)
# Fork it per request — a page-table copy, not a memory copy
curl -s -X POST "localhost:11435/v1/contexts/$ID/fork?n=8"
# Or write it to a file and restore it on another machine
curl -s "localhost:11435/v1/contexts/$ID/snapshot" > handbook.ctxThat matters for one shape in particular: a large shared prefix and many short completions against it. A corpus, a codebase, a manual, a long instruction. Every request otherwise re-reads the whole prefix, and on a 4,000-token instruction that re-read is the workload.
So which
Ollama if you want to stop reading now and have something working.
llama.cpp if you are optimising a specific model on specific hardware and want every knob.
llamay if you are building a product on top: many requests sharing a prompt, structured output you need to be structured, or contexts that have to survive a process.
They are not exclusive. llamay reads Ollama’s store, so you can run both and point one client at each — different ports, same weights, same machine.
What is the same
All three run GGUF, all three take the same quantisations, and all three keep your data on your hardware. That last one is the reason most people are reading this page, and none of the three differ on it.