Comparison · Last reviewed

llama.cpp vs llamay

llama.cpp is the MIT-licensed reference engine for GGUF, with the widest hardware support and, in our own measurements, faster generation than llamay. llamay is a separate engine written from scratch in Go that reads the same files — it does not use llama.cpp's code. Choose llama.cpp for speed, every backend and every new model first. Choose llamay for one binary that serves OpenAI, Anthropic and Ollama APIs, reads Ollama's store, and treats a context as an object you fork, snapshot and move.

Side by side

llama.cpp (llama-server)llamay
LicenceMIT, open source — LICENSEProprietary, free to download and run — pricing
BackendsCPU, Metal, CUDA, HIP, Vulkan, SYCL, OpenCL, WebGPU, MUSA, CANN and more — READMECPU (hand-written NEON and AVX), Metal, CUDA, Vulkan — backends
InstallInstall scripts, Docker, prebuilt binaries, or build from source — READMEOne static binary from an install script, package or app — install
Server APIsOpenAI-compatible chat, completions, responses and embeddings, and an Anthropic-compatible /v1/messages — llama-server READMEOpenAI, Anthropic and Ollama APIs — HTTP API
Default port9931 in the current README — llama-server README11435
Web UIBuilt in, on by default — llama-server READMEStudio at /studio — the apps
Constrained outputGBNF grammars and JSON Schema — grammarsGBNF grammars, and JSON Schema enforced while sampling — structured output
Several modelsRouter mode with --models-dir — llama-server READMEServes every model in its store and Ollama's by name — serving
Reusing prompt statePrompt caching on by default; --slot-save-path saves and restores a slot's KV cache to a file — llama-server READMENamed contexts forked copy-on-write, snapshots that restore on another machine and check which weights made them — state engine
SpeedFaster. In llamay's own interleaved runs on the same file and machine, llama.cpp decodes faster on Metal and is faster at both prefill and decode on CUDA and VulkanMeasure it yourself with llamay bench — throughput

Which to choose

Choose llama.cpp if

  • tokens per second is what matters most — it is faster than llamay today;
  • you need a backend llamay does not have: AMD HIP, Intel SYCL, OpenCL, WebGPU and others, or a GPU on Windows;
  • you want a new model architecture or quantisation the week it appears — llamay reads a fixed list and refuses the rest (models);
  • you want an open-source engine you can build and audit yourself.

Choose llamay if

  • you want one server that answers OpenAI, Anthropic and Ollama clients without flags or a proxy;
  • you want Ollama's downloaded models served in place, alongside files and Hugging Face pulls;
  • a context has to outlive the process: forked many ways without copying, written to a file, restored on another machine (contexts);
  • you want the same input to produce the same output regardless of thread count or caching — checked by llamay verify (validation).

Moving from llama-server

llama-serverllamay
llama-server -m model.ggufllamay serve -m model.gguf
-c 8192Not needed: the context length is read from the file
-ngl 99Not needed: the GPU is used when the build has one
-np 4-concurrency 4, and -batch for continuous batching
--grammar-file x.gbnf-grammar-file x.gbnf
--lora x.gguf-lora name=x.gguf
-ctk q8_0 -ctv q8_0-kv q8, or -kv vq8 for exact keys

The full list is on Migrating.

Questions

Is llamay built on llama.cpp?

No. llamay is its own engine, written from scratch in Go. It reads the same GGUF file format, and that is the whole relationship: llama.cpp releases and fixes do not reach llamay.

Is llama.cpp faster than llamay?

Yes, in llamay's own measurements on the same file and machine: llama.cpp decodes faster on Metal and is faster at prefill and decode on CUDA and Vulkan. Measure your own case with llamay bench.

Can llama.cpp save a KV cache to disk?

Yes. llama-server's --slot-save-path lets a slot's KV cache be saved to a file and restored. llamay's contexts add copy-on-write forks and snapshots that restore on another machine and check which weights produced them.

Does llama-server support the Anthropic Messages API?

Yes. The llama-server README lists an Anthropic-compatible /v1/messages endpoint; tool use there needs the --jinja flag.

Is llamay open source like llama.cpp?

No. llama.cpp is MIT-licensed. llamay is proprietary software from Kosa8 Labs, free to download and run.

Sources

Every statement above about another project is taken from that project's own documentation, repository or licence, read on 31 October 2026. Anything we could not confirm there is left out. If one of these has changed, write to [email protected] and we will correct the page.

  1. llama.cpp — LICENSE (MIT)
  2. llama.cpp — README (backends, install)
  3. llama.cpp — llama-server README
  4. llama.cpp — GBNF grammars README
  5. ggml — the GGUF specification

Last reviewed .