The AZMX inference engine
Local-first AI, with a memory
Every other engine throws a conversation away when the process ends, and recomputes it from the first word when you come back. llamay keeps the thinking itself — so a conversation resumes on another machine without re-reading a line of it.
macOS, Linux and Windows. No account, no card, and nothing leaves the machine until you ask it to.
One context, forked twice. Both branches begin from thinking that already happened — on this machine, or on the next one you sign in from.
Three commands
Install, pull a model, talk to it. llamay has its own store and its own downloader, so nothing else has to be installed first — and it reads a model Ollama already pulled, as a courtesy rather than a dependency.
serve with no -m takes the first model in your
store and listens on 127.0.0.1:11435 — beside Ollama's
11434, and deliberately away from 8081.
$ llamay pull qwen2.5:0.5b pulling manifest pulling c5396e06af29 [========================] 100.0% 379.4 MiB / 379.4 MiB (26.7 MiB/s) verifying sha256 digest stored qwen2.5:0.5b (379.4 MiB) $ llamay run -m qwen2.5:0.5b -p "The capital of France is" Paris. It has been the capital since 987, when Hugh Capet… $ llamay serve llamay v0.4.0 metal+accelerate 17 architectures listening on http://127.0.0.1:11435
What you get
One 8 MB binary with no dependencies, no Python and no runtime to install. It reads the same model files everyone else does and uses whatever the machine has — CPU, Metal, CUDA or Vulkan.
Models far larger than your laptop can hold, through the same address and the same API. Sign in and conversations follow you between machines, with the thinking already done.
OpenAI, Anthropic and Ollama, all three on one port. Point an existing client at llamay by changing a base URL — no SDK swap, no rewrite, nothing to port.
For the people who read the source
llamay publishes its benchmarks including the columns where it loses, the invariants its kernels are held to, and the architecture of the state engine that makes a context an object rather than a buffer. None of it is behind a form.
Before you install
A list of what an engine cannot do is more useful than a list of what it can, and it is the first thing anyone finds out anyway.
On an M4, 68% of llama.cpp's decode rate and 64% of Ollama's. llama-server remains the right answer when raw single-stream throughput is all that matters. Shipping both is a feature.
gpt-oss and phi3-mini exceed the 5% total-variation bound under vq8. Run them with -kv f32; llamay verify says so per file.
State-space models are declined at load rather than approximated. DeepSeek-V2 and V3 are refused by name: latent attention changes what a KV page holds, which is the allocator, the fork and the snapshot format, not the attention alone.
Windows and Linux get a launcher entry that starts the server and opens the same interface in a browser, which is what a build with no cgo can honestly offer. The signed, notarised chat window is macOS.
IQ2, IQ3 and IQ4 load and run on the portable Go kernels only. llamay quantize will not produce one: picking grid points well is a search against an importance matrix, and a bad i-quantiser is worse than none.
Not a limitation, but it belongs in the same list. The server listens on loopback and no provider is configured until you add one. There is no llamay registry and there will not be one.
Free forever, on your own machine
The engine costs nothing and always will — every model it runs, it runs locally. The plans buy the two things a single machine cannot give you: a conversation that resumes on the next one, and models far larger than yours can hold.
Loading plans…
Billed through Polar, which is the merchant of record — llamay never sees a card. Cancel in a click. Full comparison →
Free, open source, and everything runs on your machine.