Getting started

Install llamay

Every platform, every package, and what to do after it lands. Pick your row, follow it to the end, and you will have a model answering on your own machine with nothing configured and no account.

Which one do you want?

macOS

The app (easiest)

1. Download llamay_<version>_macos.dmg from the download page.
2. Open it and drag llamay to Applications.
3. Launch it. It is signed and notarized, so it opens without a warning.

The app bundles the engine and manages the model store under ~/.llamay. It also puts the llamay command on your PATH the first time you open a terminal from it, so the CLI and the app share one set of downloaded models rather than each keeping its own.

The command line

curl -fsSL https://llamay.com/install.sh | sh

The script detects your architecture, downloads the matching build and the release's SHA256SUMS, and checks the sum before installing — that check is not optional, and the script refuses rather than skipping it on a machine with no sha256 tool. The binary goes to /usr/local/bin, which it will ask for sudo to write.

To avoid the password prompt entirely, install somewhere you already own:

LLAMAY_INSTALL_DIR="$HOME/.local/bin" curl -fsSL https://llamay.com/install.sh | sh

Apple Silicon and the GPU

On an M-series Mac, take the Metal build. It is a separate artifact because it needs cgo and the plain one does not:

Check which one you are running at any time — the backend field says device when the GPU is in use and cpu when it is not:

curl -s localhost:11435/healthz

{"backend":"device","model":"Qwen2.5 7B Instruct","status":"ok","version":"0.1.54"}

From the tarball, by hand

VER=0.1.54
curl -fsSLO https://github.com/AzmxAI/llamay-releases/releases/download/v$VER/llamay_${VER}_darwin_arm64_metal.tar.gz
curl -fsSLO https://github.com/AzmxAI/llamay-releases/releases/download/v$VER/SHA256SUMS

# Verify before you extract, not after.
shasum -a 256 -c SHA256SUMS --ignore-missing

tar xzf llamay_${VER}_darwin_arm64_metal.tar.gz
sudo install -m 0755 llamay /usr/local/bin/llamay
llamay --version

xattr -d com.apple.quarantine llamay

Windows

The installer

1. Download llamay_<version>_windows_amd64_setup.exe — or the arm64 one on a Snapdragon or Surface Pro X.
2. Run it. It is signed with Azure Trusted Signing, so SmartScreen lets it through.
3. It installs to %LOCALAPPDATA%\Programs\llamay, adds that to your PATH, and offers to start llamay at logon.

Per-user by default, which is why there is no elevation prompt. For a machine-wide install into Program Files, run it elevated.

PowerShell, one line

irm https://llamay.com/install.ps1 | iex

Same shape as the shell script: detect, download, verify the checksum, install, add to PATH. Add -AllUsers to install into Program Files instead, which needs an elevated shell.

& ([scriptblock]::Create((irm https://llamay.com/install.ps1))) -AllUsers

Open a new terminal after installing. A PATH change does not reach a shell that was already running.

The portable zip

No installer, no PATH change, nothing written outside the folder — useful on a locked-down machine or a USB stick.

$ver = "0.1.54"
Invoke-WebRequest "https://github.com/AzmxAI/llamay-releases/releases/download/v$ver/llamay_${ver}_windows_amd64.zip" -OutFile llamay.zip
Expand-Archive llamay.zip -DestinationPath .\llamay
.\llamay\llamay.exe --version

Running it at logon

A scheduled task rather than a Windows service, and deliberately: llamay wants a user's model store and a desktop session, which a service running as SYSTEM has neither of.

schtasks /Query /TN llamay          # the installer creates this
schtasks /End   /TN llamay          # stop it now
schtasks /Delete /TN llamay /F      # stop it starting at logon

Linux

Debian, Ubuntu, Mint, Pop!_OS

VER=0.1.54
curl -fsSLO https://github.com/AzmxAI/llamay-releases/releases/download/v$VER/llamay_${VER}_amd64.deb
sudo apt install ./llamay_${VER}_amd64.deb

Use arm64.deb on a Raspberry Pi 5, an Ampere box or Graviton.

Fedora, RHEL, Rocky, Alma, openSUSE

VER=0.1.54
sudo dnf install https://github.com/AzmxAI/llamay-releases/releases/download/v$VER/llamay-${VER}-1.x86_64.rpm

Use aarch64.rpm on ARM. On openSUSE, zypper install takes the same URL.

The service the packages install

Both packages ship a systemd unit that runs as its own llamay user with a store at /var/lib/llamay/models. It is not started automatically — a service that begins downloading models the moment a package lands is not a good guest.

sudo systemctl enable --now llamay
systemctl status llamay
journalctl -u llamay -f

It listens on loopback. To change the address, the model or anything else, override rather than editing the unit — a package upgrade replaces the unit and would take your edit with it:

sudo systemctl edit llamay

[Service]
ExecStart=
ExecStart=/usr/bin/llamay serve -m /var/lib/llamay/models/qwen.gguf -conc 4

The empty ExecStart= is required. Without it systemd appends to the existing one and the service tries to start twice.

Any distribution, from the tarball

curl -fsSL https://llamay.com/install.sh | sh

Works on anything with a shell and curl: Arch, Alpine, NixOS, a container, a CI runner. No package manager involved, and no root needed if you set LLAMAY_INSTALL_DIR to somewhere you own.

NVIDIA and AMD

Take linux_amd64_gpu.tar.gz. It carries both the CUDA and Vulkan backends in one binary, and the loader is dlopened at first use — so there is no CUDA toolkit to install, no libvulkan-dev, and the same file runs on a machine with no GPU at all.

llamay serve -m model.gguf
# then, in another terminal
curl -s localhost:11435/healthz | grep -o '"backend":"[a-z]*"'

"device" means a GPU is doing the work. "cpu" means the driver was not found or declined, and the server says which in its first log line rather than quietly running slowly.

Docker

The image is built FROM scratch — the binary and a CA bundle, nothing else. There is no shell in it, which is a feature and also means docker exec will not give you one.

The image listens on 8081, not 11435. Its default command is serve -addr 0.0.0.0:8081, spelled out rather than left to the default because serve binds to loopback and loopback inside a container is reachable by nothing.

docker run --rm -p 8081:8081 \
  -v llamay-models:/models \
  ghcr.io/azmxai/llamay:latest

The volume is where models live: LLAMAY_MODELS=/models is set in the image because the container runs as uid 65532 with no home directory to fall back to. Pull into it once and the models survive the container.

docker run --rm -v llamay-models:/models \
  ghcr.io/azmxai/llamay:latest \
  pull hf:Qwen/Qwen2.5-7B-Instruct-GGUF/qwen2.5-7b-instruct-q4_k_m.gguf

Publishing that port puts the server on your network, so set a key before you do:

docker run --rm -p 8081:8081 \
  -e LLAMAY_API_KEY=$(openssl rand -hex 32) \
  -v llamay-models:/models \
  ghcr.io/azmxai/llamay:latest

For NVIDIA, use Dockerfile.cuda and pass --gpus all.

Verifying what you downloaded

Every release publishes SHA256SUMS over the artifacts as they were uploaded. The installers check it for you; if you download by hand, check it yourself.

--ignore-missing matters: the file covers every artifact in the release and you downloaded one of them. Without it the check reports fourteen missing files and one OK, which reads like a failure.

Your first five minutes

Identical on every platform once llamay is on your PATH.

# 1. Get a model. A 7B is a good first choice; a 3B is fine on 8 GB.
llamay pull hf:Qwen/Qwen2.5-7B-Instruct-GGUF/qwen2.5-7b-instruct-q4_k_m.gguf

# 2. Ask it something, with no server involved.
llamay run -m qwen2.5-7b-instruct-q4_k_m.gguf -p "why is the sky blue?"

# 3. Start the server.
llamay serve

# 4. Use it from anything that speaks OpenAI.
curl -s localhost:11435/v1/chat/completions \
  -H 'content-type: application/json' \
  -d '{"model":"","messages":[{"role":"user","content":"hello"}]}'

Already have Ollama? llamay reads its store, so every model you have pulled is available with no re-download: llamay run -m llama3.2:3b.

Which model fits your machine

The weights are the number people plan for and the KV cache is the one that surprises them: it grows with context length times concurrency. -kv vq8 roughly halves it, at a cost the verifier measures rather than asserts.

Upgrading

Your models are not touched by an upgrade — they live in the store, not beside the binary. Restart the server afterwards; a running process keeps the old binary mapped.

Uninstalling

# macOS or Linux, installed by the script or a tarball
sudo rm /usr/local/bin/llamay

# Debian / Ubuntu
sudo apt remove llamay

# Fedora / RHEL
sudo dnf remove llamay

# the models, which no uninstall removes for you
rm -rf ~/.llamay          # or /var/lib/llamay for a packaged service

On Windows, use Add or Remove Programs, or delete the folder if you took the zip. The model store is left alone on purpose: it is often tens of gigabytes somebody spent an evening downloading, and an uninstaller that deletes it without asking is one people only forgive once.

When it does not work

Where to go next

Quickstart is the same first session in fewer words. The live playground runs against the server you just started. Recipes is what to build with it, and Serving is what to do when it is not only your machine calling.