Start here

Deploying llamay for a team

On one laptop llamay needs no configuration: it listens on loopback and only that machine can reach it. Put it on a network and four things have to be decided — who may call it, how the wire is protected, what is recorded, and how you watch it. This page walks through all four, then turns them into a service that survives a reboot.

1 · Choose the address

-addr decides who can connect at all. The default, port 11435 on the loopback address, is this machine only. -addr :11435 — a port with no host — is every interface. Everything below matters from the moment you widen it.

2 · Name every caller

Three ways, and they can be used together on one server. When a request carries several, a key wins, then a directory token, then the certificate.

Named keys

A file of id:key lines, one per caller. The id is what the audit record says; the key is what the caller sends as Authorization: Bearer or X-Api-Key.

umask 077
printf 'claims-app:%s\nanalyst:%s\n' "$(openssl rand -hex 32)" "$(openssl rand -hex 32)" > /etc/llamay/keys
llamay serve -addr :11435 -api-keys /etc/llamay/keys

• A key must be at least 16 characters. A shorter one stops the server at startup: the key for "analyst" is 15 characters; use at least 16, and random.
• Lines starting with # are comments. Ids may not repeat, two ids may not share a key, and - and anonymous are reserved.
• Keys are held as SHA-256 digests and compared in constant time.
• A single -api-key (or LLAMAY_API_KEY) still works; its caller is recorded as anonymous.

Directory sign-in (OIDC)

If people already sign in through Entra ID, Okta, Keycloak, ADFS, Ping or Clerk, llamay can accept the token that directory issues. Nobody is given a llamay key, and leaving the directory is leaving llamay.

llamay serve -addr :11435 \
  -oidc-issuer https://login.microsoftonline.com/<tenant-id>/v2.0 \
  -oidc-audience api://llamay

The round trip below was run against a test issuer whose keys were passed as a file with -oidc-jwks:

sign-in: tokens from https://login.example.com/tenant/v2.0, callers named by preferred_username

$ curl -s localhost:11435/v1/models -H "Authorization: Bearer $TOKEN"
{"data":[{"created":1790482111,"id":"all-minilm:latest","object":"model","owned_by":"azmx"}, ...

$ curl -s localhost:11435/v1/models -H "Authorization: Bearer $TOKEN_FOR_ANOTHER_APP"
{"error":{"message":"this server did not accept your sign-in token: the token is not for audience \"api://llamay\"","type":"Unauthorized"}}

The first request is in the audit record as "actor":"[email protected]". The flags for each directory, and what is checked, are on the security page.

Client certificates and smart cards

Where people carry a CAC or PIV card, or services hold certificates from an internal authority, the certificate can be the identity:

llamay serve -addr :11435 \
  -tls-cert /etc/llamay/server.pem -tls-key /etc/llamay/server.key \
  -tls-client-ca /etc/llamay/client-ca.pem -tls-client-id cn

-tls-client-ca makes the TLS handshake demand a certificate from that authority. -tls-client-id cn names the caller by the certificate's common name (email uses its email address). A card whose common name is EXAMPLE.ANALYST.A.1234567890 is recorded as exactly that:

$ curl -s --cacert ca.pem --cert client.pem --key client.key https://localhost:11435/v1/chat/completions -d '...'
{"choices":[{"finish_reason":"stop","index":0,"message":{"content":"OK.","role":"assistant"}}], ...

$ tail -1 audit.jsonl
{"seq":1,"time":"2026-09-27T04:08:06.367425Z","actor":"EXAMPLE.ANALYST.A.1234567890","remote":"localhost:64228","method":"POST","path":"/v1/chat/completions","status":200, ...

3 · Protect the wire

-tls-cert and -tls-key make the engine serve HTTPS itself: TLS 1.2 at least, 1.3 when the client offers it. There is no proxy to run beside it. The certificate is read at startup, so rotating it is a restart. Clients then need the authority that issued it:

curl --cacert /etc/llamay/ca.pem https://llm.example.com:11435/healthz

4 · Let the profile check your work

-profile enterprise refuses to start until the command line has all three: named callers, TLS on a reachable address, and an audit file. It lists everything missing at once:

$ llamay serve -profile enterprise -addr :11435
llamay serve: -profile enterprise will not start without:
  - -api-keys <file> (one `id:key` per line), -tls-client-id or -oidc-issuer, because :11435 is reachable from the network and every request must name its caller
  - -tls-cert and -tls-key, because :11435 is reachable from the network and requests must not cross it in plain text
  - -audit <file>, because every request must be recorded
(or bind the loopback address, where only this machine can reach the server and the first two do not apply)

It also turns on -audit-strict and -audit-sync unless you set them yourself, and it refuses context snapshots that name their weights by layout alone (see the same weights). A profile never overrides a flag you typed.

5 · Record every request

llamay serve ... -audit /var/lib/llamay/audit/audit.jsonl -audit-syslog

One JSON line per request: who, from where, which route, the status, the time taken, the model and the token counts. Never the prompt or the answer. Each line carries the SHA-256 of the one before it:

{"seq":1,"time":"2026-09-27T04:02:41.857946Z","actor":"analyst","remote":"localhost:64124","method":"POST","path":"/v1/retrieve","status":200,"ms":27,"model":"all-MiniLM-L6-v2","tokens_in":8,"prev":"0000…0000","hash":"8ac2ad07…"}

$ llamay audit verify audit.jsonl
chain intact: 3 record(s) in audit.jsonl
every record carries the hash of the one before it, and every hash matches its contents

$ llamay audit verify audit-edited.jsonl      # one status changed by hand
chain broken after 2 record(s)
llamay audit: audit: record 3 was altered: it carries 28a470ed5dea, its contents hash to a3150a257c32

• -audit-syslog also hands every record to the local syslog daemon, facility authpriv, tag llamay-audit, for your SIEM. Windows has no syslog daemon: point the SIEM agent at the file instead.
• -audit-strict stops serving if a record cannot be written: every route but /healthz then answers 503. -audit-sync flushes each record to disk.
• -audit-continue old.jsonl starts a new file that carries on the old one's chain, for rotation. llamay audit verify old.jsonl new.jsonl checks both as one chain, oldest first.
• llamay audit show prints records as a table; llamay audit who summarises them by caller.

Estates that must keep the conversations themselves add encrypted capture; see content capture.

6 · Watch it

# prometheus.yml
scrape_configs:
  - job_name: llamay
    scheme: https
    authorization: {credentials_file: /etc/prometheus/llamay.key}
    tls_config: {ca_file: /etc/prometheus/llamay-ca.pem}
    static_configs: [{targets: ["llm.example.com:11435"]}]

/metrics needs a key like every route but /healthz, so give Prometheus a key of its own. Add -otlp http://collector:4318 for traces. The metric names, and what to alert on, are on the observability page.

7 · Run it as a service

The .deb and .rpm packages install a systemd unit that runs llamay serve as its own llamay user, on loopback, with the store in /var/lib/llamay/models. Override its command rather than editing it, because an upgrade replaces the unit:

sudo install -d -o llamay -g llamay -m 0750 /var/lib/llamay/audit
sudo systemctl edit llamay

[Service]
ExecStart=
ExecStart=/usr/bin/llamay serve -profile enterprise -addr :11435 \
  -m qwen2.5:7b -api-keys /etc/llamay/keys \
  -tls-cert /etc/llamay/server.pem -tls-key /etc/llamay/server.key \
  -audit /var/lib/llamay/audit/audit.jsonl -audit-syslog -pull=false

The unit makes the whole filesystem read-only except /var/lib/llamay, which is why the audit file lives there. To write it elsewhere, add ReadWritePaths=/var/log/llamay to the same override. Files in /etc/llamay must be readable by the llamay group. -pull=false stops callers from making the server download models; pull them yourself with llamay pull.

sudo systemctl restart llamay
journalctl -u llamay -f

A checklist

1. -addr is the narrowest address that works.
2. Every caller is named: -api-keys, -oidc-issuer or -tls-client-id.
3. -tls-cert and -tls-key are set, and clients check the certificate.
4. -audit is set, on a disk with room, and something verifies it.
5. -profile enterprise is on, so a restart without one of the above fails loudly.
6. -pull=false, unless callers should be able to download models.
7. -cors is unset, unless a web page you control must call the server.
8. Prometheus scrapes /metrics with a key of its own.

Where to go next