Glossary · Last reviewed
The words you meet when you run a language model on your own machine, each defined in a sentence or two, with a link to the primary source. Start with GGUF (the file), quantisation (why it fits), the KV cache (why memory runs out) and prefix caching (why a repeated prompt is fast).
The file format almost every local model ships in. One GGUF file holds a model's tensors and the metadata needed to run it — architecture, tokenizer, context length — so it can be loaded without anything else. It is specified in the ggml project (the GGUF spec), and Hugging Face describes it as a binary format optimised for quick loading and saving (Hub docs). llama.cpp, Ollama, LM Studio and llamay all read it; llamay info -m file.gguf shows what one contains.
Storing a model's weights in fewer bits than they were trained in — 8, 5 or 4 bits instead of 16 — so the model takes less memory and reads faster, at some cost in accuracy (Hugging Face). Smaller files fit on smaller machines; the loss grows as the bit count falls.
Names of GGUF quantisation recipes, as produced by llama.cpp's llama-quantize (README). Q8_0 stores weights at 8 bits. Q4_K_M is built mostly from the Q4_K format, about 4.5 bits per weight (Hub docs), with some tensors kept at higher precision. A file's name is a claim: llamay info prints the formats its tensors actually use (models).
The keys and values a model computes for every token it has read, kept so it does not recompute them for each new token (Transformers docs). It grows with context length and is often what runs a machine out of memory at long context. llamay can store it at lower precision with -kv q8 or -kv vq8 (contexts).
The two phases of a model call. Prefill reads the whole prompt in one pass and is bound by arithmetic; decode writes the answer one token at a time and is bound by memory bandwidth. Time to first token is mostly prefill; tokens per second is decode (throughput).
Reusing the KV cache computed for the start of a prompt when a later prompt begins the same way, so the shared part skips prefill. The model still generates every answer; nothing stored is returned. vLLM calls it automatic prefix caching (vLLM docs); llamay keeps a radix tree of prefixes and reports what was reused in usage.prompt_tokens_details.cached_tokens (guide).
The most tokens a model can attend to at once — prompt and answer together. A server can run a model with less than its full window to save memory; Ollama, for one, picks a default by GPU memory (Ollama docs), while llamay uses the length in the file.
A server that answers the same HTTP routes and JSON shapes as OpenAI's API — chiefly /v1/chat/completions (OpenAI reference) — so OpenAI's SDKs and the tools built on them work by changing the base URL. Ollama, LM Studio, llama-server, vLLM and llamay all offer one; llamay's is at http://localhost:11435/v1 (integrating).
A server that answers Anthropic's Messages API, /v1/messages (Anthropic reference), so the Anthropic SDK and tools built on it can talk to a local model. Coverage differs: Ollama documents a subset (Ollama docs).
llamay's name for treating a computed context as an object. A fork is a copy-on-write page-table copy, so many requests can continue one long prompt without copying it; a snapshot writes it to a file that can be restored later or on another machine (state engine).
Restricting which tokens a model may produce, at each step, to those a grammar allows — so output such as JSON cannot come back malformed. GBNF is the grammar notation llama.cpp defined (grammars README); llamay accepts it and also enforces JSON Schema while sampling (structured output).
The two speed numbers that matter locally. Time to first token is how long before the answer starts — mostly prefill. Tokens per second is how fast it is written — decode. One number for both hides which is slow; llamay bench reports them apart (throughput).
GGUF is the file format most local language models ship in. One file holds the model's tensors and the metadata needed to run it, such as its architecture, tokenizer and context length. It is specified by the ggml project and read by llama.cpp, Ollama, LM Studio and llamay.
It is a GGUF quantisation recipe from llama.cpp's llama-quantize: most weights in the 4-bit Q4_K format, about 4.5 bits per weight, with some tensors kept at higher precision. Q8_0 stores weights at 8 bits.
The keys and values a model has computed for every token it has read, kept so they are not recomputed for each new token. It grows with context length and is often what limits memory at long context.
Reusing the KV cache of a prompt's beginning when a later prompt starts the same way, so that part skips prefill. The model still generates a fresh answer every time.
A server that answers the same routes and JSON as OpenAI's API, such as /v1/chat/completions, so OpenAI's SDKs and tools work by changing the base URL. llamay's is at http://localhost:11435/v1.
Every statement above about another project is taken from that project's own documentation, repository or licence, read on 31 October 2026. Anything we could not confirm there is left out. If one of these has changed, write to [email protected] and we will correct the page.
Last reviewed .