How to run an LLM locally in 2026, and which one fits your machine
Five minutes from nothing to a model answering on your own hardware, plus the sizing table that decides whether you want a 3B, a 7B or a 70B.
There are three things people mean by “run an LLM locally”, and only one of them takes five minutes.
You do not want to train anything. You do not want to convert weights. You want a model answering questions on your machine, reachable by code you already wrote, with nothing leaving the building.
In short three commands, a 7B model, and anything that speaks the OpenAI API points at localhost. The sizing table below decides which model.
Five minutes
curl -fsSL https://llamay.com/install.sh | sh
llamay pull hf:Qwen/Qwen2.5-7B-Instruct-GGUF/qwen2.5-7b-instruct-q4_k_m.gguf
llamay serveThat is a static binary with no runtime, no Python environment, no CUDA toolkit and no account. The third line starts a server on 127.0.0.1:11435 that speaks the OpenAI API, so anything you have already built points at it by changing one string:
from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:11435/v1", api_key="not-used")
client.chat.completions.create(model="", messages=[{"role": "user", "content": "hello"}])Tip already using Ollama? Your models come with you. llamay reads its store, so llamay run -m llama3.2:3b works with nothing re-downloaded and both can run at once — different ports, same weights.
Which model fits
The weights are the number people plan for. The key-value cache is the one that surprises them: it grows with context length times how many requests are in flight at once.
| Memory | Comfortable | What you get |
|---|---|---|
| 8 GB | 3B at q4_k_m | Classification, extraction, rewriting. Genuinely useful. |
| 16 GB | 7B–8B at q4_k_m | The common choice. Handles most production shapes. |
| 32 GB | 14B, or 7B at q8_0 | Better reasoning; a bigger model at lower precision usually wins |
| 64 GB+ | 32B–70B | Now watch the cache, not the weights |
A 7B is not a frontier model and does not need to be. On extraction, classification, routing, summarising and rewriting — most of what production actually runs — the gap is far smaller than the benchmark charts suggest. On open-ended reasoning it is large and obvious.
Warning the weights are not the whole memory bill. The key-value cache grows with context length times how many requests are in flight, and it is what actually runs a machine out of room. -kv vq8 roughly halves it.
What quantisation to take
q4_k_m unless you have a reason. It is roughly half the memory of q8_0 for a difference most tasks cannot measure, and memory bandwidth is what decides how fast tokens come out.
The one worth knowing beyond that is the cache precision, which is a separate setting from the weights:
llamay serve -m model.gguf -kv vq8vq8 quantises the cache’s values and keeps its keys exact, roughly halving cache memory. llamay’s verifier measures what that costs rather than asserting it is free — it reports how much of the sampler’s probability mass moved.
Getting the GPU
On an Apple Silicon Mac, take the Metal build; the plain one is CPU-only. On Linux with an NVIDIA or AMD card, take the _gpu tarball — it carries CUDA and Vulkan in one binary and loads the driver at first use, so there is no toolkit to install.
Check which you got:
curl -s localhost:11435/healthz{"backend":"device","model":"Qwen2.5 7B Instruct","status":"ok"}"device" means the GPU is doing the work. "cpu" means the driver was not found or declined, and the first log line says which.
What changes about prompting
Small models need shorter instructions, not longer ones. Below about 30B the failure is not that a rule is ignored — it is that the model follows the last rule it read and loses the question. Three rules it will follow beat twelve it will not.
And stop asking for JSON in prose. A schema is enforced while sampling, so the reply parses or the request fails:
{"response_format": {"type": "json_schema", "json_schema": {"name": "r",
"schema": {"type": "object",
"properties": {"label": {"type": "string", "enum": ["billing", "bug", "other"]}},
"required": ["label"]}}}}The sampler never sees a token outside that enum. There is no failure mode to handle.
When local is the wrong answer
Be honest about it. You now own uptime, memory and model updates. If you make a hundred calls a month and none of them touch anything sensitive, a hosted API is less work and probably cheaper.
Local wins on three shapes: high volume with low value per item, where per-token pricing is backwards; anything where the sensitive step cannot be outsourced, because the component doing redaction must see what it redacts; and anywhere the network is not guaranteed.