Guides
Prompting local models
Most prompting advice is written for frontier models. A 3B running on your laptop needs different handling, and the differences are specific enough to be worth writing down: shorter instructions, fewer of them, examples over explanations, and constraints enforced by the sampler rather than requested in prose.
What actually changes below about 30B
1. Spend your instruction budget
Every sentence in a system prompt competes with every other for attention. Frontier models have enough to go round. Small models do not, and the failure is not that they ignore a rule — it is that they follow the last one they read and lose the question.
Instead of this
You are a helpful assistant. Be concise
but thorough. Use markdown where
appropriate. Do not be verbose. Consider
edge cases. Be professional but friendly.
If unsure, say so. Avoid hedging.
Prefer active voice. Do not repeat the
question back. Use examples when useful…This
Answer in at most three sentences.
Lead with the answer.
Say plainly when you do not know.Three rules a 3B will follow beat twelve it will not. If you need more than about six, you need a bigger model or a chain of smaller calls.
2. Say what to do, not what to avoid
A prohibition puts the thing it forbids into the model's context, which is not where you want it. This is Anthropic's published guidance and it holds harder the smaller the model gets.
| Weaker | Stronger |
|---|---|
| Do not use markdown. | Write flowing prose paragraphs. |
| Do not be verbose. | Answer in two sentences. |
| Never apologise. | Open with the answer itself. |
| Do not make things up. | Answer only from the passages given. If they do not say, say so. |
3. Show one example, not three paragraphs of rules
A small model imitates far better than it reasons. One worked example will fix a format that a paragraph describing the format will not.
SYSTEM = (
"Extract the fields. Answer exactly like the example.\n\n"
"Input: Invoice 88 from Bee Co, 40 GBP\n"
"Output: {\"number\": \"88\", \"supplier\": \"Bee Co\", \"total\": 40, \"currency\": \"GBP\"}"
)Two or three examples are better than one when the cases differ in kind — show the edge case you care about, not three of the same shape. Past about five you are paying prefill on every request for diminishing returns, and a context you fork is how to stop paying it twice.
4. Do not ask for a format. Enforce it.
This is the single biggest difference between prompting a hosted model and prompting a local one, and it is llamay's advantage rather than a workaround. A schema is applied to the token distribution while sampling, so the reply cannot be malformed.
Hoping
{"messages": [
{"role": "system",
"content": "Reply with only JSON. No prose. No code fence."}
]}Works most of the time. The failures arrive in production, wrapped in ```json.
Enforcing
{"response_format": {
"type": "json_schema",
"json_schema": {"name": "r", "schema": {
"type": "object",
"properties": {"label": {"type": "string",
"enum": ["billing", "bug", "other"]}},
"required": ["label"]}}}}The sampler never sees a token outside the enum. There is no failure mode to handle.
Once the shape is guaranteed, delete the prose asking for it. Those sentences are now competing for attention against instructions that still matter.
5. Put stable material in the system turn
A model that has a system role weights it differently from a user's words — that is the entire reason the role exists. Rules, glossaries and examples go there; the thing that changes per request goes in the user turn.
It also makes the prefix cache work. Identical system turns across requests are computed once, and the second request pays only for its own tokens. Put a timestamp at the top of your system prompt and you have disabled that cache for every request you will ever make.
system: [4,000 tokens: rules, glossary, examples] ← computed once, reused
user: [40 tokens: this ticket] ← all you pay after the first
Not every model has a system role. llama-guard answers
enum system not in user,assistant and refuses outright, which is
why llamay retries once by folding the instruction into the first user turn
rather than keeping a list of which models allow what.
6. Let it think, then throw the thinking away
Small models benefit from working through a problem out loud, but you rarely want the reasoning in the answer. Ask for both and take the part you want.
Work through it step by step under "Reasoning:".
Then give your final answer under "Answer:" in one line.text = out["choices"][0]["message"]["content"]
answer = text.split("Answer:")[-1].strip()On a model with a reasoning mode of its own, do not do this — ask it to think and read its reasoning field instead. Doing both produces a model reasoning about how to present its reasoning.
7. Chain small calls instead of writing one big prompt
Two focused calls to a 3B routinely beat one elaborate call, and each is cheap. The failure mode of a big prompt on a small model is silent: it answers the part it still remembers.
8. Temperature, and when zero is wrong
| Task | Temperature | Why |
|---|---|---|
| Extraction, classification | 0 | There is a right answer; variety is noise. |
| Summarising | 0–0.3 | Faithfulness beats phrasing. |
| Drafting, rewriting | 0.7 | The second-best word is often better. |
| Brainstorming | 0.9–1.0 | You want the tail. |
Temperature 0 is not determinism. Batching and kernel order still move a token
here and there, so two identical requests can differ. It removes the variance
that would swamp a change you are measuring; it does not remove all of it. Set
seed as well if you need repeatability, and even then treat an
exact-match test on generated prose as flaky by nature.
9. Measure the prompt, do not admire it
A prompt is the one artefact where a change that reads like an improvement can make things worse and leave no trace: nothing fails to compile, no test goes red, and the regression shows up as answers that are slightly flatter for weeks.
llamay's own system prompt has a changelog that requires a score before and after, and the harness that produces it found two things worth repeating here. A prompt edit that removed a genuine internal contradiction moved the score not at all. And a second edit looked like it fixed a case, on one sample — it had not: that case produced a table two to three times in eight at temperature 0 with the prompt and question held constant, and an eight-sample A/B put the original wording ahead.
So: keep a dozen cases with known-good answers, assert what a machine can check without judgement, and run more than one sample before believing a difference.
CASES = [
("WIN A FREE IPHONE NOW", "yes"),
("your package arrives tuesday", "no"),
]
for text, want in CASES:
got = classify(text).strip().lower()
print(("ok " if got.startswith(want) else "FAIL"), text[:30], "->", got)Prompt smells
"Please" and "very important"
Emphasis does not scale. If a rule needs shouting it needs to be the only rule, or enforced by a schema rather than asked for.
A timestamp in the system turn
Different every request, so the prefix cache never hits and you pay full prefill forever. Put it in the user turn.
Asking for JSON in prose
Use response_format. If the model must also explain itself,
put the explanation in a field of the schema.
A rule that contradicts another
"Be thorough" and "be concise" cancel. A small model resolves the conflict by following whichever it read last.
Examples that are all the same shape
Three easy cases teach less than one easy and one awkward. Show the edge you actually care about.
Treating user text as instructions
Anything pasted in is data. Say so in the system turn, and see Security for the part of this llamay enforces rather than asks for.
Where to go next
Tools and schemas is what enforcement actually does. Recipes applies all of this to twenty concrete jobs. The playground lets you try a change against your own model without writing a client.