Capabilities

Structured output and tools

Ask for JSON in a prompt and a model usually complies. Constrain it and it always does: llamay compiles your JSON Schema into a filter on the model's vocabulary, so at every step it can only choose a token that keeps the output valid. Tool calls are built on the same machinery. This page is what is enforced, what is ignored, what is refused, and how tool calls behave.

How the constraint works

A JSON Schema is compiled to a byte-level pushdown automaton. A token is allowed only if every byte it contributes keeps the automaton alive. The model can still produce the wrong value, but not a syntax error, a missing required key, a string where a number belongs, a number out of its bounds, or a function nobody offered.

Transitions are cheap and numerous; masks are expensive and few; so the two are cached separately, which is what keeps the cost to tens of per cent rather than a stall per token.

Asking for structured output

Send	On	Guarantee
`response_format: {"type": "json_schema", "json_schema": {"schema": …}}`	OpenAI routes	The schema
`response_format: {"type": "json_object"}`	OpenAI routes	Any valid JSON
`format: {…schema…}` or `format: "json"`	Ollama routes	The schema, or any JSON
`llamay_schema: {…}`	OpenAI and Anthropic routes	The schema; wins over `response_format`
`llamay_format: "json"`	OpenAI and Anthropic routes	Any valid JSON

Here is a schema using $ref, anyOf, numeric bounds, multipleOf and patterns, sent to a 0.5B model:

{"model": "qwen2.5:0.5b", "temperature": 0,
 "messages": [{"role": "user", "content": "Order 7 from Acme: 3 widgets at 4.50 EUR each, ship to Lyon, postcode 69001."}],
 "response_format": {"type": "json_schema", "json_schema": {"name": "order", "schema": {
   "$defs": {"line": {"type": "object",
     "properties": {"item": {"type": "string", "maxLength": 40},
                    "qty": {"type": "integer", "minimum": 1, "maximum": 999},
                    "unit_price": {"type": "number", "minimum": 0, "multipleOf": 0.01}},
     "required": ["item", "qty", "unit_price"]}},
   "type": "object",
   "properties": {
     "order_id": {"type": "string", "pattern": "^[0-9]{1,6}$"},
     "lines": {"type": "array", "items": {"$ref": "#/$defs/line"}, "minItems": 1, "maxItems": 5},
     "delivery": {"anyOf": [
       {"type": "object", "properties": {"city": {"type": "string"}, "postcode": {"type": "string", "pattern": "^[0-9]{5}$"}},
        "required": ["city", "postcode"]},
       {"type": "null"}]}},
   "required": ["order_id", "lines", "delivery"]}}}}

{ "order_id": "7", "delivery": { "postcode": "69001", "city": "Lyon" }, "lines": [ { "item": "Widget 1", "unit_price": 4.50, "qty": 3 }, { "item": "Widget 2", "unit_price": 4.50, "qty": 3 }, { "item": "Widget 3", "unit_price": 4.50, "qty": 3 } ] }

Every constraint held: the id matches its pattern, the postcode has five digits, the prices are multiples of 0.01, there are at most five lines. And the model still got the content wrong — three lines for one order line. The schema decides the shape; the values are the model's. A bigger model, or an example in the prompt, fixes the second; nothing but a schema fixes the first.

The same request twice — once with nothing enforcing the shape, once with the sampler unable to leave it:

Transcript — the whole session as text

~/llamay $ # a 0.5B model, asked for structure, with nothing enforcing it
~/llamay $ llamay run -m qwen2.5:0.5b -chat -n 90 -p "Give me a JSON object with keys name,
port and tls for a local server."
Certainly! Below is a JSON object containing the specified keys:

```json
{
  "name": "localServer",
  "port": "8080",
  "tls": "true"
}
```

This object defines a local server with the following properties:
- `name`: The name of the server (e.g., "localServer").
- `port`: The port on which the server will listen (e.g., `8

prompt 25 tokens in 175ms (142.5 tok/s) · generated 90 in 7.357s (12.2 tok/s) · stop: length
~/llamay $ # the same request with -json: the sampler cannot emit an invalid token
~/llamay $ llamay run -m qwen2.5:0.5b -chat -json -n 90 -p "Give me a JSON object with keys
name, port and tls for a local server."
{
  "name": "Local server",
  "port": 8080,
  "tls": true
}

prompt 25 tokens in 199ms (125.7 tok/s) · generated 26 in 1.668s (15.6 tok/s) · stop: eog
json mask: 27 steps, 11.97 ms each, 19.4% of decode · no mask cache in this automaton
~/llamay $ llamay run -m qwen2.5:0.5b -chat -json -n 90 -p "Give me a JSON object with keys
name, port and tls for a local server." | jq .

prompt 25 tokens in 172ms (145.3 tok/s) · generated 26 in 1.473s (17.7 tok/s) · stop: eog
json mask: 27 steps, 11.10 ms each, 20.4% of decode · no mask cache in this automaton
{
  "name": "Local server",
  "port": 8080,
  "tls": true
}

JSON that is JSON. -json puts a grammar over the sampler. Invalid tokens are not merely unlikely, they are unreachable, so the output parses on the first try from a model far too small to be trusted to remember a brace. Recorded 2026-09-18 · Apple M4, macOS 26.0.1 · llamay v0.1.58-1-gb14f550 · metal:Apple M4 · 0:28

What is enforced, ignored and refused

Keyword	Status	Notes
`type` (one, or a list)	enforced	`object`, `array`, `string`, `number`, `integer`, `boolean`, `null`. With no `type`, `properties` implies object and `items` implies array.
`properties`, `required`	enforced	At most 64 properties per object. A required name that is not a property is refused.
`additionalProperties`	enforced	Closed by default when `properties` is declared — the opposite of the JSON Schema default. `true` opens it; a schema constrains the extra keys.
`items` (one schema), `minItems`, `maxItems`	enforced	A list of schemas under `items` (tuple form) is refused.
`enum`, `const`	enforced	At most 64 members. Members another keyword rules out are dropped.
`minLength`, `maxLength`	enforced	In characters, not bytes.
`pattern`	enforced	RE2 syntax, compiled to a DFA of at most 2,000 states. Lookaround, back-references, `\b` and multi-line anchors are refused. Unanchored unless you write `^` and `$`.
`minimum`, `maximum`, `exclusiveMinimum`, `exclusiveMaximum`	enforced	Integers exactly, up to 18 digits. A bounded non-integer is written without an exponent, to 15 significant digits.
`multipleOf`	enforced	On integers, any step. On numbers, only a whole number or a power of ten such as `0.01`; any other step is refused.
`$ref`	enforced	Local references only (`#/$defs/…`, `#/definitions/…`, any JSON pointer). Recursion works, to a depth of 64. Remote references and `$anchor` are refused.
`anyOf`	enforced	Up to 256 alternatives. Keywords beside it are merged into each alternative.
`oneOf`	as `anyOf`	Exclusivity is not enforced: a document matching two alternatives is accepted.
`allOf`	enforced	Merged when the schema is compiled: types intersect, bounds tighten, properties and required lists combine. A merge that cannot be satisfied — two different patterns, contradictory types — is refused.
`format`, `uniqueItems`, `prefixItems`, `contains`, `minProperties`, `maxProperties`, `propertyNames`, `dependentRequired`	ignored	Parsed and not enforced. `format` is ignored rather than refused because schemas generated from code carry it everywhere.
`title`, `description`, `default`, `examples`	ignored	Annotations.
`not`, `if` / `then` / `else`, `patternProperties`	refused	A 400 naming the keyword and where it is.

{"error":{"message":"properties.a uses \"not\", which llamay's constraint cannot honour; send a schema without it rather than one that is silently not enforced","type":"Bad Request"}}

Structural whitespace is budgeted

At most 64 characters of whitespace between tokens per document. Whitespace is the only byte that returns the automaton to the state it came from, so with no bound a model whose best legal token is a space emits spaces until the token budget runs out. Past the budget, output continues compact.

Tool calls

tools and tool_choice work in all three API shapes: OpenAI's tool_calls, Anthropic's tool_use blocks, and Ollama's tools.

curl -s localhost:11435/v1/chat/completions -d '{
  "model": "gemma2:2b", "temperature": 0,
  "messages": [{"role": "user", "content": "What is the weather in Paris and in Tokyo?"}],
  "tools": [{"type": "function", "function": {
    "name": "get_weather", "description": "Current weather for one city.",
    "parameters": {"type": "object",
      "properties": {"city": {"type": "string"}, "unit": {"enum": ["c", "f"]}},
      "required": ["city", "unit"]}}}],
  "tool_choice": "required"}'

"tool_calls": [
  {"function": {"arguments": "{\"city\":\"Paris\",\"unit\":\"c\"}", "name": "get_weather"}, "id": "call_baca9f0f24c5", "index": 0, "type": "function"},
  {"function": {"arguments": "{\"city\":\"Tokyo\",\"unit\":\"c\"}", "name": "get_weather"}, "id": "call_a88233a1e991", "index": 1, "type": "function"}]

finish_reason is tool_calls and content is null. The same request to qwen2.5:0.5b made one call, for Paris: how many calls to make is the model's choice.

`tool_choice`	What happens
`"auto"` (default)	The tools are described to the model, and calls are read from what it writes. Not constrained: a small model may answer in prose instead. Calls to tools that were not offered are dropped.
`"required"` (or Anthropic's `{"type": "any"}`)	Constrained: the output must be one or more calls, each matching its tool's schema.
A named function (`{"type": "function", "function": {"name": …}}`, or Anthropic's `{"type": "tool", "name": …}`)	Constrained to that tool.
`"none"`	Tools are not offered to the model at all.

Parallel calls. When a call is forced, the model may make one or more, each constrained. parallel_tool_calls: false (OpenAI) or disable_parallel_tool_use: true (Anthropic) allows exactly one. There is no way to force a particular number. Ollama's route always makes one.
A forced tool replaces other constraints. response_format, a grammar or a lexicon on the same request is set aside for the tool's schema.
Streaming holds the call back. With tools offered, the answer is generated and then sent: the deltas carry the whole call at once, because a partial call is not something a caller can act on.
Client tools only. Hosted tool types (web search, code interpreter and the like) are refused by name; llamay runs nothing on your behalf.
Not on /v1/responses in this version.

A tool call whose arguments fit the tool's own schema, because they could not have been anything else:

Transcript — the whole session as text

~/llamay $ llamay serve -m gemma2:2b -addr localhost:11439
time=2026-09-18T08:49:46.415-04:00 level=INFO msg="device graph attached" max-tokens=256
time=2026-09-18T08:49:46.415-04:00 level=INFO msg="llamay serving" addr=localhost:11439 mode
l=gemma2:2b arch=gemma2 max-loaded=2 keepalive=5m0s params="1.59 GB" context=8192 threads=7
kv=f32 lexicon=0

  OpenAI      POST http://localhost:11439/v1/chat/completions
  Anthropic   POST http://localhost:11439/v1/messages
  Embeddings  POST http://localhost:11439/v1/embeddings     (start with -embed <encoder.gguf
>)
  Rerank      POST http://localhost:11439/rerank            (start with -rerank <reranker.gg
uf>)
  Contexts    POST http://localhost:11439/v1/contexts        (then /{id}/fork?n=8, /{id}/sna
pshot)
  Models      POST http://localhost:11439/api/pull          (and /api/copy, /api/delete)
  Stats       GET  http://localhost:11439/v1/stats


~/llamay $ # the server stays up; the rest of this is a second terminal
~/llamay $ curl -s http://localhost:11439/v1/chat/completions -H 'content-type: application/
json' -d '{
>   "model": "gemma2:2b",
>   "messages": [{"role":"user","content":"What is the weather in Pune?"}],
>   "tools": [{"type":"function","function":{
>     "name": "get_weather",
>     "parameters": {"type":"object",
>       "properties": {"city":{"type":"string"},
>                      "unit":{"enum":["c","f"]}},
>       "required": ["city","unit"]}}}]
> }' | jq '.choices[0].message.tool_calls'
[
  {
    "function": {
      "arguments": "{\"city\":\"Pune\",\"unit\":\"c\"}",
      "name": "get_weather"
    },
    "id": "call_949aba1576ea",
    "index": 0,
    "type": "function"
  }
]
~/llamay $ # city is a string and unit is one of two letters, because it could not be otherw
ise

Tool calls that fit the tool's own schema. The tool's JSON Schema becomes the grammar over the sampler, so the arguments the model returns are arguments your function can take. Recorded 2026-09-18 · Apple M4, macOS 26.0.1 · llamay v0.1.58-1-gb14f550 · metal:Apple M4 · 0:22

GBNF grammars

When JSON is the wrong shape — a query language, a fixed format — a GBNF grammar constrains the sampler directly. llama.cpp's grammar files are read and compiled onto the same automaton:

cat > move.gbnf <<'G'
root   ::= piece square square
piece  ::= "K" | "Q" | "R" | "B" | "N" | ""
square ::= [a-h] [1-8]
G
llamay run -m qwen2.5:0.5b -grammar-file move.gbnf -p "White's best opening move:"

Over HTTP, send the grammar text as llamay_grammar (or grammar, llama.cpp's name); a file path is never read from a request. serve -grammar-file sets one for every request. A grammar and a JSON schema on the same request is refused. Three things are refused when the grammar loads, by name:

A left-recursive rule, which would never stop expanding. The message suggests the right-recursive form.
Token terminals (<think>, <[1000]>), which name one model's vocabulary; this constrains bytes.
A repetition bound above 4096, since each count is a separate state.

A grammar that is not JSON, applied at every step:

Transcript — the whole session as text

~/llamay $ cat testdata/grammars/list.gbnf
root ::= item+

# Excludes various line break characters
item ::= "- " [^\r\n\x0b\x0c\x85\u2028\u2029]+ "\n"
~/llamay $ llamay run -m qwen2.5:0.5b -chat -grammar-file testdata/grammars/list.gbnf -n 80
-p "List four things that break when a laptop goes offline."
- No internet connection
- No apps running
- No apps open
- No data saved
- No data accessible
- No device locked
- No device removed
- No device suspended
- No device in sleep mode
- No device in recovery mode
- No device in repair mode
- No device in repair mode
- No device in maintenance mode
- No device in maintenance

prompt 19 tokens in 151ms (125.7 tok/s) · generated 80 in 4.816s (16.6 tok/s) · stop: length
grammar mask: 80 steps, 0.65 ms each, 1.1% of decode · 74 cached (92.5% hit rate)
  states 2484/4096 (peak 2484) · masks 6/256 · 6 states wanted a mask · 0 table clears
~/llamay $ # and an arithmetic grammar, which the model cannot leave
~/llamay $ cat testdata/grammars/arithmetic.gbnf
root  ::= (expr "=" ws term "\n")+
expr  ::= term ([-+*/] term)*
term  ::= ident | num | "(" ws expr ")" ws
ident ::= [a-z] [a-z0-9_]* ws
num   ::= [0-9]+ ws
ws    ::= [ \t\n]*
~/llamay $ llamay run -m qwen2.5:0.5b -grammar-file testdata/grammars/arithmetic.gbnf -n 40
-p "2 + 2"
8 +30 +29+13 = 120


























































































































































































































prompt 4 tokens in 47ms (85.1 tok/s) · generated 40 in 1.689s (23.7 tok/s) · stop: length
grammar mask: 40 steps, 0.91 ms each, 2.2% of decode · 32 cached (80.0% hit rate)
  states 35/4096 (peak 35) · masks 8/256 · 8 states wanted a mask · 0 table clears

GBNF: a shape you wrote yourself. When the shape is not JSON, describe it. The grammar is compiled to a token mask and applied at every step. Recorded 2026-09-18 · Apple M4, macOS 26.0.1 · llamay v0.1.58-1-gb14f550 · metal:Apple M4 · 0:20

What is not verified

Tool-calling quality varies by model. The prompt convention is Hermes' <tool_call> tags, which Qwen and Hermes fine-tunes follow; other families may call tools less readily under "auto". Forced calls are valid for any model; whether they are sensible depends on it.
Constrained decoding costs throughput: the figure above is one machine and one model. Measure yours with llamay run -stats, which reports what the mask cost.