Capabilities

Images, scans and video

A vision model can read a picture in a chat, a scanned PDF page by page, or a short clip as a handful of timed frames — and answer in a JSON Schema if you ask. It all runs on this machine, on the CPU, and it is slow: plan in minutes per page, not seconds.

What you need

A vision model is two files: the language model, and its vision projector (an mmproj GGUF) that turns an image into positions the model can read. Both come from the same repository.

llamay pull hf:ggml-org/gemma-3-4b-it-GGUF/gemma-3-4b-it-Q4_K_M.gguf -as gemma3-4b-vision
llamay pull hf:ggml-org/gemma-3-4b-it-GGUF/mmproj-model-f16.gguf -as gemma3-4b-mmproj
llamay serve -m gemma3-4b-vision -mmproj gemma3-4b-mmproj -gpu=false

• Projectors: the CLIP/LLaVA MLP family and Gemma 3.
• The CPU path only. Images are read on the CPU in this version, so serve with -gpu=false. A server running its model on a GPU answers an image with a 400: this server runs its model on a GPU, and images are read on the CPU path only in this version. Restart it with -gpu=false to serve image requests.
• Checked at startup. A projector for a different model stops the server rather than producing a confused answer later.

A picture in a chat

Send the image inline, in whichever shape your API uses: an OpenAI image_url part with a data: URI, an Anthropic image block, or Ollama's images list.

import base64
from openai import OpenAI

client = OpenAI(base_url="http://localhost:11435/v1", api_key="unused-on-loopback")
png = base64.b64encode(open("invoice.png", "rb").read()).decode()

reply = client.chat.completions.create(
    model="gemma3-4b-vision", temperature=0, max_tokens=60,
    messages=[{"role": "user", "content": [
        {"type": "text", "text": "Who is the supplier and what is the total? One line."},
        {"type": "image_url", "image_url": {"url": "data:image/png;base64," + png}},
    ]}],
)
print(reply.choices[0].message.content)

Acme Ltd, 1,284.50 EUR

That took 102 seconds on an Apple M4 CPU, almost all of it the vision tower.

• Inline only. An image URL is refused rather than fetched: a server that fetches what a request names is a server that reaches out on a stranger's behalf.
• Not cached. A conversation that carries an image is never added to the prefix cache, so one picture's attention state is never reused for another.
• Not batched. Image requests take the sequential path, one at a time.

Scanned PDFs, into a schema

A scanned PDF is a picture of each page in an envelope, and it is read as one. Send it as an OpenAI file part, an Anthropic document block, or a data:application/pdf URI. Add response_format and the answer is JSON that matches your schema, enforced token by token.

curl -s localhost:11435/v1/chat/completions -d '{"model": "gemma3-4b-vision",
  "messages": [{"role": "user", "content": [
    {"type": "file", "file": {"filename": "invoice.pdf", "file_data": "data:application/pdf;base64,JVBERi0x..."}},
    {"type": "text", "text": "Extract the invoice."}]}],
  "response_format": {"type": "json_schema", "json_schema": {"name": "invoice", "schema": {
    "type": "object", "required": ["invoice_number", "total"],
    "properties": {"invoice_number": {"type": "string"}, "total": {"type": "number"}}}}}}'

• Pages compressed as JPEG or zlib are read.
• CCITT fax, JBIG2 and JPEG 2000 pages are refused by name.
• A PDF whose pages hold text rather than a picture is refused as not a scan. Extract its text instead, or index it with documents after pdftotext.

From the command line

llamay ocr -image reads a picture or a scanned PDF with the same path. It prints the text by default, or JSON with -schema:

llamay ocr -image invoice.pdf -m gemma3-4b-vision -mmproj gemma3-4b-mmproj \
  -schema invoice.schema.json -stats

{"invoice_number":"4471","supplier":"Acme Ltd","total":1284.50,"currency":"EUR"}
read 366 positions (256 of them image) in 1m19.946s · generated 31 in 27.774s · stop: stop

llamay ocr -frames is a different tool: it decodes the output of a CTC text recogniser, optionally against a lexicon. See the CLI reference.

Video, as timed frames

No projector llamay serves was trained on video, so a clip is read the way a person reads a contact sheet: a few frames, in order, each labelled with its time. Send a video_url part holding a data: URI, or the frames you cut yourself as {"type": "video", "video": ["data:image/png;base64,...", ...]}.

• fps (default 1) and max_frames (default 8, at most 64) go on the part or inside video_url. A longer clip is sampled more thinly, keeping the first and last frame, rather than cut short.
• An animated GIF is decoded in Go. Any other format needs ffmpeg on the PATH. It runs as a local subprocess allowed to open only the one file it was handed. Without it, the request is refused with a 400 that says so.
• Every frame costs a whole image's positions. Frames that do not fit the context are thinned until they do, and the response says so in the X-Llamay-Media header (and as llamay_media in a non-streaming OpenAI chat body).

llamay ocr -video lecture.mp4 -fps 0.2 -max-frames 12 -m gemma3-4b-vision -mmproj gemma3-4b-mmproj

Limits, stated plainly

• Slow. About a minute and a half per image or page for a 4B model on a laptop CPU. A video of eight frames is eight of those.
• CPU only. No GPU backend reads images in this version.
• Two projector families. A projector of another kind is refused at startup.
• No image generation. These models read pictures; nothing here draws one.