Guides

Migrating to llamay

From OpenAI, Anthropic, Ollama or llama.cpp. In most cases the change is a base URL, because llamay speaks those APIs rather than translating them — the differences that remain are listed here rather than discovered.

From the OpenAI API

Change two lines. The SDK does not need to know.

from openai import OpenAI

client = OpenAI(
    base_url="http://127.0.0.1:11435/v1",
    api_key="not-used",      # required by the SDK, ignored unless -api-key is set
)

client.chat.completions.create(
    model="",                 # "" means the server's loaded model
    messages=[{"role": "user", "content": "hello"}],
)

import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "http://127.0.0.1:11435/v1",
  apiKey: "not-used",
});

await client.chat.completions.create({
  model: "",
  messages: [{ role: "user", content: "hello" }],
});

Or set OPENAI_BASE_URL and touch no code at all — which is the version that works for a dependency you do not control.

export OPENAI_BASE_URL=http://127.0.0.1:11435/v1
export OPENAI_API_KEY=not-used

What is different

From the Anthropic API

from anthropic import Anthropic

client = Anthropic(base_url="http://127.0.0.1:11435", api_key="not-used")

client.messages.create(
    model="", max_tokens=512,
    system="Answer in one sentence.",
    messages=[{"role": "user", "content": "hello"}],
)

/v1/messages is served natively: system as a string or a list of blocks, stop_sequences, tools and tool_choice, streaming in Anthropic's event shape, and stop_reason distinguishing end_turn from stop_sequence with stop_sequence naming which one matched. max_tokens is required there, as it is upstream.

From Ollama

Your models come with you. llamay reads Ollama's store directly, so nothing is re-downloaded and nothing is converted.

llamay run -m llama3.2:3b -p "hello"      # the model Ollama already pulled
llamay serve                               # /api/chat, /api/generate, /api/tags all served

The different port is on purpose: you can point one client at each and compare them on the same machine, against the same weights, without stopping either.

From llama.cpp

Same GGUF files, same quantisation names, same tokenizer behaviour — the correctness page covers what is checked against llama.cpp and what that check actually proves.

The thing with no equivalent upstream is the state engine: a context is a first-class object you can fork N ways as a page-table copy, snapshot to disk, and restore on another machine.

LangChain, LlamaIndex and the rest

Anything that takes an OpenAI-compatible base URL works unchanged.

from langchain_openai import ChatOpenAI

llm = ChatOpenAI(
    base_url="http://127.0.0.1:11435/v1",
    api_key="not-used", model="",
)

from llama_index.llms.openai_like import OpenAILike

llm = OpenAILike(
    api_base="http://127.0.0.1:11435/v1",
    api_key="not-used", model="",
    is_chat_model=True,
)

For embeddings, point the same base URL at /v1/embeddings and start the server with -embed. One process then holds the decoder, the encoder and the cross-encoder, which is what makes a local retrieval stack a single binary rather than three services.

A migration checklist

1. Point the base URL at llamay and run your existing tests. Most will pass.
2. Pick a model that fits. A 7B is not GPT-4. See sizing, and expect to shorten prompts — prompting local models is about exactly this.
3. Replace format-by-prose with response_format. Then delete the sentences asking for JSON; they are now competing with instructions that still matter.
4. Move stable material into the system turn so the prefix cache can hold it, and take the timestamp out of it.
5. Handle 429 rather than 500. A local server has a finite queue and says so with a Retry-After.
6. Check [DONE] on streams. A truncated stream and a finished one otherwise look the same.
7. Decide what happens when it cannot answer. Local-first with hosted overflow is a documented pattern, and the redaction step in the middle only works because it is local.

What to expect, honestly

Where to go next

Prompting local models is the adjustment that matters most after the base URL. Errors is what the new failures mean. Deployment patterns is where it runs once it is more than your laptop.