How it works

Backends and hardware

One binary covers every CPU. Each GPU backend — Metal, CUDA, Vulkan — is compiled into particular release builds, so which one you downloaded decides which hardware llamay can use. This page is which build carries what, how the binary chooses, and what each backend declines.

The builds you can download

Artifact	Runs on	Needs to run
`darwin_arm64_metal.tar.gz`, and the Mac app	Metal, with Apple's Accelerate; the CPU otherwise	Apple silicon
`darwin_arm64.tar.gz`, `darwin_amd64.tar.gz`	The CPU	Nothing
`linux_amd64_gpu.tar.gz`	CUDA or Vulkan, chosen at run time; the CPU otherwise	An NVIDIA driver (compute capability 6.1 or later), or a Vulkan loader and driver
`linux_{amd64,arm64}.tar.gz`, the `.deb` and `.rpm` packages	The CPU	Nothing
Windows installers and zips	The CPU	Nothing
`android_arm64.tar.gz`	The CPU	Nothing

The CPU kernels are hand-written assembly: NEON with DotProd and i8mm on ARM, AVX2 and AVX-512 VNNI on x86. llamay version prints what a given binary carries: its kernels, the GPU backends compiled in, the tensor formats and the architectures.

dlopen, not link

Both discrete backends resolve their driver at first use rather than linking against it, so one Linux binary runs on a machine with a card and on one without, declining cleanly instead of failing to start. CI asserts this with ldd on every build, because it is the kind of property that regresses the moment somebody adds a convenient import.

How the binary decides

Three tiers, and the fall from one to the next is per weight format rather than per model. That is what stops one unsupported tensor from moving an entire forward pass back to the CPU.

CPU

Hand-written assembly for all eight block formats on both architectures, and a portable Go reference everywhere else which is correct and 2.8× to 7.6× slower per dot product on an M4. The hot shape is a W4A8 fused dot product: the activation is quantised to int8 once per matmul and weight rows are never materialised.

The CPU is not a fallback that exists on paper. The portable kernels are built for riscv64, s390x, ppc64le and wasm on every change, and the whole test suite runs through them on a machine that has SIMD — because building them is not the same as running them. LLAMAY_BACKEND=portable selects them on your machine too.

Metal

The whole forward pass on the device, and the widest block coverage of the three: thirteen of the fourteen generated shapes run, including all three mixtures, all three parallel-residual blocks and the attention sink. gpt-oss is the one it refuses, twice over — metal.Unsupported declines its expert biases, and NewGraph declines its clipped SwiGLU gate before it gets that far.

TestEveryShapeIsRunOrRefused ranges over the generated shapes rather than over that function's clauses, so a block feature added to the CPU and not implemented on the device fails the build until somebody either implements it or declares it.

Accelerate is not bit-identical

It dequantises weights to float32 for sgemm; the default path keeps them quantised and quantises the activation to int8. Two methods, not one method and its rounding error — 0.09% of perplexity end to end. LLAMAY_NO_BLAS=1 makes an accelerate build exact again, which is the flag to reach for before blaming a model.

A refusal that was costing twelve times the prefill

Gemma was refused on Metal over a device-versus-CPU perplexity difference reported as about one per cent. Re-measured, it is 0.05% — 4.3965 on the device against a CPU reference of 4.3988, and on the better side. The refusal was costing gemma3-270m 47 tok/s on the per-matmul fallback against 552 on the graph.

CUDA

Pascal (2016) onward, seven weight formats, __dp4a for the W4A8 inner product, split-position attention for decode, and a tensor-core prefill kernel on Turing and later. The toolkit is needed only to regenerate the PTX, which is checked in.

Below compute capability 6.1 it declines at open with a message naming the reason, rather than running through a slower kernel that would need separate testing.

Repacking is a per-device decision now

llamay used to rewrite Q5_0 weights to Q8_0 at load. That is worth 2.9× on a CPU, cost 10% of decode on a T4, and helps on Metal, whose unified memory makes the same trade come out the other way. tensor.Repack asks the device instead — CUDA reads CU_DEVICE_ATTRIBUTE_INTEGRATED, Vulkan reads VkPhysicalDeviceProperties.deviceType, Metal always shares — which on a T4 is worth +10.4% decode at +0.4% prefill.

Vulkan

AMD, Intel, Android, and NVIDIA as a fallback. The whole forward pass on the device, ten quantised weight formats plus F32/F16/BF16, and VK_KHR_shader_integer_dot_product where the driver has it.

It is the least optimised of the three device backends, and is said so rather than presented as parity: measure it on your card with llamay bench -gpu before relying on it. The seven i-quants have no Vulkan kernel and are declined by name at both tiers, because an i-quant decodes through a per-format codebook rather than arithmetically and that is a different kind of kernel.

Cooperative matrix has a kernel — all ten quantised formats, worth +44% of prefill on a T4 — and it is the one kernel here that is not bit-identical to the CPU, bounded at 0.1% of the output range. The reason is the operands rather than the accumulation order: this hardware's cooperative matrix takes float16, and a dequantised W4A8 weight does not fit in float16 exactly. That bound is stated rather than hidden.

What each backend refuses

Block feature	CPU	Metal	CUDA	Vulkan
Dense decoders — Llama, Qwen, Gemma 2/3, Phi-3, GPT-2, StableLM	yes	yes	yes	yes
Mixture of experts — Mixtral, Qwen2-MoE, Qwen3-MoE	yes	yes	declines	declines
Parallel residual — Falcon, GPT-NeoX, Phi-2	yes	yes	declines	declines
Attention sinks	yes	yes	declines	declines
`gpt-oss` — MXFP4 experts, clipped SwiGLU, expert biases	yes	declines	declines	declines
The i-quants — IQ2, IQ3, IQ4	portable Go only	no kernel	no kernel	declined by name
Mamba / SSM, DeepSeek-V2 latent attention	refused at load	—	—	—

A model a device graph declines still runs correctly on that machine: the CPU path is the one that executes, and Model.DeviceUnsupported is the check CUDA and Vulkan make before building a graph. One function answering for three backends can only be as permissive as the least capable, which is why Metal answers metal.Unsupported separately.

Environment

Variable	Effect
`LLAMAY_BACKEND=portable`	Force the reference Go kernels. The way to tell a SIMD bug from a block bug.
`LLAMAY_GPU`	Force a backend on or off.
`LLAMAY_NO_BLAS=1`	Make an `accelerate` build bit-exact again.

llamay version prints what a given build actually carries: its tags, its kernel sets, the GPU backends compiled in, the formats and the architectures. That is the first command to run when a number looks wrong.