~/llamay $ llamay bench -m qwen2.5:0.5b -gpu -prompt 512 -gen 128 -reps 3
llamay v0.1.58-1-gb14f550 · Qwen2.5 0.5B Instruct · qwen2 · 7 threads · kernels neon+i8mm ·
matmul metal:Apple M4 · gemm own kernels · kv f32
weights 373.71 MiB · repack on · context 32768 · prompt 512 · generate 128

phase     rate          best     note
prefill   794.3 tok/s   645ms    one command buffer per 256-token tile
decode    47.3 tok/s    2.705s   506 dispatches per token, one submission

cpu encoding 133ms · gpu execution 3.175s (4% encoding)

phase      rate   best   note
fork x8    11µs          0 pages allocated (a copy would be 64)
snapshot   35ms          12.00 MiB for 512 tokens, at f32
restore    34ms          vs 645ms to re-prefill

gpu time: staging 41ms · kernels 2.333s · copy-back 50ms

Restoring a 512-token context is 18.9x faster than recomputing it.
~/llamay $ # prefill is arithmetic-bound; decode is memory-bound. One number would hide that
.