~/llamay $ llamay info -m testdata/azmx-tiny.gguf | head -8
llamay        v0.1.58-1-gb14f550
kernels       neon+i8mm
gpu backend   metal:Apple M4
file          testdata/azmx-tiny.gguf
gguf version  3
size          2.39 MiB
tensors       38
architecture  azmx (decoder)
~/llamay $ llamay quantize -i testdata/azmx-tiny.gguf -o build/demo/work/tiny-q4.gguf -type
q4_k
29 tensors converted, 9 left as they were
25 of them are Q4_0 rather than Q4_K: their rows are not a multiple of 256
2.38 MiB -> 346.64 KiB (14.2% of the original) in 12ms
~/llamay $ ls -lh testdata/azmx-tiny.gguf build/demo/work/tiny-q4.gguf | awk '{print $5, $9}
'
353K build/demo/work/tiny-q4.gguf
2.4M testdata/azmx-tiny.gguf
~/llamay $ llamay info -m build/demo/work/tiny-q4.gguf | head -8
llamay        v0.1.58-1-gb14f550
kernels       neon+i8mm
gpu backend   metal:Apple M4
file          build/demo/work/tiny-q4.gguf
gguf version  3
size          353.25 KiB
tensors       38
architecture  azmx (decoder)