· News · 4 min read

llamay, Ollama 0.40.15 and llama.cpp b12122 add fusions, Rust Qwen engine, Qwen3.8 GGUFs

New llama.cpp CUDA fusions, a Rust Qwen inference engine, and updated Qwen3.8-Flash-Next GGUFs arrive; we explain what they mean for local model users.

The llama.cpp b12122 release brings several CUDA-specific fusions, a Rust-written inference engine for Qwen models appears on GitHub, Ollama 0.40.15 adds bundled CUDA libraries on Linux, and Unsloth publishes new Qwen3.8-Flash-Next GGUF files with higher accuracy. All four announcements affect people who run large language models on their own hardware, but only the Ollama and Unsloth updates intersect with llamay’s model store.

Key points

llama.cpp b12122 CUDA fusions

The latest llama.cpp release (b12122) introduces CUDA kernels that fuse NVFP4 gate/up matrix multiplications and GLU operations inside the MMQ path. The change also respects the requested source-1 precision and shares scale checks across kernels, reducing overhead when running quantised models such as W4A8. (llama.cpp releases)

For engineers who compile llama.cpp themselves, the fused kernels can improve throughput on NVIDIA GPUs, especially when using the newer NVFP4 quantisation format. However, llamay does not use llama.cpp under the hood, so these performance gains do not apply to llamay users. If you rely on llama.cpp directly, you will need to rebuild with the new source to benefit.

Rust inference engine for Qwen models

“RustingStrata” is a pure-Rust inference engine that compiles CUDA kernels at start-up with NVRTC and can also fall back to Vulkan. It supports a wide range of Qwen GGUF quantisations (IQ1_S to Q8_0, plus BF16/F16) and handles MoE models by off-loading experts that exceed GPU memory to the CPU, keeping both sides busy. (r/LocalLLaMA)

The engine demonstrates that a non-ggml stack can achieve competitive speed on a 12 GB GPU, particularly for the Qwen3.8-Flash-Next family. For users of llamay, the relevance is indirect: the same GGUF files that RustingStrata runs can also be served by llamay, provided the tensor types are among those supported (e.g. Q5_1, Q8_0). The Rust engine does not replace llamay but shows that alternative runtimes can handle the same model formats.

Ollama 0.40.15 bundled CUDA libraries

Ollama 0.40.15 adds a change that loads bundled CUDA libraries in-process on Linux, removing the need for a system-wide CUDA toolkit at runtime. The update also brings a new MiMo V2.6 renderer for Qwen3.5 models. The full changelog is available in the release notes here.

For local developers this means a simpler install path: pulling a model with ollama pull … now works even on machines without a pre-installed CUDA driver, as long as the bundled libraries match the GPU. llamay can still serve any model already present in Ollama’s store without re-downloading, because it reads Ollama’s blob-based catalogue automatically. No extra steps are required on the llamay side.

Unsloth Dynamic v3.1 Qwen3.8-Flash-Next GGUFs

Unsloth has published Dynamic v3.1 GGUF files for the Qwen3.8-Flash-Next family, claiming 10-15 % accuracy improvements on benchmarks such as DeepSWE and Terminal Bench. The release also adds an “engram switching” mechanism that lets users swap between BF16, Q8_0, Q5_1 and IQ4_NL quantisations without re-exporting the model. The files are hosted on Hugging Face and can be fetched with a standard GGUF URL. (r/LocalLLaMA)

Because llamay supports the listed tensor types (including Q5_1 and IQ4_NL), these new GGUFs can be pulled directly into a llamay store. The engram switching feature is handled by the model file itself; llamay will load whichever quantisation the GGUF contains, and the same binary can be used for all supported precisions. This gives local users a straightforward way to experiment with higher-accuracy variants without changing their inference pipeline.

What this means if you run models locally

The announcements collectively broaden the ecosystem of quantised GGUFs and improve the tooling around them. For llamay users the practical steps are:

  1. Pull the new Unsloth GGUF directly, using the generic URL syntax:

``bash llamay pull https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF/q5_1.gguf ``

  1. If you already have the model in Ollama’s store, llamay will serve it automatically—no extra import is needed.
  2. To experiment with the Rust engine you can still keep the same GGUF files; just run RustingStrata if you want to compare performance on a 12 GB GPU.
  3. The CUDA fusions in llama.cpp do not affect llamay, but if you compile llama.cpp yourself you may see speed gains on supported GPUs.

Overall, the landscape is becoming richer: more quantisation formats, easier CUDA deployment, and higher-accuracy GGUF releases. llamay remains a lightweight, single-binary server that can consume these models without additional runtime dependencies.

FAQ

How do I load the new Unsloth Qwen3.8-Flash-Next GGUF in llamay?

Use llamay pull with the full URL to the GGUF file; once downloaded the model is served automatically on port 11435.

Does the CUDA kernel fusion in llama.cpp improve llamay performance?

No. llamay is written in Go and does not link against llama.cpp, so its performance is unaffected by those kernel changes.

Can I run the RustingStrata engine instead of llamay for Qwen models?

Yes, RustingStrata can execute the same GGUF files on a 12 GB GPU, but it is a separate binary; llamay continues to work with the same models using its own Go-based engine.

Sources