The development
What happened
Tarek Ziadé reports a Transformers prototype for running the 180B-parameter Qwen3.8-Flash-Next language model on a 32GB Mac. It streams the mixture-of-experts model's selected weights from storage through a bounded cache, alongside model-specific memory handling. His final tests approach Strata's generation speed, but he still finds it too slow for his coding-agent workflow; the integration remains a proposal.
Key facts
The proposed path streams experts chosen by the model's router, rather than transferring an entire offloaded expert bank per token. A shared device tensor acts as a slot pool so existing grouped-matrix kernels can use the cached weights.
The Qwen experiment also needs a separate disk path for its large n-gram embedding table and model-specific Metal kernels. Streaming the expert weights alone is not enough to fit this configuration in 32GB.
In the author's final same-M5 tests, short decoding reached 13.9 tokens/s versus Strata's 14.8. Four-thousand-token prefill was 228 versus 220 tokens/s; decoding after a 6K document was 12.4 versus 13.7.
The results are author-reported and depend on hardware, prompts, weight formats and implementation. Earlier memory and speed figures in the post describe different stages of the experiment, not one final universal benchmark.
The generic integration is proposed in Transformers PR 50717, with the API still under discussion. The post uses a PR-specific installation path; it is not announcing a normal stable-release feature.
The context
Why it matters
Sparse routing can reduce the weights needed for each generated token, but a model larger than memory still faces storage, cache and host/GPU synchronization costs. This experiment makes those system trade-offs concrete without showing that a large local model becomes fast enough for every agent workload. Its author keeps using a desktop for his actual coding work. Readers should treat the prototype and reported speed comparison as engineering evidence to evaluate, not a production-ready recommendation or a claim of improved model quality.