The development

What happened

Tarek Ziadé reports a Transformers prototype for running the 180B-parameter Qwen3.8-Flash-Next language model on a 32GB Mac. It streams the mixture-of-experts model's selected weights from storage through a bounded cache, alongside model-specific memory handling. His final tests approach Strata's generation speed, but he still finds it too slow for his coding-agent workflow; the integration remains a proposal.

Key facts

  • The proposed path streams experts chosen by the model's router, rather than transferring an entire offloaded expert bank per token. A shared device tensor acts as a slot pool so existing grouped-matrix kernels can use the cached weights.

  • The Qwen experiment also needs a separate disk path for its large n-gram embedding table and model-specific Metal kernels. Streaming the expert weights alone is not enough to fit this configuration in 32GB.

  • In the author's final same-M5 tests, short decoding reached 13.9 tokens/s versus Strata's 14.8. Four-thousand-token prefill was 228 versus 220 tokens/s; decoding after a 6K document was 12.4 versus 13.7.

  • The results are author-reported and depend on hardware, prompts, weight formats and implementation. Earlier memory and speed figures in the post describe different stages of the experiment, not one final universal benchmark.

  • The generic integration is proposed in Transformers PR 50717, with the API still under discussion. The post uses a PR-specific installation path; it is not announcing a normal stable-release feature.

The context

Why it matters

Sparse routing can reduce the weights needed for each generated token, but a model larger than memory still faces storage, cache and host/GPU synchronization costs. This experiment makes those system trade-offs concrete without showing that a large local model becomes fast enough for every agent workload. Its author keeps using a desktop for his actual coding work. Readers should treat the prototype and reported speed comparison as engineering evidence to evaluate, not a production-ready recommendation or a claim of improved model quality.