The development
What happened
An Amazon-authored explainer published on 9 November describes HOPE, a method for shrinking mixture-of-experts language models by considering which experts work together. These models route each token through only some of their neural-network components, but retain the wider expert pool in memory. HOPE uses pairwise activity statistics to help choose a fixed set of experts to remove, rather than ranking each expert in isolation. The underlying research paper was first submitted in September; this is a later explanation of that work, not a newly released November paper.
Key facts
Calibration records expert-pair activity, routing weights and output norms in a matrix for each layer. HOPE solves a relaxed quadratic optimisation problem, rounds the result into a removal set and physically deletes selected expert weights.
The paper compares three models, two calibration datasets and four first-order pruning baselines using language-model and agentic-coding benchmarks. The authors report stronger advantages at 40–50% expert removal; lighter-pruning results are less decisive.
Pruning itself requires no retraining, but preparation has a cost. The paper reports roughly 6–7% longer calibration than REAP on a same-size dataset and some additional data needs. Separate fine-tuning experiments are not prerequisites for applying HOPE.
The main method considers expert pairs within each layer, not interactions across layers or larger expert groups. The cross-layer extension remains preliminary, and at 10–20% pruning first-order competitors can remain competitive.
The underlying paper was first submitted on 16 September and its current version is dated 29 September. The 9 November blog is a later author explainer, not evidence that the paper or code first appeared that day.
The context
Why it matters
TideReach assessment: expert removal offers a way to reduce a model’s stored parameter footprint, but the useful question is how much task performance survives under the actual deployment workload. HOPE makes expert cooperation part of that selection decision. Its calibration choices, approximations and uneven benchmark advantages still matter: a smaller checkpoint is not by itself evidence of faster inference, lower total operating cost or preserved agent reliability. TideReach has not run the package, reproduced its experiments or measured a production deployment.