Amazon explains pair-aware pruning for mixture-of-experts models
An Amazon-authored explainer published on 9 November describes HOPE, a method for shrinking mixture-of-experts language models by considering which experts work together. These models route each token through only some of their neural-network components, but retain the wider expert pool in memory. HOPE uses pairwise activity statistics to …