Tootfinder

ElasticMoE: An Efficient Auto Scaling Method for Mixture-of-Experts Models
Gursimran Singh (Huawei Technologies Canada), Timothy Yu (Huawei Technologies Canada), Haley Li (Huawei Technologies Canada), Cheng Chen (Huawei Technologies Canada), Hanieh Sadri (Huawei Technologies Canada), Qintao Zhang (Huawei Technologies China), Yu Zhang (Huawei Technologies China), Ying Xiong (Huawei Technologies Canada), Yong Zhang (Huawei Technologies Canada), Zhenan Fan (Huawei Technologies Canada)

ElasticMoE: An Efficient Auto Scaling Method for Mixture-of-Experts Models
Mixture-of-Experts (MoE) models promise efficient scaling of large language models (LLMs) by activating only a small subset of experts per token, but their parallelized inference pipelines make elastic serving challenging. Existing strategies fall short: horizontal scaling provisions entire replicas of the current configuration, often tens to hundreds of accelerators, leading to coarse granularity, long provisioning delays, and costly overprovisioning. Vertical scaling offers finer adjustments …