Tootfinder

Hecto: Modular Sparse Experts for Adaptive and Interpretable Reasoning
Sanskar Pandey, Ruhaan Chopra, Saad Murtaza Bhat, Ark Abhyudaya
https://arxiv.org/abs/2506.22919

Hecto: Modular Sparse Experts for Adaptive and Interpretable Reasoning
Mixture-of-Experts (MoE) models enable conditional computation by routing inputs to specialized experts, but these experts rely on identical inductive biases, thus limiting representational diversity. This static computation pathway is inefficient for inputs that require different types of reasoning and limits specialization and interpretability. We propose Hecto, a lightweight MoE architecture that leverages architectural heterogeneity by combining a GRU expert for temporal reasoning and an FF…