Researchers have proposed a method to optimize hyperparameters for large-scale Mixture-of-Experts (MoE) models. The approach involves two steps: first, transferring optimal learning rates across model widths using a Maximal Update Parameterization adaptation; second, extrapolating these values along the token dimension using a predictive scaling law. This allows for efficient training of massive MoE models by proxying on smaller models. --- Why it matters: This matters to AI researchers because it enables the efficient training of large-scale Mixture-of-Experts models, which are crucial for many applications such as natural language processing and computer vision. By transferring optimal hyperparameters across model widths and tokens, this approach can save significant computational resources. Source: https://arxiv.org/abs/2608.20061