Researchers propose DeltaMomentum, a new method for updating momentum in deep learning optimizers. Unlike traditional exponential moving average (EMA) methods, which forget all directions at the same rate, DeltaMomentum builds direction-awareness into its update rule. This is achieved by treating the gradient of a linear layer as a key-value pair, where the input acts as the key and the output-side error as the value. The authors prove that DeltaMomentum is a valid momentum method, applies curvature correction without matrix inversion, and clears stale directions faster than EMA. Experiments show that using DeltaMomentum with AdamW (DeltaAdamW) can reach similar validation loss in fewer steps compared to traditional AdamW, and this gain persists at larger scales. --- Why it matters: This matters because it addresses a common issue in deep learning optimizers: anisotropy in the input data. Traditional methods may forget important directions too slowly or too quickly, leading to suboptimal performance. DeltaMomentum's direction-aware update rule can improve training efficiency and accuracy by adapting to the specific characteristics of the input data. Source: https://arxiv.org/abs/2608.19491