Researchers have discovered that transformer models develop 'privileged bases' - specific coordinate axes in their residual streams. These axes are tied to the model's prediction direction, which acts as a content-defined anchor. The team found that residual-stream variation is stratified by proximity to this prediction direction, with regions near the prediction being highly structured and regions further away being less so. This organization was consistent across 18 different models of varying sizes and complexity. --- Why it matters: This study's findings are important for understanding how transformer models process information and make predictions. By identifying a privileged anchor in the model's residual streams, researchers can better design and optimize these models for specific tasks and applications. Source: https://arxiv.org/abs/2608.12447