Researchers have proposed a new neural network architecture called WhiteMatter that allows for all-to-all cross-layer connections. This is in contrast to traditional Transformer models, which only allow each layer to attend to past tokens through key-value (KV) pairs produced at its own depth. WhiteMatter achieves this by implementing a router that mixes the states of multiple layers into a set of KV channels, which are then cached for subsequent tokens. The architecture has been shown to outperform a vanilla Transformer with 50% more layers in pretraining experiments and retains most of this gain when compressing the KV-cache by 50%. The authors attribute these improvements to the ability of WhiteMatter to adapt connection weights across consumer layers. --- Why it matters: This matters because it shows that traditional Transformer models may be limiting their potential performance by restricting cross-layer connections. Engineers working on AI architectures can learn from this new approach and consider how to implement similar all-to-all connections in their own models, potentially leading to improved performance and efficiency. Source: https://arxiv.org/abs/2608.18486