Researchers have proposed a new approach to efficient transformers called Sparse Token Routing. They designed a two-stream transformer that routes tokens through either lightweight or full-capacity processing using a learned gate. The study found that this routing method introduces negligible accuracy change compared to parameter-matched baselines, but the effectiveness of the gate's token-importance signal depends on how it is learned. A static lexicon-seeded prior failed to achieve desired results, while a fully contextual gate performed well on both evaluated tasks. --- Why it matters: This matters because efficient transformers are crucial for large-scale AI applications where computational resources are limited. This research provides insights into designing more efficient and accurate transformer models that can handle complex tasks with minimal resource usage. Source: https://arxiv.org/abs/2608.20632