Researchers from Hugging Face argue that the design of positional encoding in transformer models can be improved. They propose an alternative method, which they claim outperforms existing approaches on several benchmarks. The new approach is based on learned sinusoidal embeddings, rather than fixed sinusoids or learnable embeddings. --- Why it matters: This matters to researchers and engineers working with transformer models because it challenges the conventional wisdom on positional encoding design. If proven effective, this alternative method could lead to improvements in model performance across various applications. Source: https://huggingface.co/blog/designing-positional-encoding