Large transformer models have become popular due to their ability to achieve state-of-the-art results in various tasks. However, they come with a high cost, both in terms of time and memory required for inference. This is a significant bottleneck for using these models at scale. According to Pope et al. (2022), the main factors contributing to this challenge are the increasing size of state-of-the-art models and two other key factors. --- Why it matters: This matters because large transformer models have the potential to solve complex real-world tasks, but their high inference cost makes them impractical for widespread use. Engineers working on AI applications need to find ways to optimize these models' performance without sacrificing accuracy. Source: https://lilianweng.github.io/posts/2023-01-10-inference-optimization/