The Open LLM Leaderboard, a widely-used benchmark for measuring large language model performance, has been experiencing issues. The leaderboard's creator, MMLU, announced that the rankings have become 'unreliable' due to a combination of factors, including data quality problems and changes in model architecture. As a result, some models have seen significant drops in their rankings. This has led to concerns among researchers and developers about the accuracy and reliability of the leaderboard. --- Why it matters: This matters because the Open LLM Leaderboard is a crucial tool for evaluating and comparing large language models, which are increasingly used in applications such as chatbots, virtual assistants, and natural language processing systems. Source: https://huggingface.co/blog/open-llm-leaderboard-mmlu