Researchers have developed a new benchmark for evaluating large language models (LLMs), called the Metanym Game, which measures their ability to generate analogous statements and rate each other's work. Unlike traditional benchmarks that rely on pre-existing ground truth, this game uses only the LLMs' own ratings to determine scores. The study found that the best-performing LLMs were not necessarily the most accurate judges, but rather those that struck a balance between generating good analogies and providing consistent ratings. This benchmark has been shown to correlate strongly with another evaluation method, GPQA Diamond, which uses human-written questions. --- Why it matters: This matters because it provides a new way to evaluate LLMs without relying on pre-existing ground truth, which can be biased or incomplete. The Metanym Game also allows for more nuanced analysis of LLM intelligence by evaluating multiple aspects of their performance simultaneously. Source: https://arxiv.org/abs/2606.21008