Researchers have found that large language models (LLMs) used to evaluate code can be biased against certain types of code variations. The study, which tested five programming languages and multiple LLMs, discovered that these models are susceptible to both positive and negative biases, leading to inflated or unfairly low scores. This means that an otherwise correct piece of code may be penalized due to superficial differences such as variable names, comments, or formatting. The authors argue that this highlights the need for more robust code evaluation methods. --- Why it matters: This study matters because it reveals a critical flaw in the use of LLMs for code evaluation, which could have significant implications for software development and maintenance. If LLM judges are biased against certain types of code variations, it may lead to incorrect or unfair assessments of code quality, potentially resulting in wasted time and resources. Source: https://arxiv.org/abs/2505.16222