Researchers developed a controlled test to evaluate the performance of large language models (LLMs) in medical reasoning. They created synthetic patient vignettes with varying levels of evidence strength, conflicting information, and missing data to assess how accurately an LLM can diagnose conditions such as Alzheimer's disease or depression-related cognitive impairment. The study found that the LLM, gpt-4.1-nano, was highly accurate (93.5%) but showed inconsistent confidence in its answers, with higher confidence often not corresponding to actual accuracy. This suggests that while LLMs have some metacognitive sensitivity, they can still make errors when faced with complex or ambiguous cases. --- Why it matters: This study matters because it provides a framework for evaluating the performance of medical LLMs and highlights areas where these models may be unreliable, such as in cases with conflicting evidence. Understanding how to improve the metacognitive abilities of LLMs is crucial for their adoption in real-world clinical settings. Source: https://arxiv.org/abs/2608.14552