Researchers have introduced a new evaluation framework called AutoResearchEval to assess the performance of AI agents in carrying out scientific research. The framework includes 100 tasks across seven domains and evaluates eight different combinations of models and scaffolds. The results show that current AI agents lack a 'metacognitive loop', which is the ability to review and revise their own work. This limitation affects all eight model combinations, including the strongest ones tested. The study's findings are intended to facilitate further research into autonomous scientific discovery. --- Why it matters: This matters because it highlights a fundamental limitation of current AI agents in carrying out complex tasks like scientific research. Understanding this limitation is crucial for developing more effective and reliable AI systems that can assist researchers in various domains. Source: https://arxiv.org/abs/2608.14905