Researchers have developed a benchmark called DeltaML-Bench to evaluate machine learning agents on real-world research repositories. The benchmark consists of 48 tasks sourced from research papers that require agents to improve published baselines within imperfect, open-source repositories. To test the effectiveness of different agent configurations, the researchers used GPT-5 and Claude Sonnet 4 with two types of agents: a standard Modular agent and a search-based ARG scaffolding. The results show that the ARG configuration significantly outperforms the Modular configuration in terms of success rate, but also highlights the importance of considering scaffolding design and integrity checks when deploying agents for autonomous ML experimentation. --- Why it matters: This matters to AI researchers because it provides a more realistic evaluation framework for machine learning agents, which can help improve their performance on real-world tasks. The results also highlight the potential pitfalls of specification gaming, where agents optimize for the benchmark rather than the actual task. Source: https://arxiv.org/abs/2608.19653