Researchers have identified an issue with how AI coding agents are evaluated, specifically with the pass@k estimator used in benchmarks. The current implementation misapplies the metric by conflating test-suite size with attempt independence. A new reliability@k estimator is proposed, which correctly applies the metric using independent rollouts and fully-passing rollouts per task-agent pair. Additionally, a security-adjusted reliability@k estimator is introduced to account for high-severity insecure patterns in rollouts. Initial results show that the adjustment does not change rankings under current evaluation methods, but its potential impact requires further investigation. --- Why it matters: This matters because it highlights the importance of accurate and robust evaluation metrics for AI coding agents. The proposed reliability@k and security-adjusted reliability@k estimators can help improve the trustworthiness of these agents and ensure they meet functional correctness and security safety standards. Source: https://arxiv.org/abs/2608.14711