Researchers have systematized the failures of large-language-model (LLM) agents used in penetration testing by evaluating ten widely used security tools. They introduced an Integration Friction Index to measure the costs of implementing and maintaining these systems. The study found that long-lived sessions lose evidence, while short-lived sub-agents can extend the usable horizon. It also showed that a two-stage verdict cascade has limited benefits when scorer errors are correlated. The researchers formulated planner-versus-worker model routing as a knapsack problem and derived an execution cap for heavy-tailed tools. They concluded that scope and budget enforcement cannot be delegated to system prompts, which do not constrain what actually executes. --- Why it matters: This matters because it provides a framework for understanding the operational failures of LLM-driven penetration testing systems, which can inform the development of more effective and efficient security tools. It also highlights the importance of careful planning and execution in these systems to avoid biases and errors. Source: https://arxiv.org/abs/2608.21423