AI

Prompt-Induced Waste in Coding Agents: Reasoning, Effort, Harness Design, and End-to-End Cost

Researchers have found that coding agents' efficiency cannot be measured solely by token count or model price. They conducted experiments to study the relationship between end-to-end cost and task success, considering factors such as prompt semantics, inference effort, harness policy, model, task difficulty, tool use, context management, and provider accounting. The results show that wording in prompts can significantly impact reasoning and verification behavior, while additi
Researchers have found that coding agents' efficiency cannot be measured solely by token count or model price. They conducted experiments to study the relationship between end-to-end cost and task success, considering factors such as prompt semantics, inference effort, harness policy, model, task difficulty, tool use, context management, and provider accounting. The results show that wording in prompts can significantly impact reasoning and verification behavior, while additional inference effort may improve performance for some models but add unnecessary cost. A new efficiency model is proposed to measure success and end-to-end cost while controlling system variables. --- Why it matters: This study matters because it highlights the complexity of optimizing coding agents' efficiency. Understanding how different factors interact can help researchers and developers design more effective systems, reducing waste and improving performance. Source: https://arxiv.org/abs/2608.01347

This article was originally published at: https://arxiv.org/abs/2608.01347