ContractScrub is a benchmark designed to evaluate the ability of large language models (LLMs) to review and correct errors in legal contracts. The benchmark consists of hand-crafted contracts with diverse error categories, such as misuse of defined terms or incorrect references. Despite strong performance on related general benchmarks, frontier LLMs perform surprisingly poorly on ContractScrub, highlighting the practical limits of current models and the need for domain-specific benchmarks to measure real-world impact. --- Why it matters: This matters because it shows that even with strong performance on general tasks, LLMs may not be suitable for specific domains like contract review. Engineers working on AI will need to consider the limitations of their models when applying them to real-world problems. Source: https://arxiv.org/abs/2608.20204