FVSpec: Real-World Property-Based Tests as Lean Challenges
Researchers have developed a benchmark for evaluating AI models on real-world formal software verification tasks. They scraped 11,039 property-based tests from Python repositories and translated 2,772 of them into Lean 4 specifications. This is challenging due to the need to model Python semantics in Lean and infer logical properties encoded in imperative PBTs. A three-agent LLM pipeline was used for translation, and baselines were established for proof generation using autom
Researchers have developed a benchmark for evaluating AI models on real-world formal software verification tasks. They scraped 11,039 property-based tests from Python repositories and translated 2,772 of them into Lean 4 specifications. This is challenging due to the need to model Python semantics in Lean and infer logical properties encoded in imperative PBTs. A three-agent LLM pipeline was used for translation, and baselines were established for proof generation using automated and model-based approaches. The benchmark aims to drive progress on AI-assisted formal verification of real-world software.
---
Why it matters: This matters because it provides a way to evaluate the ability of AI models to assist in formal verification of software, which is increasingly important as more code is written by AI. This could help improve the reliability and security of AI-generated code.
Source: https://arxiv.org/abs/2606.01008
This article was originally published at: https://arxiv.org/abs/2606.01008