Towards Query-Agnostic RAG Evaluation via Query Coverage and Claim Verifiability
Researchers propose a new evaluation framework called Q-CARE for Retrieval-Augmented Generation (RAG) models. The framework assesses the performance of RAG models by decomposing user queries into sub-queries and answers into atomic claims. This allows for fine-grained diagnostics across different types of user queries, from fact-seeking to explanatory requests. Q-CARE establishes a unified evaluation principle based on query coverage and claim verifiability, providing metrics
Researchers propose a new evaluation framework called Q-CARE for Retrieval-Augmented Generation (RAG) models. The framework assesses the performance of RAG models by decomposing user queries into sub-queries and answers into atomic claims. This allows for fine-grained diagnostics across different types of user queries, from fact-seeking to explanatory requests. Q-CARE establishes a unified evaluation principle based on query coverage and claim verifiability, providing metrics such as C-Prec@k and Completeness. The framework is shown to be more effective than existing RAG evaluation metrics in correlating with human judgments.
---
Why it matters: This matters because it provides a reliable and automated way to evaluate the performance of Retrieval-Augmented Generation models, which are widely used for tasks such as question answering and text generation.
Source: https://arxiv.org/abs/2608.11238
This article was originally published at: https://arxiv.org/abs/2608.11238