AI

Towards Safer RAG: Only Agents Capable of System 2 Thinking may Access Untrusted Documents

Researchers propose a new security principle for Retrieval-Augmented Generation (RAG) systems, which rely on large language models to access untrusted documents. The current approach, known as the Cordon Principle, isolates models from raw evidence but can be computationally expensive. The proposed principle allows only agents capable of System 2 thinking - a type of deliberative reasoning - to access untrusted documents. To evaluate this principle, the authors introduce new
Researchers propose a new security principle for Retrieval-Augmented Generation (RAG) systems, which rely on large language models to access untrusted documents. The current approach, known as the Cordon Principle, isolates models from raw evidence but can be computationally expensive. The proposed principle allows only agents capable of System 2 thinking - a type of deliberative reasoning - to access untrusted documents. To evaluate this principle, the authors introduce new metrics and compare state-of-the-art reasoning language models with standard language models. They find that reasoning-capable models are more robust to corrupted evidence without requiring strict isolation. --- Why it matters: This matters because RAG systems are vulnerable to knowledge-poisoning attacks, which can have significant consequences in applications like question-answering and text generation. The proposed principle could lead to more secure and efficient RAG system design. Source: https://arxiv.org/abs/2608.17153

This article was originally published at: https://arxiv.org/abs/2608.17153