Researchers propose a framework called SCOPE for calibrating large language models (LLMs) used in pairwise evaluation. The goal is to reduce miscalibration and biases by setting an acceptance threshold based on user-specified error rates. To achieve this, the authors introduce Bidirectional Preference Entropy (BPE), which provides a bias-neutral uncertainty signal. BPE outperforms standard confidence proxies in calibration and discrimination across various benchmarks. SCOPE consistently meets target risk bounds while accepting up to 2.4 times more judgments compared to vanilla baselines. --- Why it matters: This matters because it addresses a significant challenge in AI research: calibrating large language models for pairwise evaluation. By providing a reliable and high-coverage framework, SCOPE enables researchers to make more accurate comparisons between models, which is crucial for advancing the field. Source: https://arxiv.org/abs/2602.13110