Researchers argue that counterfactual prompting, a method used to evaluate language model bias and faithfulness, is flawed because it doesn't account for baseline 'meaning-preserving' modifications to text. These changes can have the same effect as targeted interventions, making it impossible to attribute observed effects to the intended factor without statistical testing. A new framework proposes comparing differences between target interventions and paraphrasing inputs to robustly measure the effects of targeted interventions. --- Why it matters: This matters because it highlights a potential pitfall in evaluating language model bias and faithfulness, which could lead to incorrect conclusions about model sensitivity. By accounting for general model sensitivity, researchers can gain more accurate insights into the effects of targeted interventions. Source: https://arxiv.org/abs/2605.01048