Evidence-Consistent Generative Detection under Scenario-Level Distribution Shift
Researchers propose a new framework called ECoG for detecting out-of-distribution (OOD) scenarios in social-engineering fraud detection. They found that models can overestimate their robustness when trained and tested on similar patterns or cues. To address this, they developed an evidence-consistent generative framework that combines two objectives: evidence-span supervision and rationale-label consistency. The results show improved performance on OOD challenging instances,
Researchers propose a new framework called ECoG for detecting out-of-distribution (OOD) scenarios in social-engineering fraud detection. They found that models can overestimate their robustness when trained and tested on similar patterns or cues. To address this, they developed an evidence-consistent generative framework that combines two objectives: evidence-span supervision and rationale-label consistency. The results show improved performance on OOD challenging instances, with a 3.22-point increase in Macro-F1 score and a reduction in prediction-rationale inconsistency by 4.22 points.
---
Why it matters: This matters to engineers because it shows that current models can be misled by familiar patterns rather than decision-relevant evidence, leading to overestimation of robustness. The proposed ECoG framework provides a new approach for improving OOD detection in social-engineering scenarios.
Source: https://arxiv.org/abs/2608.21043
This article was originally published at: https://arxiv.org/abs/2608.21043