Researchers have discovered a vulnerability in the way some AI systems handle safety at inference time. When multiple outputs are sampled and filtered by a learned safety model, there's a risk that an 'unsafe' output can be selected even if it passes the safety check. This is because the safety model may not accurately reflect true safety criteria. The researchers have derived bounds on this phenomenon, showing that as more samples are taken, the likelihood of selecting an unsafe output increases, even with small errors in the safety model. They also propose a new approach to mitigate this issue. --- Why it matters: This study is important for engineers working on AI systems that require high levels of safety and reliability, such as those used in autonomous vehicles or healthcare applications. The findings highlight the challenges of scaling up inference-time safety with learned models and provide insights into how to improve the robustness of these systems. Source: https://arxiv.org/abs/2608.22915