Researchers have discovered that vision-language models (VLMs) designed to balance safe generation behavior with grounded visual reasoning may prioritize safety over providing accurate answers. Under certain instructions, these models often abstain from answering questions that they can still answer correctly, raising concerns about the trade-off between safety and accuracy. The study investigates the internal workings of VLMs when faced with safety-constrained instructions and finds that while the models may refuse to provide an answer, their internal representations are still influenced by visual evidence. This suggests that perceptual grounding is preserved despite the refusal behavior. The findings have implications for the development of VLMs, highlighting a potential failure mode where safety alignment can override grounded visual expression. --- Why it matters: This matters because it highlights a trade-off between safety and accuracy in vision-language models, which are increasingly used in real-world applications such as image captioning and question answering. Understanding this phenomenon is crucial for developing more robust and reliable VLMs that balance safety with performance. Source: https://arxiv.org/abs/2608.18628