Researchers have audited several defenses against 'jailbreak attacks' on locally deployed Large Language Models (LLMs). These attacks exploit weaknesses in the models to produce output that is not intended by the user. The study found that some defenses, such as SmoothLLM and Erase-and-Check, provide formal guarantees but still fail under certain conditions. Others, like Semantic Smoothing, rely on empirical detection results which also failed in many cases. The researchers identified specific assumptions underlying each defense mechanism and tested them using a large dataset of 13,800 evaluation records. --- Why it matters: This research is important for engineers working on LLMs because it highlights the need to re-evaluate existing defenses against jailbreak attacks. The study's findings suggest that current defenses are not robust enough and may fail under specific conditions, which could have significant implications for the safety of these models in real-world applications. Source: https://arxiv.org/abs/2608.21895