Learning What Not to Learn: Adversarial Disentangled Prompt Tuning for Robust Vision-Language Models
Researchers have proposed a new method called ADAPT to improve the robustness of vision-language models. They found that existing methods can actually make these models worse at handling unseen classes. To fix this, they introduced a dual-prompt mechanism where one prompt is trained to be orthogonal to the other in the embedding space. This helps prevent the model from learning 'shortcuts' that don't generalize well. The authors claim their method improves robustness on unsee
Researchers have proposed a new method called ADAPT to improve the robustness of vision-language models. They found that existing methods can actually make these models worse at handling unseen classes. To fix this, they introduced a dual-prompt mechanism where one prompt is trained to be orthogonal to the other in the embedding space. This helps prevent the model from learning 'shortcuts' that don't generalize well. The authors claim their method improves robustness on unseen classes and provide theoretical guarantees for its performance.
---
Why it matters: This matters because vision-language models are widely used in applications like image captioning, visual question answering, and more. Improving their robustness to adversarial examples is crucial for real-world deployment. ADAPT's ability to prevent overfitting and learn robust features can have significant impact on the reliability of these systems.
Source: https://arxiv.org/abs/2608.17306
This article was originally published at: https://arxiv.org/abs/2608.17306