AI

From Inference to Adaptation: A Unified Optimal Transport View of Vision Language Model

Researchers from various institutions have proposed a new method for improving the performance of vision-language models (VLMs) in adapting to real-world distribution shifts during inference. The method, called extit{algname}, formulates zero-shot image classification as a cross-modal alignment problem using a Wasserstein Optimal Transport formulation. This approach provides robust pseudo-labels at the sample level for effective adaptation of VLMs. In contrastive learning, f
Researchers from various institutions have proposed a new method for improving the performance of vision-language models (VLMs) in adapting to real-world distribution shifts during inference. The method, called extit{algname}, formulates zero-shot image classification as a cross-modal alignment problem using a Wasserstein Optimal Transport formulation. This approach provides robust pseudo-labels at the sample level for effective adaptation of VLMs. In contrastive learning, fine-grained supervisions are leveraged to explicitly model relationships between individual image-text pairs, leading to accurate inference at the same granularity. The method also unifies the objectives of inference and adaptation by reformulating the InfoNCE loss as a Wasserstein OT formulation. --- Why it matters: This matters because current VLMs struggle with adapting to real-world distribution shifts during inference, leading to suboptimal performance. This new method provides a principled approach to bridging the gap between inference and adaptation objectives, enabling more accurate and robust VLMs. Source: https://arxiv.org/abs/2608.18339

This article was originally published at: https://arxiv.org/abs/2608.18339