PatchGate: Narrowing the Verbalization Gap with Intrinsic Object Inventories in Frozen Vision-Language Models
Researchers have proposed PatchGate, a framework that improves the accuracy of image captioning in Vision-Language Models (VLMs). PatchGate extracts object evidence from the model's internal state and uses it to promote accurate but under-verbalized objects and suppress inaccurate ones. This approach is training-free, requiring only one extra forward pass through the model. In experiments on the AMBER dataset, PatchGate increased visible-object coverage by 13.4% and reduced o
Researchers have proposed PatchGate, a framework that improves the accuracy of image captioning in Vision-Language Models (VLMs). PatchGate extracts object evidence from the model's internal state and uses it to promote accurate but under-verbalized objects and suppress inaccurate ones. This approach is training-free, requiring only one extra forward pass through the model. In experiments on the AMBER dataset, PatchGate increased visible-object coverage by 13.4% and reduced object hallucination by 12.0%. The framework consists of two stages: Visual Evidence eXtraction (VEX) and Visual-Evidence Inclusion-Exclusion Decoding (VIED). VEX extracts patch-level lexical evidence from the model's decoder layers, while VIED uses this evidence to calibrate decoding logits.
---
Why it matters: This matters because it addresses a long-standing challenge in image captioning: ensuring that models accurately describe all visible objects without introducing unsupported or hallucinated objects. PatchGate's approach could lead to more reliable and accurate image captioning systems, which have applications in areas like computer vision, robotics, and autonomous vehicles.
Source: https://arxiv.org/abs/2608.21819
This article was originally published at: https://arxiv.org/abs/2608.21819