SyncAI.news, a Varaisys broadcasting
Contrast Where It Matters: Visual Evidence-Guided Decoding for Mitigating Hallucinations in LVLMs
YL

Yiheng Li, Wenhao Wang, Xingyi Li, Mingda Jia, Zichang Tan, Yinhang Tang, Zhen Lei, Yang Yang

· 1 min read

ResearcharXiv cs.CV

Contrast Where It Matters: Visual Evidence-Guided Decoding for Mitigating Hallucinations in LVLMs

arXiv:2611.25524v1 Announce Type: new Abstract: Despite strong multimodal capabilities, large vision-language models (LVLMs) often hallucinate when linguistic priors outweigh visual evidence or when predictions are grounded in irrelevant or misleading image regions. Training-free contrastive decoding mitigates hallucinations by contrasting the original prediction with an auxiliary one that exposes linguistic priors. Yet existing methods focus on altering the auxiliary input or computation and do not ensure that either branch uses query-relevant visual evidence during decoding. We propose Visual Evidence-Guided Decoding (VEGD), a training-free framework that identifies query-relevant visual evidence and guides its use throughout contrastive decoding. VEGD locates this evidence by aggregating localization heads with strong visual responses and coherent spatial structure, then selectively weakens it in the auxiliary image. During generation, VEGD directs visual-carrier heads in both branches toward the same evidence location, strengthening access to intact evidence in the main branch while anchoring the auxiliary branch to its weakened counterpart. An adaptive distribution fusion strategy then combines or contrasts their predictions according to distribution divergence. Experiments across diverse LVLMs and benchmarks demonstrate that VEGD consistently reduces hallucinations.

Original source

This story was published by arXiv cs.CV and written by Yiheng Li, Wenhao Wang, Xingyi Li, Mingda Jia, Zichang Tan, Yinhang Tang, Zhen Lei, Yang Yang. SyncAI.news shows a preview; the complete article is on the publisher's site.

Read the full story on arxiv.org

Similar News