SyncAI.news, a Varaisys broadcasting
Semantic-Guided Geometric Decoding for Token-Efficient Visual Grounding
ZC

Zhaojie Chen, Junyan Yang, Le Yang, Qi Dang, Yucheng Huang, Qianyi Che, Zhaoqiang Chen, Xinqi Zhu, Qiang Zhao

· 1 min read

ResearcharXiv cs.CV

Semantic-Guided Geometric Decoding for Token-Efficient Visual Grounding

arXiv:2611.22577v1 Announce Type: new Abstract: Multimodal large language models (MLLMs) provide a unified interface for language-conditioned visual perception. In pixel-to-sequence grounding, autoregressive coordinate generation requires multiple sequential decoding steps per box, increasing inference cost with the number of instances. Pixel-to-embedding methods reduce this overhead but lack explicit spatial encoding when using a shared special token. We introduce SemGeo, a token-efficient visual grounding framework that represents each instance with a single grid token for coarse localization. A geometric decoder reuses visual features and hidden states to recover precise discrete box coordinates in parallel. To train proposal generation and box localization together, we introduce Gaussian soft labels that encode coordinate distances during supervised fine-tuning (SFT). Reinforcement post-training further combines box-set rewards with coordinate supervision on model-generated proposals. This stage trains the language model and geometric decoder on their own predictions to mitigate the mismatch between teacher-forced training and inference. Across eight benchmarks spanning object detection, referring object detection, and document layout grounding, SemGeo achieves the highest F1@mIou among compared MLLMs on six benchmarks. It also delivers $2.07\times$ and $1.25\times$ the mean box-generation throughput of representative grounding baselines under vLLM and Transformers, respectively, while achieving higher accuracy in both comparisons.

Original source

This story was published by arXiv cs.CV and written by Zhaojie Chen, Junyan Yang, Le Yang, Qi Dang, Yucheng Huang, Qianyi Che, Zhaoqiang Chen, Xinqi Zhu, Qiang Zhao. SyncAI.news shows a preview; the complete article is on the publisher's site.

Read the full story on arxiv.org

Similar News