SyncAI.news, a Varaisys broadcasting
Where to Focus: Query-Modulated Multimodal Keyframe Selection for Long Video Understanding
SW

Shaoguang Wang, Weiyu Guo, Ziyang Chen, Xuming Hu, Hui Xiong

· 1 min read

ResearcharXiv cs.CV

Where to Focus: Query-Modulated Multimodal Keyframe Selection for Long Video Understanding

arXiv:2604.17422v2 Announce Type: replace Abstract: Long video understanding remains a formidable challenge for Multimodal Large Language Models (MLLMs) due to the prohibitive cost of processing dense frame sequences. Prevailing keyframe-selection methods rely on either a single visual-centric metric (e.g., CLIP similarity) or a static fusion of heuristic scores. This "one-size-fits-all" paradigm frequently fails: visual-only metrics are ineffective for plot-driven narrative queries, while indiscriminately adding textual scores introduces severe "modal noise" for purely visual tasks. To break this bottleneck, we propose Q-Gate, a plug-and-play, training-free framework that treats keyframe selection as a dynamic modality routing problem. We decouple retrieval into three lightweight expert streams: Visual Grounding for local details, Global Matching for scene semantics, and Contextual Alignment for subtitle-driven narratives. Crucially, Q-Gate introduces a Query-Modulated Gating Mechanism that uses the in-context reasoning of an LLM to assess query intent and dynamically allocate weights across the experts, activating necessary modalities while "muting" irrelevant ones to maximize the signal-to-noise ratio. Extensive experiments on LongVideoBench and Video-MME across multiple MLLM backbones show that Q-Gate outperforms representative keyframe-selection baselines in most settings, with particularly strong gains on long and medium videos, providing a robust and interpretable solution for scalable video reasoning.

Original source

This story was published by arXiv cs.CV and written by Shaoguang Wang, Weiyu Guo, Ziyang Chen, Xuming Hu, Hui Xiong. SyncAI.news shows a preview; the complete article is on the publisher's site.

Read the full story on arxiv.org

Similar News