
ZX
Zijie Xin, Tianyi Wang, Daoze Zhang, Jie Yang, Changli Tang, Qijie Wei, Fengyun Rao, Xirong Li
· 1 min read
ResearcharXiv cs.CV
Can Omni-modal LLMs Understand Spoken Intentions?
arXiv:2611.24706v1 Announce Type: new
Abstract: Omni-modal large language models (omni-modal LLMs) can respond to spoken requests directly through voice and video calls. Such requests rely on the scene shared by the user and the assistant, so the same utterance may call for different help or for no response. Most audio-visual benchmarks provide questions as separate text input, and interactive benchmarks mostly score a response as a whole. It thus remains unclear whether a model recognizes when the user is asking for help, and whether its reply meets the user's need once the request is recognized. We propose O-IN-Bench to evaluate how well omni-modal LLMs understand spoken intentions. A model watches a complete video with native speech and outputs one text response, or [SILENCE] when no response is needed. O-IN-Bench contains 1,000 Chinese and English samples in five categories and 16 subcategories. Contrastive pairs combine the same request with different scene conditions to test whether the response changes with the context. A sample-specific rubric judges language, intent, and content separately, and O-IN-Engine verifies each rubric against the video. Across 30 omni-modal model configurations, the best overall score is 59.6, achieved by Gemini3.7-Flash. Gemini3.7-Flash passes 74.4% of the intent judgments but only 42.6% of the content judgments. When the speech is a conversation between peers rather than a request to the assistant, even strong models often keep responding. When a reply must satisfy several people, the best score is only 25.9. These results indicate that recognizing a request and meeting its requirements are separate abilities, and O-IN-Bench measures each of them.
Original source
This story was published by arXiv cs.CV and written by Zijie Xin, Tianyi Wang, Daoze Zhang, Jie Yang, Changli Tang, Qijie Wei, Fengyun Rao, Xirong Li. SyncAI.news shows a preview; the complete article is on the publisher's site.
Read the full story on arxiv.org


