
JJ
Jihoo Jung, Ji-Hoon Kim, Suyeon Lee, Gia Thinh Pham, Joon Son Chung
· 1 min read
ResearcharXiv cs.CV
Beyond Co-Generation: Rethinking Jointness in Joint Audio-Video Generation
arXiv:2611.22857v1 Announce Type: new
Abstract: Our aim is to advance the understanding of genuine audio-visual coordination in joint text-to-audio-video (T2AV) generation, with particular emphasis on semantic consistency. Evaluating audio-visual consistency in T2AV generation is nontrivial, as detailed text prompts obscure whether such consistency stems from genuine cross-modal coordination or merely from independent text following. To isolate the former, we introduce intrinsic jointness: the ability of audio and video to produce the same realization for details left underspecified by the prompt. Using semantically underspecified prompts (e.g., "musical instrument playing" rather than "electric guitar playing"), we investigate four recent models and reveal their limited intrinsic jointness. To address this limitation, we propose JointREPA, a simple representation alignment-based post-training method to facilitate cross-modal coordination. Since existing benchmarks use semantically detailed prompts, we introduce UNSAIDBench, a benchmark designed to evaluate intrinsic jointness under semantic underspecification. Experiments across two distinct models and three benchmarks, including UNSAIDBench, demonstrate that JointREPA consistently improves semantic consistency.
Original source
This story was published by arXiv cs.CV and written by Jihoo Jung, Ji-Hoon Kim, Suyeon Lee, Gia Thinh Pham, Joon Son Chung. SyncAI.news shows a preview; the complete article is on the publisher's site.
Read the full story on arxiv.org


