SyncAI.news, a Varaisys broadcasting
Seeing Speech: Learning Visible Articulatory Dynamics for Speech-Driven 3D Facial Animation
HK

Hyung Kyu Kim, Byungchan Hwang, Hak Gu Kim

· 1 min read

ResearcharXiv cs.LG

Seeing Speech: Learning Visible Articulatory Dynamics for Speech-Driven 3D Facial Animation

arXiv:2609.30517v1 Announce Type: cross Abstract: Recent progress in speech-driven 3D facial animation has improved vertex-level reconstruction quality, but speech-consistent visible articulation remains difficult. This is because speech production follows structured and constrained articulators' coordination and the mapping from acoustics to motion is inherently one-to-many. Motivated by the structured patterns of visible articulation, we propose a novel articulation-aware framework that models visible speech through directional articulatory motions and composes them into surface-consistent 3D facial motion. To represent visible articulation with three directional articulatory motions, spreading, opening, and protrusion, we propose a Speech--Articulatory Memory (SAM) that captures the correspondence between speech and these motions under phonetic context through retrieval and decoding based on a key-value memory structure. Then, a Topology-aware Articulatory Composition (TAC) integrates the predicted directional articulatory motions under mesh topology to produce surface-consistent 3D facial motion. Experiments on VOCASET and TFHP show that our method achieves state-of-the-art performance on standard reconstruction metrics and improves visible articulatory distance and velocity errors for lip articulation, while a user study confirms clear preference in lip sync and realism.

Original source

This story was published by arXiv cs.LG and written by Hyung Kyu Kim, Byungchan Hwang, Hak Gu Kim. SyncAI.news shows a preview; the complete article is on the publisher's site.

Read the full story on arxiv.org

Similar News