Real-time speech-to-speech translation

Real-time communication is an integral part of both our professional and personal lives. When speaking to people remotely across language barriers, it can be difficult to truly connect by just relying on state-of-the-art translated captions, as they lack personality and real-time responsiveness essential for fluid conversation. The arrival of speech-to-speech translation (S2ST) bridges this gap by directly generating translated audio, leading to more natural communication. Existing speech-to-speech translation systems often incur significant delays (4–5s), tend to accumulate errors, and typically lack personalization.

Today we describe an innovative end-to-end S2ST model that overcomes these limitations, enabling live translation in the original speaker's voice with only 2 second delay. The novel architecture leverages a streaming framework and, with training on time-synchronized data, significantly reduces the delay between the original input and the translated speech. To support a breadth of languages, we introduce a scalable time-synced data acquisition pipeline that allows us to gradually expand the system to include more languages. This technology has demonstrated its effectiveness through successful deployment in real-time sensitive use cases.

Cascaded S2ST

Prior real-time speech-to-speech technologies employed a cascaded pipeline of individual processing blocks:

1. Firstly, the source audio is transcribed to text using automatic speech recognition (ASR) AI models.
2. Next, the transcribed text is translated word-for-word to the target language using automatic speech translation (AST).
3. Finally, the translated text is converted back to audio using text-to-speech pipelines (TTS).

Despite the high quality of the individual cascade components, achieving a seamless, real-time S2ST experience has been challenging due to three primary factors:

A novel end-to-end, personalized S2ST

An example of our personalized S2ST applied to a Spanish original translated to English.