In August 2023, Meta revealed an “all-in-one” AI translation model capable of understanding nearly 100 different languages.
Dubbed SeamlessM4T (Massively Multilingual and Multimodal Machine Translation), it is Meta’s attempt to create a “universal translator” similar to the Babel fish in Douglas Adams’ classic sci-fi series. The Hitchhiker’s Guide to the Galaxy.
The team behind the SeamlessM4T tool has now detailed their work in a journal article Naturerevealing that the advanced system offers an all-in-one solution for text-to-text, speech-to-text, speech-to-speech and text-to-speech translations in an impressive and growing range of languages.
Over 400 years of raw audio
SeamlessM4T, which is used, among other things, to automatically dub videos on Facebook and Instagram, currently supports speech-to-speech translation from 101 to 36 languages, speech-to-text translation from 101 to 96 languages, text-to-text translation . -text translation for 96 languages, text-to-speech translation from 96 to 36 languages and automatic speech recognition for 96 languages. This unified approach overcomes the limitations of traditional cascaded systems, which often require separate subsystems for speech recognition, translation and text-to-speech synthesis.
By streamlining these processes, Meta claims that SeamlessM4T outperforms existing models, achieving up to 23% higher Bilingual Evaluation Understudy (BLEU) scores in translation accuracy and demonstrating impressive resilience to background noise and speaker variation. .
To create SeamlessM4T, Meta started with 4 million hours (over 400 years) of multilingual raw audio from a publicly available repository of crawled web data. The team developed SeamlessAlign, a multimodal corpus containing over 470,000 hours of aligned speech, and combined the dataset with cutting-edge machine learning techniques, including Sentence-level Multimodal and Language-Agnostic (SONAR) embeddings. Representations), which allow multilingual and modality-independent coding. for text and speech.
Meta says that by addressing social and ethical challenges through the use of safeguards, SeamlessM4T can be a valuable tool for global communication. These safeguards reduce gender bias – errors in grammatical gender determination – and alleviate the problem of additional toxicity – when offensive words appear in translations but not in the original source.
