On multimodal speech-text pre-trained models

Multimodal pre-training is a potential game changer in spoken language processing. In this blog, we review 3 recent papers on the topic by Meta (Data2Vec), Microsoft and academic partners (SpeechT5) and Google (mSLAM), and discuss how these multimodal speech-text pre-trained models are used to build more holistic speech systems and what’s required to extend their capabilities and make them more accessible.

Brief background on self-supervised learning for both speech and text

Self supervised pre-training has been successful for both speech and text. In speech, the models are pre-trained through resolving pseudo-tasks such as contrastive predictive coding (CPC) which distinguishes a future speech frame from distractor samples using latent representations from a convolutional architecture (1). In 2020 the same group introduced wav2vec2.0 (2) which is based on the CPC idea, but where discrete speech units are used as latent representations which are fed to a transformer architecture to build contextualized representations. In 2021, HuBERT (3) kept a similar encoder but leveraged cross-entropy loss (like BERT (4)) instead of contrastive loss and scaled to 1 billion parameters. Both wav2vec2.0 and HuBERT are now considered essential to extract good representations and/or to initialize speech encoders for tasks such as automatic speech recognition (ASR). 

Until very recently, pre-trained models in speech were encoder-only models. However, for text, encoder-decoder models such as BART (5) and T5 (6) were introduced in 2020. These  transformer-based sequence-to-sequence models basically consist of an encoder and an autoregressive decoder. They are trained by reconstructing (i.e. denoising) the original text from a noisy version corrupted with a set of noising functions. Their multilingual counterparts, mBART (7) and mT5 (8), appeared shortly afterwards.

1: Data2Vec, a ‘modality agnostic’ extension of encoder-only models

Data2Vec (9) by Meta proposes a unique training procedure to learn transformer models from either speech, images or textual data. The authors highlight the fact that, while the concept of self-supervised learning is now common to various modalities, the actual algorithms and objectives are widely different and often modality dependent. They believe that Data2Vec is a first step towards a general self-supervised learning paradigm.  

Data2Vec uses the concept of masked predictions; but instead of doing it on modality dependent symbols (subword units for text, visual tokens for images or abstract speech units) the model tries to predict contextualized latent representations. As shown, in Figure 1, the model is trained in a teacher/student loop (teacher and student are actually the same model at different steps in the training time). At each step, teacher and student take as input the same data (speech, image or text): without a mask for the teacher and with a mask for the student (masking here is done using classical masking methods specific to each modality). The goal of the student is to predict the teacher’s representations on these masked inputs. This regression task is learned on an average of K layers from the teacher (which seems to work as well as predicting individual layers while being more efficient in terms of computation). This training procedure, unique for the three modalities, shows impressive results on various downstream tasks for images, speech and text. For example, if we focus on speech and text, it performs better than prior work on a) wav2vec2.0 and HuBERT for ASR on Librispeech (10) and b) RoBERTa (11) for the tasks defined in GLUE.

Key takeaways: Data2Vec is not a multimodal model in the sense that several modalities are mixed together in the model, but it unifies the self-supervised learning process by making it ‘data agnostic’ and it utilizes contextualized input representations and makes predictions from it. The code is available at this link.

2: SpeechT5, a multimodal extension of encoder-decoder models

SpeechT5 (12) is a multimodal extension of transformer encoder-decoder models which can encode or decode both speech and text in a single model. One can easily imagine how such a pre-trained model could be used to initialize ASR (speech-to-text), TTS (text-to-speech), Voice Conversion (VC – speech-to-speech) or any task that could take advantage of a pre-trained speech/text encoder-decoder. A key aspect of this model is that it maps both acoustic and text information in a shared quantized vector space (à la wav2vec2.0).

More precisely, SpeechT5 is made up of a single encoder/decoder module and several modality-specific pre-post nets. Nothing special to mention for text pre/post nets but for the speech modality the encoder pre-net re-uses the convolutional feature extractor of wav2vec2.0 while for the decoder, the pre-net uses log-Mel filterbank features fed into a fully connected network with ReLU activation (and a speaker embedding is concatenated to the output of the speech-decoder pre-net to support voice conversion and multi-speaker TTS tasks). On its side, the speech decoder post network predicts log-Mel filterbanks (so an external vocoder will be needed to generate a speech output).

A composite pre-training loss is optimized using unlabeled speech and text data which includes: (a) a masked language modeling (MLM) loss on discrete latent speech representations (à la HuBERT); (b) a speech reconstruction L1 loss (the reason for adding this loss component is unclear and no ablation study supports its usefulness); (c) a cross-entropy loss specific to the stop token when speech is decoded; (d) a text denoising objective (à la BART); and (e) a cross-modal objective to better align speech and text representations (it is unclear, from the paper whether or not  this specific loss needs aligned speech-text data to be computed).

A medium-size encoder-decoder model (12 encoder blocks, 6 decoder blocks, model dimension 768, inner dimension 3072) is trained using LibriSpeech for speech pre-training and LibriLM for text pre-training. Authors experiment on several speech tasks including ASR, VC, TTS, and speaker identification. SpeechT5 pre-training leads to promising results for the tasks addressed.

Key takeaways: SpeechT5 is a real multimodal model that can both encode and generate speech and text. This single model can be used to initialize task-specific architectures (AST, TTS, speech translation) as opposed to modality-specific initializations made in the past. At the time of writing this blog, the code of SpeechT5 was not available (but you should probably keep an eye on this repo).

3: From SLAM to mSLAM, a multilingual extension of speech-text joint pre-training

In the SLAM architecture (13), speech and text are unified in a common encoder model. The multimodal encoder is made up of convolution-augmented transformer (conformer) blocks that were introduced earlier for ASR by (14). As input it takes either speech, text, or concatenated speech-text. The text embedding layer and a speech-embedding layer (a convnet followed by conformer layers) produce latent representations to the multimodal encoder. Speech-text pre-training is a mix of self-supervised learning objectives (not much different to the ones mentioned for SpeechT5) and supervised cross-modal learning objectives which leverage aligned speech-text pairs. Among the joint speech-text objectives, translation language modeling (TLM) predicts masked text or speech spans from a concatenated speech-text input to encourage the use of cross-modal context. Speech-text matching (STM) is a contrastive task which predicts whether a pair of speech and text is positive (matched) or negative (not matched). Downstream evaluations on speech and language tasks (ASR, Speech Translation, GLUE tasks) are encouraging but also highlight significant interference challenges when pre-training simultaneously on medium-resource modality (speech) and high-resource modality (text). For instance, the authors observe a significant decrease in performance on GLUE when adding the speech modality and hypothesize that the capacity of the model is a limiting factor for understanding both modalities simultaneously.

A massively multilingual extension of this model, mSLAM (15), extends previous work by pre-training on large amounts of unlabeled speech and text in multiple languages (51 languages for speech and 101 languages for text). The pre-training objectives are simplified: speech-text matching (STM) contrastive learning is skipped while the TLM cross-modal objective is kept; a Connectionist Temporal Classification (CTC) loss is also applied on the speech part of concatenated speech-text using character transcript as a target in order to learn better speech-text alignments. Another difference on the text part is that sentence-piece subword units are replaced by characters for better multilingual coverage (4096 tokens spanning 101 languages).

After pre-training, downstream task experiments are conducted on multilingual speech translation, multilingual ASR, spoken language identification, spoken intent classification and text classification. Experiments show that speech-text pre-training is better than speech-only pre-training for multilingual ASR and translation, suggesting that using text during pre-training makes encoders more flexible to downstream tasks that make use of text. Preliminary experiments also confirm the effect of cross-modal losses to align speech and text as zero-shot cross-modal translation seems possible: an mSLAM model fine-tuned for speech translation is evaluated in a zero-shot setting using a text input modality and displays BLEU>5 for half of the language pairs evaluated.

Key takeaways: mSLAM is a multilingual and multimodal pretrained model representing speech and text in a shared space. At the moment it is encoder only and at the time of writing this blog, the code of mSLAM is not yet available. This model displays promising zero shot cross-modal properties, such as zero shot text translation from a fine-tuned speech translation model.

What’s next ?

Training the models described here requires access to powerful computing platforms i.e. our estimation of the time to train a Data2Vec model on LibriSpeech (1K hours) with 4 GPUs is >15 days. Simpler and more efficient pre-training approaches would be needed to democratize access and make these multimodal models more widely adopted. As these models become more universal and multilingual, standardization in the evaluation process for comprehensive comparisons of these architectures is also needed. Moreover, their zero-shot capabilities have only recently been investigated and we can expect more related work to appear in 2022 on transferring speech translation to low resource languages by exploiting their acoustic or written similarities (for instance a model pre-trained on Hindi speech should transfer well to Urdu as both spoken languages are similar, even if they use a different writing script). More research is also needed on the decoder side, especially to generate expressive speech with adequate prosody, as recently investigated in (16). Finally, multilingual decoder models would be also of interest to provide a pre-trained backbone for massively multilingual speech-to-speech translation. 

Interested in these topics?

We have internships and open positions at NAVER LABS Europe – see https://europe.naverlabs.com/careers/ 

References

1. wav2vec: Unsupervised Pre-Training for Speech Recognition. Steffen Schneider, Alexei Baevski, Ronan Collobert and Michael Auli. In Proceedings of Interspeech 2019, pp. 3465–3469, 2019.
2. wav2vec2.0: A framework for self-supervised learning of speech representations. Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed and Michael Auli. In Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
3. HuBERT: Self-supervised speech representation learning by masked prediction of hidden units. Hsu, W., Bolte, B., Tsai, Y. H., Lakhotia, K., Salakhutdinov, R., and Mohamed, A. (2021). IEEE ACM Transactions on Audio Speech Language Processing, 29:3451–3460.
4. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL) 2019.
5. BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, Luke Zettlemoyer. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL) 2020.
6. Exploring the limits of transfer learning with a unified text-to- text transformer. Colin Raffel, Noam Shazeer, Adam Roberts, Kather- ine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Journal of Machine Learning Research (JMLR), 21(140):1–67.
7. Multilingual Denoising Pre-training for Neural Machine Translation. Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, Luke Zettlemoyer. Transactions of the Association for Computational Linguistics (TACL), 2020.
8. mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer. Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, Colin Raffel. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), 2021.
9. Data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language. Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, Michael Auli. arXiv 2202.03555, 2022.
10. Librispeech: An ASR corpus based on public domain audio books. Vassil Panayotov, Guoguo Chen, Daniel Povey, Sanjeev Khudanpur. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2015.
11. RoBERTa: A Robustly Optimized BERT Pretraining Approach. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, Veselin Stoyanov. arXiv:1907.11692, 2019.
12. SpeechT5: Unified-Modal Encoder-Decoder Pre-training for Spoken Language Processing. Junyi Ao, Rui Wang, Long Zhou, Shujie Liu, Shuo Ren, Yu Wu, Tom Ko, Qing Li, Yu Zhang, Zhihua Wei, Yao Qian, Jinyu Li, Furu Wei. arXiv 2110.07205, 2021.
13. SLAM: A Unified Encoder for Speech and Language Modeling via Speech-Text Joint Pre-Training. Ankur Bapna, Yu-an Chung, Nan Wu, Anmol Gulati, Ye Jia, Jonathan H. Clark, Melvin Johnson, Jason Riesa, Alexis Conneau, Yu Zhang. arXiv 211010329, 2021.
14. Conformer: Convolution-augmented Transformer for Speech Recognition. Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, Ruoming Pang. arXiv 2005.08100, 2020.
15. mSLAM: Massively multilingual joint pre-training for speech and text. Ankur Bapna, Colin Cherry, Yu Zhang, Ye Jia, Melvin Johnson, Yong Cheng, Simran Khanuja, Jason Riesa, Alexis Conneau. arXiv 2202.01374, 2022.
16. Text-Free Prosody-Aware Generative Spoken Language Modeling. Eugene Kharitonov, Ann Lee, Adam Polyak, Yossi Adi, Jade Copet, Kushal Lakhotia, Tu-Anh Nguyen, Morgane Rivière, Abdelrahman Mohamed, Emmanuel Dupoux, Wei-Ning Hsu. arXiv 2109.03264, 2021.