Highlights of Interspeech 2022

Interspeech 2022 took place in Incheon (Korea) from the 18th to the 22nd September. This article covers the papers I was most interested in based on my current research interests in self-supervised speech models, scaling speech-to-text models, large language models and ASR architectures and end-to-end speech to text translation.

The first thing that marked this year’s conference was the number of papers on self-supervised speech (SSL) models such as XLS-R (a multilingual wav2vec2.0 model that covers 128 languages). The XLS-R paper by Babu et al. (1) may have been presented at Interspeech but the model itself has been widely adopted for several months by the speech research community! Other SSL models that caught my attention were MAESTRO (2), a speech-text encoder model that learns aligned representations from raw speech, raw text or paired speech-text. MAESTRO is an additional member to the existing family of multimodal speech-text SSL models described in this earlier blog. The key aspect of MAESTRO is that it trains unified representations through sequence alignment between speech and text (using TTS-like duration prediction) to match embeddings in the learned space. Secondly, not only pre-training encoders was addressed but also the pre-training of transformer decoders for speech-to-text tasks. This was the key feature of Speech2C model (3), (by the authors of SpeechT5) which could be seen as a BART-like encoder-decoder pre-trained model for speech tasks (this other piece of work by Arunkumar & Umesh (4) is similar in spirit). Speech2C means « speech-to-code » and it pre-trains an encoder-decoder model using only speech data. It does so by « transcribing » speech into discrete units such as the ones that can be extracted with HuBERT. The decoder of Speech2C then learns to auto-regressively reconstruct reduced units (similar consecutive units collapsed into a single one) instead of generating real text transcription. According to the authors’ observations, this works because a certain amount of units are correlated with the text transcripts so the decoder can start learning how to make text prediction during this pre-training framework. The authors have released their code here. Some attempts to compress these SSL models (for low memory footprint ASR) were also presented during the conference. An example is this one by authors at NTT Corp. (5) which compares different student architectures (shallow with large embedding sizes versus deep with small embedding sizes) but it’s worth checking out if there are more in the proceedings. Using multiple SSL models through ensembling would also appear to be beneficial for performance, as shown by Arunkunar et al. (again!) in this work (6) where HuBERT, Wav2Vec2.0, and WavLM representations are used all together to improve ASR performance. It’s also worth pointing out, to my knowledge, the first attempt to apply data selection for SSL (7) (using unsupervised discrete acoustic units) by Lu et al. at Google. Analyses of these pre-trained models, and if they could convey some biases, were also out this year (including my own work on the analysis of gender impact in SSL models (8) for ASR and speech translation systems), presented during a special session on inclusive and fair speech technologies). 

Another important line of research was the attempt to scale up very large speech-to-text models. It was notably demonstrated, and asserted by the Meta AI authors in the title of this paper that scaling ASR improves zero and few shot learning (9). A « universal English ASR model » with 10B parameters was trained using 4.5 million hours of English speech from 10 different sources. We’re talking about supervised or semi-supervised (using pseudo-labeling) ASR here. It’s shown that the model generalizes well to novel domains and styles of speech. Interesting work but unfortunately the authors’ mentioned during the poster session that this model will NOT be open-sourced due to some subset of training data under non-permissive licensing. Interestingly, a few days before this paper was presented, the Whisper (10) model from OpenAI was announced. Similar in spirit, it’s an ASR system trained on 680,000 hours of multilingual supervised or weakly-supervised data collected from the Web. A multitask training framework is also implemented (not only a multilingual ASR task but also a language identification task and speech-to-text translation into English). It shows that scaling and augmenting data diversity improves ASR robustness to accents, background noise and technical language. It also allows transcription in multiple languages and translation from those languages into English. In the case of Whisper, the models and inference code are open-sourced and there are already countless comments on Twitter discussing its use on diverse self-recorded speeches: meetings, seminars, courses, recorded TV…! Note that there were also other (smaller scale) papers at Interspeech 2022 attempting multi-task training of speech-to-text systems (such as this one by Kons et al. (11) addressing joint ASR, speech emotion recognition and language classification with a single model).

One aspect of ASR that generated a lot of attention again this year is how to properly integrate a large language model (LLM) into ASR architectures (as an alternative to shallow fusion, LM re-scoring or synthesizing large amounts of raw text to boost the internal ASR language model). This question was actually already asked 5 years ago in this paper by Gulcehre et al. (12) that last week received the best research paper award of the Computer Speech and Language journal. In the following lines I attempt to structure what I heard and saw on this topic but it’s going in multiple directions so I can but provide a partial view.  First of all, LLM re-scoring is still a popular approach: this empirical study by Udagawa et al. at IBM (13) evaluates its impact on very competitive ASR systems along multiple axes: unidirectional versus bidirectional LLMs; use or not of context (context augmentation); in-domain fine-tuning of the LLM. It was recently shown that to fuse an external LLM in a principled way, the log-likelihood produced by the internal LM (ILM) must be properly estimated and subtracted from the overall combined score: this paper by Liu et al. (14) addresses this ILM log-likelihood estimation and shows that it’s worth investigating to boost ASR performance. On the other hand, some papers try to inject knowledge of some domain specific data in the internal LM itself (to get rid of the use of an external LM). For instance this paper by Meng et al. at Microsoft (15), (which by the way presents a good survey on LM integration in ASR in its introduction), proposes to fine tune the internal LM (i.e. E2E components excluding the encoder) using in-domain data. The advantage of such an approach is that it enables a text-only adaptation of the ASR architecture without increasing the inference cost. In a similar fashion (but for a different ASR architecture) this non-IS2022 paper by Futami et al. at Kyoto University (16), aims at distilling the knowledge of BERT for CTC-based ASR. Last but not least, LM prompting is also becoming popular in speech processing (even though it is more frequent in NLP). I spotted this work by Dingliwal et al. at Amazon (17) entitled « Domain Prompts: Towards memory and compute efficient domain adaptation of ASR systems » which proposes training a small number of domain embedding parameters to bias the predictions of an external language model (such as GPT-2 or GPT-3) to a particular domain (this is rather similar to the use of adapter modules, popular in NLP and now progressively adopted in speech); a similar idea is introduced by the authors in this non Interspeech2022 paper (18).To conclude on this topic, it’s worth mentioning that a natural extension of all these works is to use context (not only the current sentence) to improve LLM estimation (and thus ASR performance). Actually, a lot of work in TTS also addresses the same idea, notably this one, by Stephenson et al. (19) and this one from Yoon et al. at NAVER and LINE (20) that aims at paragraph-level generation of emotional speech). Talking about context-aware ASR, in addition to the empirical study already mentioned above (13), I found this work to be relevant by Hou et al. at ByteDance Inc.(21): they explore how to use dialogue-context in a neural transducer ASR model through different training strategies. They obtain tiny gains on the Switchboard-2000h task. Context-aware ASR is interesting but to address it more deeply, dedicated datasets would be needed and current word error rate (WER) is probably ill-equipped to properly capture improvements made by context-aware systems. 

On the topic of datasets, several interesting papers were presented at the conference. I was specifically interested by UserLibri (22), an enriched version of the LibriSpeech corpus which aims at evaluating ASR personalization using raw text: LibriSpeech data is clustered into multiple speakers (simulating multiple users) and for each one, personalization text data (the actual full text book corresponding to the utterances read by a given speaker) is provided (to simulate pre-existing user specific text data). Baseline implementations of personalized ASR (taking advantage of the user-specific text) are presented in this work which has the merit of providing the first dataset for personalized ASR. The results of multiple challenges were also displayed during the conference: I was attracted by the VoiceMOS challenge (23) which aims at automatically predicting the MOS score from a TTS system’s output. In the same spirit, but for a different goal, the challenge on speech intelligibility prediction for hearing-impaired listeners also held its special session in Incheon. Learning metrics that correlate with human judgements has been addressed for years in machine translation and it’s nice to see it spread to the speech community. Another challenge that caught my attention was the audio deep packet loss concealment challenge (24) which aims at addressing situations where audio/speech packet transmission breaks down in real time communication settings. This problem is not new: VoIP has been around for more than two decades! Here the organizers revisit it through the lens of modern neural ASR architectures (but unfortunately fail at crediting pioneering work done 20 years ago such as this 2002 paper from Mayorga et al. (25) (Mayorga was one of my first PhD students!)

Following is a heterogenous list of papers I could not put under a common banner. First, I have a personal interest for end-to-end speech-to-text translation (ST) for which traditional approaches use attention-based encoder-decoder models. But this work by Xue et al. (26) attempted to adapt the popular neural transducer ASR architecture (for which training efficiency is still a hot topic, as illustrated by this ASR paper by Zhou et al.(27) and this other one from NAVER (28), to the speech translation task and allows streaming the speech-to-text task. In ST, a lot of work attempts at bridging the gap between speech output from the encoder and text input to the decoder as done in M-adapter by Zhao et al.(29) which leverages adapter modules. End-to-end speech-to-speech translation is also addressed in this work by Popuri et al.(30) at Meta AI that tries to overcome the challenge to train such a complex task end-to-end. Secondly, there is a renewed interest in rich transcription (ASR that outputs cased and punctuated transcripts) for instance by leveraging prosody (Cho et al.(31)) or by using a multitask framework for end-to-end speech-to-punctuated-text recognition (Nozaki et al.(32)) Producing a well capitalized and punctuated transcript is key for further downstream tasks such as natural language understanding. Actually, even if end-to-end methods for spoken language understanding (SLU) are appealing, cascade approaches are still widely used (notably in industrial settings) and scientists always try to improve SLU robustness to ASR errors (Feng et al. (33)) using ASR error correction techniques or by simulating a noisy training scenario.