We’re at ICASSP 2024 conference this year in Seoul presenting our work in the realms of multi-task, multimodal, and multilingual speech processing technologies. We’re also co-organizing the workshop on Self-supervision in Audio Speech and Beyond (SASB), highlighting the importance of self-supervised learning in advancing speech and audio processing technologies. If you’re attending don’t hesitate to reach out to Ioan Calapodescu and Marcely Zanon Boito who will be on-site at COEX from the 14th to the 19th of April.
NAVER LABS Europe at ICASSP 2024
at
Introduction
Multi-task, multimodal and multilingual speech processing technologies continue to have a number of often interconnected challenges in making them flexible and robust without becoming too computationally heavy. One of the most significant issues is the efficient selection and utilization of data where the availability of large, annotated datasets is often very limited. This is especially the case for certain languages or domains making it difficult to fine-tune models for specific applications. Secondly, as models become more accurate, their size and computational requirements increase, posing a challenge in developing models that can be deployed in environments with limited resources. Multilingual speech models also need to adapt to multiple languages and domains without losing performance on any of them which is complex given the variety of linguistic features and domain-specific requirements. On top of languages and domains, there’s still a need to improve the performance models models in terms of accuracy and speed, especially when dealing with real-world, noisy data. Finally, designing models that can handle multiple speech processing tasks simultaneously is challenging because it requires the model to generalize well across different tasks while maintaining high performance.
Following is a short description of the work we’re presenting at ICASSP this year that attempts to address the above, including results that have been made available to the community.
NAVER LABS Europe research at ICASSP
Unsupervised multi-domain data selection for ASR fine-tuning
Fine-tuning has become one of the standard approaches in developing automatic speech recognition (ASR) models yet, in real-world settings, the availability of the amounts of target-domain data required for fine-tuning can be problematic, especially in certain languages or domains, for reasons of privacy or because they’re expensive. To overcome these issues, you can bootstrap the ASR creation process using a pre-existing (e.g. open-source), large pool of transcribed audio data and select only very small subsets of it, as little as one minute, that represent the target domain. In our ICASSP paper (1), we propose a k-nearest neighbor (KNN)-based data selection method to fine-tune the self-supervised learning based pre-trained speech model wav2vec 2.0. This unsupervised approach helps to select relevant data from a large pool, improving word error rates by up to 13% over random selection. This method is particularly beneficial in low-resource settings.
In ‘An Adapter-based unified model for multiple spoken language processing tasks’ (2) we present an innovative approach to managing the increasing computational demands of high-quality speech processing models. We developed a scalable and efficient unified model capable of handling multiple spoken language processing tasks by employing adapters. These lightweight modules enable the model to adapt to new tasks without extensive retraining, demonstrating an 18.4% average improvement across five speech processing tasks (Automatic Speech Recognition, Phoneme Recognition, Intent Classification, Slot Filing, and Spoken Emotion Recognition). This approach offers a new pathway for creating more versatile and efficient speech processing models. You can see the overall architecture in Figure 2 below.
In our third paper we introduce Multilingual DistilWhisper (3). This model addresses the performance and capacity of Open AI’s Whisper which, despite being a robust multilingual speech model across diverse out-of-domain datasets in ASR, has inference performance issues and, although smaller versions are faster, they underperform in multilingual settings. We distil knowledge from a larger Whisper model into a smaller one, augmented with conditional language-specific routing (CLSR) modules, leading to performance improvements with a minimal increase in parameter count. Our approach makes the development and usage of Whisper models more accessible to researchers and practitioners as we also give access to the models’ weights and code.
Fostering innovation: Workshop on Self-supervision in Audio Speech and Beyond (SASB)
The SASB workshop, co-organized by Naver Labs Europe, is a highlight of ICASSP and was the most popular satellite event last year with the highest number of attendees! The workshop is focused on the use of self-supervision learning (SSL) for speech and audio, and their iteration with other modalities. In the speech and audio domains, most state-of-the-art systems now rely on large Transformer neural networks pretrained on thousands of hours of signal following various methods such as contrastive or multitask learning. However, SSL models currently suffer from critical complexity issues, the lack of a standardized and widely adopted evaluation protocol, biases and robustness concerns as well as disconnection with others closely related modalities (e.g. text or video). Throughout a schedule that maximizes interactions within the audience via multiple panels and a poster session the SASB workshop aims at fostering interactions from the whole SSL community.
References:
1: An Adapter-based Unified Model for Multiple Spoken Language Processing Tasks, Varsha Suresh, Salah Aït-Mokhtar, Caroline Brun, Ioan Calapodescu, IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), Seoul, Korea, 14-19 April, 2024
2: Unsupervised Multi-domain Data Selection for ASR Fine-tuning, Nikolaos Lagos, Ioan Calapodescu, IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), Seoul, Korea, 14-19 April, 2024
3: DistilWhisper: efficient distillation of multi-task speech models via language-specific experts, Thomas Palmeira Ferraz, Marcely Zanon Boito, Caroline Brun, Vassilina Nikoulina, IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), Seoul, Korea, 14-19 April, 2024
To make robots autonomous in real-world everyday spaces, they should be able to learn from their interactions within these spaces, how to best execute tasks specified by non-expert users in a safe and reliable way. To do so requires sequential decision-making skills that combine machine learning, adaptive planning and control in uncertain environments as well as solving hard combinatorial optimisation problems. Our research combines expertise in reinforcement learning, computer vision, robotic control, sim2real transfer, large multimodal foundation models and neural combinatorial optimisation to build AI-based architectures and algorithms to improve robot autonomy and robustness when completing everyday complex tasks in constantly changing environments.
VISION
The research we conduct on expressive visual representations is applicable to visual search, object detection, image classification and the automatic extraction of 3D human poses and shapes that can be used for human behavior understanding and prediction, human-robot interaction or even avatar animation. We also extract 3D information from images that can be used for intelligent robot navigation, augmented reality and the 3D reconstruction of objects, buildings or even entire cities.
Our work covers the spectrum from unsupervised to supervised approaches, and from very deep architectures to very compact ones. We’re excited about the promise of big data to bring big performance gains to our algorithms but also passionate about the challenge of working in data-scarce and low-power scenarios.
Furthermore, we believe that a modern computer vision system needs to be able to continuously adapt itself to its environment and to improve itself via lifelong learning. Our driving goal is to use our research to deliver embodied intelligence to our users in robotics, autonomous driving, via phone cameras and any other visual means to reach people wherever they may be.
This web site uses cookies for the site search, to display videos and for aggregate site analytics.
Learn more about these cookies in our privacy notice.
Cookie settings
You may choose which kind of cookies you allow when visiting this website. Click on "Save cookie settings" to apply your choice.
This content is currently blocked. To view the content please either 'Accept social media cookies' or 'Accept all cookies'.
For more information on cookies see our privacy notice.