This year, NAVER LABS Europe is presenting two papers at Interspeech 2024 in Kos, Greece, one on the multilingual speech dataset Speech-MASSIVE and the other, the multilingual speech representation model mHUBERT-147. Here we summarize these contributions and introduce a demo for spoken language understanding that leverages both resources for French. If you’re attending Interspeech, don’t hesitate to reach out to authors Beomseok and Marcely to discuss their work.
Speech-MASSIVE: A Multilingual Speech Dataset for SLU and Beyond [paper | dataset]
Spoken Language Understanding (SLU) involves interpreting spoken input using Natural Language Processing (NLP). Voice assistants like Alexa and Siri are real-world examples of SLU applications. The core tasks in SLU include intent classification, which determines the goal or command behind an utterance, and slot-filling, which extracts specific details such as dates or music genres from the utterance. However, gathering SLU data presents challenges due to the complexities of recording, validation, and associated costs. Additionally, most existing datasets are primarily English-focused, limiting language diversity and cross-linguistic applications.
To address these gaps, we introduce Speech-MASSIVE, a multilingual SLU dataset that builds on the translations and annotations from the MASSIVE corpus. The dataset was curated through crowd-sourcing with strict quality controls and spans 12 languages (Arabic, German, Spanish, French, Hungarian, Korean, Dutch, Polish, European Portuguese, Russian, Turkish and Vietnamese), covering 8 language families and 4 different scripts, with a total of over 83,000 spoken utterances.
We also established baseline models under various system and resource configurations to facilitate broad comparisons. We hope this resource helps to advance multilingual SLU research and encourages the development of future cascaded and end-to-end models. The dataset is freely accessible on HuggingFace under the CC BY-NC-SA 4.0 license.
mHuBERT-147: A Compact Multilingual HuBERT Model [paper | weights | code]
Speech representation models form the foundation of most modern speech-related technologies. These models are trained using unsupervised learning on vast datasets, where deep encoder networks learn to capture rich, nuanced speech patterns. Once trained, they can be applied to various speech applications, often achieving impressive results even with minimal labeled data.
We introduce mHuBERT-147, a compact yet highly effective multilingual speech representation model that supports 147 languages. mHuBERT-147 achieves an exceptional balance between performance and efficiency, reaching 2nd and 1st place in the ML-SUPERB (10min/1h) leaderboards whilst being 3 to 10 times smaller than its competitors. Moreover, our model is trained on nearly five times less data than comparable multilingual models, highlighting the crucial role of data curation in efficient self-supervised learning. You can explore the mHuBERT-147 model on HuggingFace, where, lime Speech-MASSIVE, it’s available under the CC-BY-NC-SA 4.0 license.
DEMO: Leveraging these resources for spoken language understanding
To showcase these resources that we’re releasing, we built a demo for French spoken language understanding at HuggingFace. The SLU model is made of a French ASR component, trained using mHuBERT-147 and Speech-MASSIVE, and a NLU component, made of a mT5 fine-tuned on MASSIVE data.
You can access the demo at: https://huggingface.co/spaces/naver/French-SLU-DEMO-Interspeech2024
We detail the deployment of these resources, with code snippets at our HuggingFace article.
INTERSPEECH PAPER DETAILS
- Monday, September 2nd, 4:00 – 4:20 pm, Beomseok Lee presenting “Speech-MASSIVE: A Multilingual Speech Dataset for SLU and Beyond,” which has been nominated for Best Student Paper.
- Wednesday, September 4th, 4:00 – 6:00 pm, Marcely Zanon Boito presenting “mHuBERT-147: A Compact Multilingual HuBERT Model” in Poster Area 2A.
To make robots autonomous in real-world everyday spaces, they should be able to learn from their interactions within these spaces, how to best execute tasks specified by non-expert users in a safe and reliable way. To do so requires sequential decision-making skills that combine machine learning, adaptive planning and control in uncertain environments as well as solving hard combinatorial optimisation problems. Our research combines expertise in reinforcement learning, computer vision, robotic control, sim2real transfer, large multimodal foundation models and neural combinatorial optimisation to build AI-based architectures and algorithms to improve robot autonomy and robustness when completing everyday complex tasks in constantly changing environments.
VISION
The research we conduct on expressive visual representations is applicable to visual search, object detection, image classification and the automatic extraction of 3D human poses and shapes that can be used for human behavior understanding and prediction, human-robot interaction or even avatar animation. We also extract 3D information from images that can be used for intelligent robot navigation, augmented reality and the 3D reconstruction of objects, buildings or even entire cities.
Our work covers the spectrum from unsupervised to supervised approaches, and from very deep architectures to very compact ones. We’re excited about the promise of big data to bring big performance gains to our algorithms but also passionate about the challenge of working in data-scarce and low-power scenarios.
Furthermore, we believe that a modern computer vision system needs to be able to continuously adapt itself to its environment and to improve itself via lifelong learning. Our driving goal is to use our research to deliver embodied intelligence to our users in robotics, autonomous driving, via phone cameras and any other visual means to reach people wherever they may be.
This web site uses cookies for the site search, to display videos and for aggregate site analytics.
Learn more about these cookies in our privacy notice.
Cookie settings
You may choose which kind of cookies you allow when visiting this website. Click on "Save cookie settings" to apply your choice.
This content is currently blocked. To view the content please either 'Accept social media cookies' or 'Accept all cookies'.
For more information on cookies see our privacy notice.