Blog home

Topics

Multimodal, multilingual and trustworthy AI models and resources for XR

Published by Laurent Besacier at 6 October 2025

Laurent Besacier, Marcely Zanon Boito

2025

Careers home

An array of tools and resources published within the Unified Transcription and Translation for Extended Reality (UTTER ) project.

As the Horizon Europe UTTER project draws to a close, this article recaps the overall technical progress that was made from 2022-2025 and describes the main contributions of NAVER LABS Europe. Ranging across multilingual foundational models to long-context LLM applications, this work has helped shape cutting-edge innovations across speech processing, LLM evaluation and conversational AI.

A European initiative in language and XR technology

A collaborative Research and Innovation Project funded within the Horizon Europe framework UTTER was about building upon LLMs to create the next generation of multimodal extended reality (XR) technologies for translation, summarization and minuting.

The project brought together a diverse consortium of research institutions and companies, including the University of Edinburgh, IT Lisboa, University of Amsterdam, Unbabel, and NAVER LABS Europe. All working toward a common goal of enabling natural, multilingual and trust-centered communication between humans and machines.

Models and resources by NAVER LABS Europe

mHuBERT-147

We led the development of mHuBERT, a multilingual self-supervised model designed to learn robust speech representations. The mHuBERT-147 model was trained with special attention to maintaining language balance across its 147 languages, resulting in a compact yet highly effective model for multilingual tasks. Upon its release, mHuBERT-147 achieved the top position on the ML-SUPERB benchmark.

The model supports UTTER’s multilingual ambitions by providing a robust and compact backbone for multilingual speech recognition and translation tasks.

UTTER Meeting Assistant

As an industrial partner, NAVER LABS Europe was in charge of one of the project use cases, namely the UTTER Meeting Assistant. This prototype was developed to test long-context large language models (LLMs) in realistic settings. The assistant evolved over three years:

Year 1: We built a general-purpose, LLM-powered assistant for friendly, informal meeting interactions.
Year 2: A production-grade version was developed, robust to ambiguity, noise, and edge cases.
Year 3: A trustworthy-by-design assistant built on our Trust Mediator (TM) framework, became ready for real-world deployment.

This final version incorporates input filtering, safeguards and compliance checks which are core components for building accountable AI systems. The system is demonstrated in the video below and is better with volume to hear the explanations!

This video walks through the main features of the Trust Mediator workbench tool.

Datasets

Benchmarking instruction-following in multilingual meetings

ELITR-Bench is a multilingual evaluation dataset that targets meeting Q&A and instruction-following scenarios. This resource was instrumental in organizing the AutoMin 2025 Shared Task in collaboration with ÚFAL. The dataset enables rigorous evaluation of a model’s ability to answer long-context questions based on hour-long meeting transcripts.

Speech-MASSIVE: A multilingual speech dataset for SLU and beyond

After ELITR-Bench we introduced Speech-MASSIVE, a multilingual spoken language understanding (SLU) dataset that adds a speech counterpart to the pre-existing MASSIVE textual corpus. Speech-MASSIVE spans 12 languages from different families and carries over MASSIVE’s annotations for intent prediction and slot-filling. We built this extension to address the scarcity of large-scale multilingual SLU datasets and the growing need for versatile speech resources to evaluate foundation models (such as LLMs and speech encoders) across languages and tasks.

Multimodality

Contributions to International Conference on Spoken Language Translation (IWSLT) 2023 and 2025 shared tasks

Over a period of two years the team focused on carefully leveraging multimodal and multitask data to build lightweight solutions that efficiently exploit pre-trained text and speech backbones. We contributed to the IWSLT conference on speech translation in both 2023 and 2025, presenting top-performing systems in the low-resource track in 2023 and the instruction-following tracks in 2025.

IWSLT
Winning submissions

2025
INSTRUCTION FOLLOWING
SHORT TRACK

2023
LOW-RESOURCE
TRACK

Adding Speech Modality to a Translation-Specialist LLM

An UTTER-wide effort was to extend the translation-specialist LLM Tower, to the speech modality via discretization. Tower originally supports ten languages, delivering state-of-the-art performance in translation-related tasks. We co-developed Spire, an open source multimodal LLM that extends Tower. Spire can do English speech transcription and from-English translation to nine languages, all without losing Tower’s original state-of-the-art text-to-text translation capabilities.

Efficient distillation of multi-task speech models

Multilingual DistilWhisper is a lightweight way to make smaller Whisper speech models more accurate across many languages. Standard Whisper models work well for some languages but struggle with less-represented ones, especially when using smaller versions. This approach makes improvements by adding simple, language-specific modules, and transferring knowledge from a larger Whisper model. This boosts recognition accuracy while keeping models fast and efficient, making advanced multilingual speech technology more practical and widely usable.

Other contributions

EuroLLM: Long-context modeling for open source European LLMs

We contributed to the EuroLLM model series, a multilingual open source LLM initiative. Our focus was on long-context modeling which is an essential feature for meeting summarization, minute generation and XR applications. EuroLLM supports all EU languages and Korean, aligning with the project’s commitment to linguistic diversity.

Supporting European innovation through cascade funding

NAVER LABS Europe also played a role in evaluating projects submitted under UTTER’s Cascade Funding (FSTP) Call, which supported SMEs and research institutions developing novel applications of LLMs in XR and Human Computer Interaction. This mechanism broadened UTTER’s reach by encouraging experimentation and innovation across the European AI ecosystem.

Wrapping Up

From foundational models mHuBERT, SPIRE and EuroLLM, to real-world systems such as the Meeting Assistant and ELITR-Bench and Speech-MASSIVE resources, we hope to have made significant, cross-cutting contributions to UTTER’s mission of building trustworthy, multilingual and multimodal AI for the next generation of interaction.

Publications

The list of publications that NAVER LABS Europe authored or co-authored within the UTTER project.

From Tower to Spire: adding the speech modality to a translation-specialist LLM, Kshitij Ambilduke, Ben Peters, Sonal Sannigrahi, Anil Keshwani, Tsz Kin Lam, Bruno Martins, Andre Martins, Marcely Zanon Boito (EMNLP 2025)

NAVER LABS Europe submission to the Instruction-following track, Beomseok Lee, Marcely Zanon Boito, Laurent Besacier, Ioan Calapodescu, (IWSLT 2025)

ELITR-Bench: A Meeting Assistant Benchmark for Long-Context Language Models, Thibaut Thonet, Jos Rozen, Laurent Besacier (COLING 2025)

Speech Foundation Models and Crowdsourcing for Efficient, High-Quality Data Collection Beomseok Lee, Marco Gaido, Ioan Calapodescu, Laurent Besacier, Matteo Negri (COLING 2025)

Findings of the Third Automatic Minuting (AutoMin) Challenge, Kartik Shinde, Laurent Besacier, Ondrej Bojar, Thibaut Thonet, Tirthankar Ghosal (arXiv:2509.13814)

Multilingual DistilWhisper: Efficient Distillation of Multi-Task Speech Models via Language-Specific Experts, Thomas Palmeira Ferraz, Marcely Zanon Boito, Caroline Brun, Vassilina Nikoulina (ICASSP 2024)

Speech-MASSIVE: a multilingual speech dataset for SLU and beyond, Beomseok Lee, Marco Gaido, Ioan Calapodescu, Laurent Besacier (Interspeech 2024)

m-HuBERT-147: a compact multilingual HuBERT model, Marcely Zanon Boito, Vivek Iyer, Nikolaos Lagos, Laurent Besacier, Ioan Calapodescu (Interspeech 2024)

Encoding Sentence Position in Context-Aware Neural Machine Translation with Concatenation, Lorenzo Lupo, Marco Dinarelli, Laurent Besacier (Fourth Workshop on Insights from Negative Results in NLP 2023)

LeBenchmark 2.0: A standardized, replicable and enhanced framework for self-supervised representations of French speech, Titouan Parcollet, Ha Nguyen, Solène Evain, Marcely Zanon Boito, Adrien Pupier, Salima Mdhaffar, Hang Le, Sina Alisamir, Natalia Tomashenko, Marco Dinarelli, Shucong Zhang, Alexandre Allauzen, Maximin Coavoux, Yannick Estève, Mickael Rouvier, Jerôme Goulian, Benjamin Lecouteux, François Portet, Solange Rossato, Fabien Ringeval, Didier Schwab, Laurent Besacier (arXiv:2309.05472)

NAVER LABS Europe’s Multilingual Speech Translation Systems for the IWSLT 2023 Low-Resource Track, Edward Gow-Smith, Alexandre Berard, Marcely Zanon Boito, Ioan Calapodescu (IWSLT 2023)

SMaLL-100: Introducing Shallow Multilingual Machine Translation Model for Low-Resource Languages, Alireza Mohammadshahi, Vassilina Nikoulina, Alexandre Berard, Caroline Brun, James Henderson, Laurent Besacier (EMNLP 2022)

What Do Compressed Multilingual Machine Translation Models Forget?, Alireza Mohammadshahi, Vassilina Nikoulina, Alexandre Berard, Caroline Brun, James Henderson, Laurent Besacier (EMNLP Findings 2022)

Multimodal, multilingual and trustworthy AI models and resources for XR

An array of tools and resources published within the Unified Transcription and Translation for Extended Reality (UTTER ) project.

A European initiative in language and XR technology

Models and resources by NAVER LABS Europe

mHuBERT-147

UTTER Meeting Assistant

Datasets

Benchmarking instruction-following in multilingual meetings

Speech-MASSIVE: A multilingual speech dataset for SLU and beyond

Multimodality

Contributions to International Conference on Spoken Language Translation (IWSLT) 2023 and 2025 shared tasks

Adding Speech Modality to a Translation-Specialist LLM

Efficient distillation of multi-task speech models

Other contributions

EuroLLM: Long-context modeling for open source European LLMs

Supporting European innovation through cascade funding

Wrapping Up

Publications

INTERACTION

Equip robots to interact safely with humans, other robots and systems.

VISION

Perception to help robots understand and interact with the environment.

ACTION

Providing embodied agents with sequential decision-making capabilities to safely execute complex tasks in dynamic environments.

NAVER FRANCE Gender Equality 2023

Action

NAVER FRANCE Gender Equality 2024

All

Publications

Blog

News

Code & Data

Careers

People

Topics

Multimodal, multilingual and trustworthy AI models and resources for XR

An array of tools and resources published within the Unified Transcription and Translation for Extended Reality (UTTER ) project.

A European initiative in language and XR technology

Models and resources by NAVER LABS Europe

mHuBERT-147

UTTER Meeting Assistant

Datasets

Benchmarking instruction-following in multilingual meetings

Speech-MASSIVE: A multilingual speech dataset for SLU and beyond

Multimodality

Contributions to International Conference on Spoken Language Translation (IWSLT) 2023 and 2025 shared tasks

Adding Speech Modality to a Translation-Specialist LLM

Efficient distillation of multi-task speech models

Other contributions

EuroLLM: Long-context modeling for open source European LLMs

Supporting European innovation through cascade funding

Wrapping Up

Publications

All

Publications

Blog

News

Code & Data

Careers

People

Cookie settings