Roundtable: Foundation models making robots useful in everyday life

At NAVER LABS Europe, our long-term mission is to make robots a genuine part of everyday spaces such as homes, offices and public spaces. Getting there means advancing robot capabilities well beyond what’s possible today.

In 2021, before there was any proof it would pay off, the lab made a deliberate strategic bet that foundation models were the right way to close that gap. We sat down with Martin Humenberger (Lab Director), Florent Perronnin (Research Fellow), and Vincent Leroy (Team Lead, Geometric Deep Learning) to trace how it went: what convinced them it was the right call, how the research team on 3D vision pivoted toward it, what the first real payoff looked like and where they’re heading next.

What’s actually stopping robots from being part of everyday life?

Martin: Robots are a trendy topic and you see them everywhere online, at events in the news where it looks like they can do anything and everything but that’s pretty misleading. They can do certain things quite well but they also break down in lots of situations. To understand why that happens, it helps to look at how we got here. In the 60s, industrial robots showed up in factories with big arms doing welding, lifting, assembly and other tasks that were dangerous or undesirable for people. That worked because the environment was fully controlled and the tasks could be precisely defined and programmed.

Over time we pushed toward less controlled settings. In the 2000s, robot vacuum cleaners became possible because robots could finally navigate an unknown environment. Then came delivery robots which was another step further because, altough the robot has a clearly defined task, it still has to handle unpredictable things like people, traffic and obstacles. That’s true for outdoor delivery, but also indoor delivery like the robots delivering coffee and parcels at our own 1784 headquarters building in Korea but delivery is still hard because of these unknown factors.

But the real limitation today is versatility: the ability to get out of a narrow zone and handle situations the robot wasn’t explicitly programmed for. That’s where we think AI and specifically foundation models, can be the enabler.

Are delivery robots like this actually out there today?

Martin: It depends a lot on where you look. With the exception of Finland, you don’t see many delivery robots on the street in Europe, partly because the ecosystem of companies building them is smaller here, and partly because getting autonomous systems into public spaces is harder due to regulation, public acceptance, that kind of thing. In China, there are lots operating outdoors; you can find plenty of videos of them working and plenty of videos of them failing in funny ways. In Korea, indoor delivery is more mature and our own 1784 office building is a real, functioning testbed for exactly that. And one place robots have become genuinely normal in Korea is restaurants, where they help carry food and clear plates. That’s partly driven by necessity because, outside the biggest cities, it’s increasingly hard to find people willing to do that job and robots are what keeps some Korean restaurants staffed and open.

Why foundation models, specifically?

Florent: Good question and it’s worth being precise about what we mean. Foundation models are enormous models with often hundreds of billions of parameters, trained on internet-scale data and, crucially, not built for one narrow purpose. They can be adapted to many different tasks.

I first got excited about this in 2021, around the time Stanford published its report on foundation models and we actually had Percy Liang, one of the lead authors, present it to us. What struck me was exactly the thing Martin just described: robots today lack versatility and foundation models are, almost by definition, versatile. That felt like a very natural fit.

But foundation models for the physical world are not the same as foundation models for the digital world, and the differences matter a lot. Take data for example. Digital foundation models train on billions of images or documents scraped from the web. Robotics doesn’t have anything close to that scale of data so we have to find other solutions like create synthetic data. Another major difference is real-time constraints. A chatbot taking an extra second to answer is a non-issue. A robot taking one extra second to make an important decision can be a real problem. And then there’s safety. For language models, safety work often comes after large-scale pretraining. You build the capable model, then work on making it safe. In robotics, safety has to be designed in from the start; it can’t be an afterthought

How did that turn into an actual research direction here?

Martin: At the end of 2021, we made a deliberate strategic decision to treat foundation models as an enabling technology to help robots do things they weren’t specifically programmed for, and do it safely and robustly. That mattered because, ultimately, robots need to be useful enough to justify existing. Nobody’s paying real money for a robot that can dance in the livingroom. 

As an example, we were already working on geometric computer vision, helping robots understand the 3D shape of the spaces around them, which traditionally meant complex, multi-step reconstruction pipelines. We were good at that, but it had real limits and Vincent and the team pushed back. Their reasoning was that essentially this wasn’t the future. You can’t extract and reconstruct every detail of a room, or an entire city, without even knowing in advance what detail you’ll actually need. There had to be a better way,  a model that just knows it and that understands an environment the way a human does, without needing to reconstruct every last piece of it first.

Vincent: Yes, classical 3D reconstruction relies on projective geometry, mathematical rules and optimization schemes to reconstruct an environment from images but this was built on assumptions that get constantly violated in the real world: blank walls with no texture to match, cameras too far apart to triangulate, cameras too close together or even just a single camera. It’s impossible to hand program all the explicit rules you’d need to deal with all possible situations in the real world, so we wanted to train a foundation model to implicitly handle them. We hoped to get the same kind of generalization that LLMs show and that are robust to exactly the situations that break the classical 3D approach.

What was the first real breakthrough?

Vincent: That was DUSt3R. It’s our original foundation model for 3D perception that infers the geometry of a scene directly from images, with no prior knowledge of the camera or scene needed. We had a first prototype in 2023.

What convinced us it was something very different and something that was close to what we were aiming for, in the sense that it was robust and general, was what we ended up calling the “impossible matching” case. Classical geometry needs overlap between viewpoints to reconstruct a scene and we kept pushing DUSt3R into cases we knew were theoretically impossible for the old methods like single cameras, non-overlapping views, degenerate configurations. And it kept working!

Martin: I tried it myself. I took four photos of my living room with zero overlap between them, and it reconstructed the whole room. For anyone who’s spent years working on stereo matching, where overlap is a hard mathematical requirement, that’s a genuinely strange thing to watch work.

Vincent: The first prototype was trained on a small synthetic dataset, and I was already testing it on my phone, on my apartment, on the office cafeteria and it held up! I remember being so excited I had trouble sleeping. We went to a conference right before we could talk about it publicly, and I spent the whole time being deliberately vague about what we had but desperately wanting to talk about it.

Florent: And once it was open-sourced, it created a real buzz. People in the community didn’t believe it would work until they tried it themselves, on their own kitchens and living rooms.

Martin: It ended up being more than just a good 3D reconstruction model as it’s shifted how a lot of the field now approaches the problem. That’s the kind of impact you hope for and rarely get.

What’s happened since DUSt3R?

Florent: We’ve had similar success elsewhere in computer vision. One example is a model focused on understanding humans, their shape, their pose, what they’re doing and this is directly useful in robotics to know if a person is looking at the robot, gesturing to it and so on. Because there’s limited real data of humans in every configuration, we leaned on synthetic data from game engines to build a large, diverse dataset.

The longer-term goal isn’t a separate model for geometry or one for humans. It’s a genuinely universal vision foundation model. That’s harder than it sounds, because robots need real-time perception and running several large models simultaneously is often incompatible with that.

So one project I’m particularly proud of is how we’ve distilled several of these specialized foundation models into a single one. The surprising part is that the combined model isn’t just ‘almost as good’ as the individual specialist models (which is what we expected) but, in some cases, it’s better than any of them individually. The models seem to reinforce each other.

Martin: In that work we combine multiple foundation models into a single universal encoder usable for things like robot navigation. Almost all modern robot navigation relies on some understanding of the world derived from images; an encoder that understands 3D geometry, semantics and human behavior all at once is a genuinely more powerful signal for a robot to act on. And we’re not optimizing for pixel-perfect reconstruction here – we’re optimizing for robust, safe task execution.

Where is this headed?

Florent: Right now we combine multiple vision models but the longer-term goal is combining vision with language models to make a truly universal system, not confined to one modality.

Vincent: We’re drawing on how the brain itself separates function with a visual cortex, language areas, memory… We’ve been building each of these capabilities somewhat separately – perception, semantics, language – and we’re now working on unifying them into a system versatile and robust enough that you can talk to it in natural language and have it understand and adapt to an unfamiliar environment.

Martin: Whether it ends up being one model or a system of models where agents use or guide each other is still an open research question. But the underlying goal is the same: connecting understanding the world with acting in it. That’s been the missing link.

It’s personal for me, too. I did my PhD on home robotics starting around 2008 trying to build a platform that could handle all kinds of household tasks. Back then it was barely more than a vision statement; nobody knew how to give a robot a real understanding of, say, walking into a room with a table and chairs and recognizing it as a living room rather than a meeting room. The world is too big and too varied to hand code. When the ChatGPT moment happened, it was clear there was the potential to finally connect the vast, general knowledge in language models to the physical world so that a robot can actually reason and safely act in the space it’s in.

This is the first in a series of conversations with NAVER LABS Europe researchers. Next time: how we take these universal foundation models and put them to work on real robots.

For legibility reasons, this article is an edit of the transcript of the recording of the roundtable.