DAN23 – Team NAVER AI conference

Generative AI is changing the world. To be more precise, we are changing the ‘digital world’.

What changes are we facing recently? First, there are chatbots. Chatbots provide natural, human-like answers 24 hours a day, 365 days a year. And there is also image-generating AI that pushes the boundaries of creativity. This tool opens up new experiences not only for experts, but also for people like me who don’t have much drawing skills. Avatars have also become more realistic. This is mainly an example of what happens in the digital world.

But there is another world worth paying attention to. This is the physical world we live in. The methodologies used in generative AI can also be utilized in the physical world. Plus, it has greater potential. Today we would like to introduce our research on this.

AI research for the physical world

Until now, Naver Labs’ main research target has been the intersection between AI and the physical world. The areas of focus are summarized as ‘action’, where the robot performs tasks in various environments, ‘vision’, where the robot understands and recognizes the environment, and ‘interaction’ between robots and humans. . One of its representative achievements is Naver’s new headquarters, ‘ 1784 ‘. Here, 100 robots provide a variety of robotic services, such as serving coffee to employees or delivering packages. This is where AI technology plays an important role. AI should make robots increasingly smarter and more versatile.

We are also expanding our scope beyond buildings to the entire city. This is a story about Digital Twin technology , which digitally replicates a vast physical space in 3D . AI’s ability to understand the complex physical world is also important for these technologies, which are core data for smart cities such as service robots, urban simulation, autonomous driving, and AR navigation. This will enable data-based urban planning and decision-making and create smart and sustainable cities.

In this way, AI is already playing an important role in understanding the physical world. And we started focusing a few years ago on new methodologies to accelerate this research. This is the Foundation Model.

Decided to switch to the foundation model

Researchers at Naver Labs Europe made a very important decision in 2021. The idea is to convert the research projects that have been carried out so far to be based on the foundation model. Foundation models are pre-trained using very large datasets, so to speak, and are a kind of semi-finished product, so they can be used for a variety of purposes. I am strongly convinced that this methodology will become mainstream. Still, the decision was not an easy one, with high transition costs involved.

Why did you make this choice at the time? To explain this, we must first talk about the limitations of traditional AI approaches.

Previous AI research primarily involved identifying a problem, collecting relevant data, and then training neural networks to find a solution. It is difficult to apply them one by one to various real-life situations. As details of the environment change, performance may degrade during the transition from training to deployment. From a service perspective, there are many difficulties in developing AI that accommodates the diverse needs of individual users.

To be more specific, let’s take AI for robots as an example. From a technological standpoint, it is extremely difficult to get robots to perform tasks autonomously in an uncontrolled environment. Being out of control means being unpredictable. Therefore, finding a way to respond even when a robot stops working unexpectedly is very complicated. So the more unexpected variables there are, the more choices the robot makes that limit its capabilities.

The foundation model is the key to breaking through these limitations. By training it comprehensively on massive amounts of data and then fine-tuning it for specific purposes, you can minimize the effort required to apply it to new tasks. In fact, since switching to the Foundation model, our AI performance has improved significantly. More importantly, researchers in different domains have been able to increase synergy with each other through the foundation model.

Foundation model that overcomes the complexity of reality

Of course, there are many difficulties. First of all, it is not easy to obtain data from the physical world, and since the real world is always changing, data must be updated quickly. The speed of transition between training and deployment is also an important issue. For this reason, compared to its achievements in the digital world, foundation model research in the physical world appears to remain in its infancy globally. Fortunately, we have a head start and have robots operating in our everyday spaces all the time. We also have a huge testbed. There are also many strengths in securing and updating datasets. Thanks to this, we are working on many projects simultaneously, one of which is CROCO, a 3D foundation model for robots and digital twins.

CROCO stands for Cross-view Completion, which teaches AI to understand the real world using images from different viewpoints corresponding to the same scene. It is similar to how humans perceive 3D with both eyes. After training CROCO, you can fine-tune it and use it to make your robot more adaptable to the complex world of physics. For example, many robots connected to the cloud will not only provide good services in a specific space, but will also be able to provide good services in other spaces by adapting to environmental changes. Additionally, performance such as mobility can be stabilized even when environmental information changes or is incomplete. It can be more effectively applied to the mobility of large-scale robots, which is impossible with existing AI methods alone. Ultimately, we want robots to be able to navigate space like humans when they need to get somewhere.

It’s not just about understanding space. We can also expect big changes in terms of our interactions with people. For robots, humans are objects of very complex understanding. If robots can understand human behavior and intentions in a variety of situations, it will be possible to provide robot services that allow safe interaction. This research is very important for popularizing robot services.