Summary of the 2023 AI for Robotics Workshop

In the same vein of exploiting inductive biases to improve the sample efficiency of the learning process, Sylvain Calignon, senior researcher at IDIAP Research Institute, emphasized the benefits of leveraging the structures of both the data and the geometry when addressing complex tasks in robotics. In particular, he explained how data factorization techniques using Tensor Networks could be useful for global discrete/continuous optimization in hard planning problems, and how geometric algebra could offer an elegant and efficient framework to acquire skills from a small set of interactions.

Modular approaches were also suggested by Devendra Chaplot, researcher at Mistral AI, in particular for indoor navigation. He showed that this kind of structured decomposition is capable of being deployed in real environments of various types after being trained in simulation only. He also showed that the goal specification can be generalized using large language models (LLM) and visual language models (VLM), allowing goal specifications through coordinates, goal images, or language input, and targeting various different embodiments.

Structured representations were also suggested by Martial Herbert, Dean of Carnegie Mellon University, in particular for perception as a means to structure video input. He showed how self-supervised learning can be leveraged for learning with minimal supervision to exploit temporal consistencies and multi-view consistencies. This is enabled through slot-attention on the one hand or, on the other hand, extensions of implicit representations (NERFs) and multi-task learning.

Cordelia Schmid, research director at Inria and researcher at Google, also argued in favor of structured representations. She presented representations for end-to-end trained agents, used for perception, mapping and decision taking and, application-wise, covering navigation as well as manipulation. As an example she presented neural implicit maps, which are differentiable latent representations respecting projective 3D geometry and trained with imitation learning. A CLIP encoder ensures that the features are linked to semantic content and can be used with language queries

Learning to take decisions

The question of whether learning should be done from demonstrations through imitation learning, or from reward with reinforcement learning (RL), was hotly debated with several speakers suggesting that RL should be avoided whenever possible due to its difficulty. While there seemed to be a consensus that imitation learning seems to be an easier learning problem, enabling learning for a large variety of problems, RL was also shown to be the method of choice in several successful cases. Davide Scaramuzza, Professor at the University of Zürich, presented a spectacular success case, where RL was used to beat world champions in drone racing from first person views. Trained in simulation combined with recorded trajectories from real drones, this work, combining seven years of research of his group also showed that RL can beat optimal control for this task: whereas optimal control decomposes the problem into planning and control, limiting the range of behaviors, RL can directly optimize a task level objective and also cope with model uncertainty. RL was also the method of choice presented by Olivier Sigaud, Professor at Sorbonne University, who insisted on social aspects in particular in situations where artificial agents are either teachers or students, in which case they require pedagogical teaching capabilities or pragmatic interpretation capabilities, respectively. Oya Celiktutan, Senior Lecturer at King’s College London has also been studying social robotics for several years working on robots assisting humans on the physical, social and cognitive levels. She specifically introduced a method for identifying clusters of individuals engaged in conversation, enabling the robot to navigate between these groups rather than within the same group of people (social navigation). Laure Soulier, associate professor at Sorbonne University/ISIR lab, conducted a comprehensive review of the literature regarding the application of LLMs in robotics, with a particular focus on harnessing their planning capabilities to generate sequences of actions.

Simulation as an infinite source of data

Gül Varol of the IMAGINE research team at Ecole des Ponts ParisTech, summarized recent work on bridging the gap between natural language and 3D human motions, with several examples on human motion control and synthesis from textual descriptions. The possible applications range from the control of humanoid robots to human-robot interactions. Motion capture is limited and expensive, and generative models are a solution to provide unlimited data. In this domain she stated that “Leveraging physics would be the cherry on top of the cake, it would bring more realism but would not help the lack of data in terms of semantics”.

Generating large-scale data to compensate for the relatively small datasets available in robotics (typically millions of times smaller than the ones used to pre-train foundation models in NLP and vision) was also the focus of the presentation done by Markus Wulfmeier, Research Scientist at Google DeepMind. He illustrated how to semi-automatically generate diverse and relevant training data for learning navigation or visuo-motor control policies on two use cases, namely training a real robot in soccer and recommending the best routes in Google Maps.

Hao Su, Associate Professor of Computer Science and Director of the Embodied AI lab at the University of California, San Diego (UCSD) presented his work on modeling the 3D physical world for embodied intelligence. His team has spent many years building a full-stack system for robot learning, called SAPIEN, which provides physical simulation for robots, rigid body and articulated objects. He strongly believes that robot simulation is an important ingredient for the community as it allows scalability, replicability and fast prototyping and sim2real should be solved by making simulators closer to the real world.

Poster session

There were 19 student posters presented at the workshop. Rabbia Asghar from Inria received the best poster award for  ‘Vehicle Motion Forecasting Using Prior Information and Semantic-Assisted Occupancy Grid Maps’.

The AI for Robotics international workshop is a biennial event organized by NAVER LABS Europe. The videos of the latest edition and the two previous workshops (2021, 2019) are available online.

This article is an edited version of the summary published in the December issue of RSIP vision.