NAVER LABS Europe @ECCV 2024

NAVER LABS Europe has a high level of participation at ECCV 2024 across multiple themes and events including pre-conference workshops and invited talks and orals, posters and several live demonstrations in the main conference.

On September 29th Bulent is co-organizing the workshop “Self Supervised Learning: What is Next?“, Fabien, Ginger and Gabriela are presenting their latest research on multi-person whole-body human mesh recovery (Multi-HMR), advanced pose representation (PoseEmbroider) in T-CAP and computer vision failure modes in eXplainableCV. You can catch Diane Larlus giving an invited talk on geometric understanding in egocentric videos in the TRICKY and CV4Metaverse workshops and Vincent Leroy will be investigating whether DUSt3R spells the doom of traditional geometric vision in the TradiCV workshop!

As Day 2 kicks off MASt3R takes central stage at the Map-free Relocalization workshop, winning the single frame challenge track organised by Niantic. Vincent is invited to present how this innovative approach to 3D image matching stands out from other approaches. If you missed Multi-HMR and PoseEmbroider on Sunday, you get a second chance in the Foundation Models for 3D Humans workshop and we invite you to go meet Gabriela Csurka who is co-organising the Foundation models Creator Meets Users workshop in the afternoon.

Getting into the main conference, Tuesday is our busiest day with the Oral on MASt3R (official title ‘Grounding image matching in 3D with MASt3R’). If Orals are not your thing you can meet authors Vincent and Jerome at their poster session on the topic at the end of the day (Session 2) and then move on over to the Exhibition Area that evening for the real time MASt3R demo. At the same afternoon poster session you’ll find Fabien and Philippe presenting Multi-HMR which they’re also demonstrating but not until 9am the next day  Wednesday – same place as MASt3R in the exhibition area.

Our last posters are on Thursday with Pau, Ginger and Philippe at Session 5 in the morning. Pau will be at #121 presenting the POC pipeline (Placing Objects in Context) for semantic segmentation and Ginger and Philippe, PoseEmbroider. Bulent and Philippe round up our presence in the afternoon at Poster Session 6 with the universal classification model UNIC.

You’ll find details on our agenda below and please reach out to our researchers then and any time throughout the conference. Have a great ECCV24!

Workshops

Sunday, September 29^th

09:00-13:00 CEST Space 2

Self Supervised Learning: What is Next? (W41)
Co-organiser: Mert Bulent Sariyildiz

14:20-15:30 CEST Amber 7&8
T-CAP: Towards a Complete Analysis of People (W34)
Spotlight presentation paper: Multi-HMR: Multi-Person Whole-Body Human Mesh Recovery in a Single Shot
Fabien Baradel, Matthieu Armando, Salma Galaaoui, Romain Brégier, Philippe Weinzaepfel, Grégory Rogez, Thomas Lucas

Spotlight presentation paper: PoseEmbroider: Towards a 3D, Visual, Semantic-aware Human Pose Representation
Ginger Delmas, Philippe Weinzaepfel, Francesc Moreno, Grégory Rogez

15:15 CEST Brown 1
eXCV: Explainable Computer Vision:Where are we and where are we going? (W62)
Paper: What could go wrong? Discovering and describing failure modes in computer vision, Gabriela Csurka, Tyler L. Hayes, Diane Larlus, Riccardo VolpiGinger Delmas, Philippe Weinzaepfel, Francesc Moreno-Noguer, Grégory Rogez

16:40 CEST Amber 2
TradiCV: Traditional Computer Vision in the Age of Deep Learning (W42)
Invited speaker Vincent Leroy: DUSt3R: the doom of traditional geometric vision?

17:05 CEST Room Suite 8

TRICKY: Transparent & Reflective objects In the wild Challenge (W06)
Invited speaker: Diane Larlus: “Towards a Geometric Understanding in Egocentric Videos”

17:05 CEST Amber 1

CV4Metaverse: 3rd Edition of Computer vision for Metaverse (W13)
Invited speaker: Diane Larlus: ‘Towards a Geometric Understanding in Egocentric Videos’

Monday, September 30^th

11:00-11:45 CEST Suite 6
Map-free Visual Relocalization (W71)
Invited speaker: Vincent Leroy “Grounding Image Matching in 3D with MASt3R”
This workshop Includes the challenge on Map-free relocalization. ***MASt3R ranked #1 in Single frame challenge track***

15:40-16:20 CEST Tower Lounge
Foundation Models for 3D Humans (W32)
PoseEmbroider and Multi-HMR papers (see conference papers above) presented in this workshop.

14:00-18:00 CEST Suite 2

FOCUS: FOundation models Creators meet USers (W60)
Co-organiser: Gabriela Csurka

Conference papers (orals, posters, demos):

Tuesday, Oct 1^st, 14:40 CEST | Orals

Grounding image matching in 3D with MASt3R (Oral 2C)
Vincent Leroy, Yohann Cabon, Jérome Revaud

Abstract: Image Matching is a core component of all best-performing algorithms and pipelines in 3D vision. Yet despite matching being fundamentally a 3D problem, intrinsically linked to camera pose and scene geometry, it is typically treated as a 2D problem. This makes sense as the goal of matching is to establish correspondences between 2D pixel fields, but also seems like a potentially hazardous choice. In this work, we take a different stance and propose to cast matching as a 3D task with DUSt3R, a recent and powerful 3D reconstruction framework based on Transformers. Based on pointmaps regression, this method displayed impressive robustness in matching views with extreme viewpoint changes, yet with limited accuracy. We aim here to improve the matching capabilities of such an approach while preserving its robustness. We thus propose to augment the DUSt3R network with a new head that outputs dense local features, trained with an additional matching loss. We further address the issue of quadratic complexity of dense matching, which becomes prohibitively slow for downstream applications if not treated carefully. We introduce a fast reciprocal matching scheme that not only accelerates matching by orders of magnitude, but also comes with theoretical guarantees and, lastly, yields improved results. Extensive experiments show that our approach, coined MASt3R, significantly outperforms the state of the art on multiple matching tasks. In particular, it beats the best published methods by 230% (relative improvement) in VCRE Precision on the extremely challenging Map-free localization dataset.
[PDF] 

#130. Multi-HMR: Multi-Person Whole Body Human Mesh Recovery in a Single Shot
Fabien Baradel, Thomas Lucas, Matthieu Armando, Salma Galaaoui, Romain Brégier, Philippe Weinzaepfel, Grégory Rogez

Abstract: We present Multi-HMR, a strong single-shot model for multi-person 3D human mesh recovery from a single RGB image. Predictions encompass the whole body, i.e., including hands and facial expressions, using the SMPL-X parametric model and 3D location in the camera coordinate system. Our model detects people by predicting coarse 2D heatmaps of person locations, using features produced by a standard Vision Transformer (ViT) backbone. It then predicts their whole-body pose, shape and 3D location using a new cross-attention module called the Human Prediction Head (HPH), with one query attending to the entire set of features for each detected person. As direct prediction of fine-grained hands and facial poses in a single shot, i.e., without relying on explicit crops around body parts, is hard to learn from existing data, we introduce CUFFS, the Close-Up Frames of Full-body Subjects dataset, containing humans close to the camera with diverse hand poses. We show that incorporating this dataset into training further enhances predictions, particularly for hands, enabling us to achieve competitive performance. Multi-HMR also optionally accounts for camera intrinsics, if available, by encoding camera ray directions for each image token. This simple design achieves strong performance on whole-body and body-only benchmarks simultaneously. We train models with various backbone sizes and input resolutions. In particular, using a ViT-S backbone and 448×448 input images already yields a fast and competitive model, while larger models and higher resolutions further improves performance.
[PDF]

Wednesday, October 2^nd

09:00 – 12:30 CEST. Demonstration, Exhibition Area
Multi-HMR: Multi-Person Whole-Body Human Mesh Recovery in a Single Shot
Thomas Lucas, Matthieu Armando, Salma Galaaoui, Philippe Weinzaepfel, Romain Brégier, Grégory Rogez, Fabien Baradel

14:40 CEST | Demonstration, Exhibition Area
Grounding image matching in 3D with MASt3R (Winner of the map-free localization challenge @ECCV2024)
Vincent Leroy, Yohann Cabon, Jérome Revaud

#121: Placing Objects in Context via Inpainting for Out-of-Distribution Segmentation
Pau de Jorge Aranda, Riccardo Volpi, Puneet Dokania, Philip Torr, Grégory Rogez

Abstract: When deploying a semantic segmentation model into the real world, it will inevitably encounter semantic classes that were not seen during training. Therefore, to ensure a safe deployment of such systems, it is crucial to accurately evaluate and improve their anomaly segmentation capabilities. However, acquiring and labelling semantic segmentation data is expensive and unanticipated conditions are long-tail and potentially hazardous. Indeed, existing anomaly segmentation datasets capture a limited number of anomalies, lack realism or have strong domain shifts. In this paper, we propose the Placing Objects in Context (POC) pipeline to realistically add any object into any image via diffusion models. POC can be used to easily extend any dataset with an arbitrary number of objects. In our experiments, we present different anomaly segmentation datasets based on POC-generated data and show that POC can improve the performance of recent state-of-the-art anomaly fine-tuning methods across several standardized benchmarks. POC is also effective for learning new classes. For example, we utilize it to augment Cityscapes samples by incorporating a subset of Pascal classes and demonstrate that models trained on such data achieve comparable performance to the Pascal-trained baseline. This corroborates the low synth2real gap of models trained on POC-generated images.
[PDF]

#10. UNIC: Universal Classification Models vis Multi-teacher Distillation
Mert Bulent Sariyildiz, Philippe Weinzaepfel, Thomas Lucas, Diane Larlus, Yannis Kalantidis

Abstract: Pretrained models have become a commodity and offer strong results on a broad range of tasks. As they resort to different learning strategies, they tend to be complementary. In this work, we focus on classification and seek to learn a unique encoder able to take from several of those pretrained models. We aim at even stronger generalization across a variety of classification tasks. We propose to learn such an encoder via multi-teacher distillation. We first thoroughly analyse standard distillation when driven by multiple strong teachers with complementary strengths. Guided by this analysis, we gradually propose improvements to the basic distillation setup. Among those, we enrich the architecture of the encoder with a ladder of expendable projectors, which increases the impact of intermediate features during distillation, and we introduce teacher dropping, a regularization mechanism that better balances the teachers’ influence. Our final distillation strategy leads to student models of the same capacity as any of the teachers, while retaining or improving upon the performance of the best teacher for each task.
[PDF] [arXiv]