One method, one pipeline: NAVER LABS Europe ranks high across three visual localization challenges at CVPR 2020

Using the KAPTURE pipeline and robust R2D2 method, we ranked 1^st, 2^nd and 4^th in the VisLocOdomMap workshop challenge.

For an autonomous robot or vehicle, knowing its exact location is fundamental to many tasks, especially one that requires it to either go somewhere or know where it’s been. One way for an autonomous robot to locate itself is through vision. This task is called visual localization, and it’s an active area of our AI for robotics research at NAVER LABS Europe.

This year, VisLocOdomMapCVPR2020 (Joint Workshop on Long-Term Visual Localization, Visual Odometry and Geometric and Learning-based SLAM at the 2020 Conference on Computer Vision and Pattern Recognition) issued three visual localization challenges to advance research on the topic.

1. visual localization for autonomous vehicles
2. visual localization for handheld devices
3. local features for long-term localization

We’re proud to announce that the entry of our team at NAVER LABS Europe performed extremely well – ranking first in challenge 1, fourth in challenge 2, and second in challenge 3 demonstrating a method that can be generalized and relevant for a broad array of applications.

The robust and versatile performance of our method is due to its combination of robust image retrieval APGeM, robust local features R2D2 (which won the 2019 local feature challenge), and our own visual localization pipeline KAPTURE, which introduces a new data format and a convenient pipeline for processing visual localization and structure from motion data.

Visual Localization

GPS, which relies on the signals of overhead satellites, is an excellent and easy way to obtain exact coordinates, but only in some scenarios. In urban areas, tall buildings form shadows and canyons that can reflect or block these signals. Worse, GPS signals can’t penetrate the tops and sides of buildings, so these signals can’t be reliably used indoors.

In contrast, visual localization can work well in these scenarios.

To locate the position and orientation of an autonomous robot using its cameras, the information in the camera images is processed and then matched with some representation of the environment. There are several ways to do this, such as structure-based methods, which use large-scale 3D reconstructions and local correspondence analysis, scene point regression, which estimates the 3D position for each pixel of the image and absolute pose regression, which directly estimates the camera pose from an input image.

Although vision-based localization can work well in many scenarios, several challenges remain. For instance, most methods rely on a representation of the environment built using images, but the time of day or season of the year can change the appearance of a scene. Furthermore, buildings can change, and the images in the map, even one that is carefully collected, might not cover all the viewpoints that a camera could encounter.

Another set of challenges arises because some types of images are difficult to process reliably; for instance, images that include changes in illumination (or not enough illumination), moving objects, noise such as motion blur, and scenes with little texture.

Finally, reliable real-time performance will be necessary in many applications that use vision-based localization such as autonomous vehicles, robots and augmented reality (AR) software.

Like many visual localization approaches, ours addresses these challenges using structure from motion, which computes local features in 2D images and then matches them to create a 3D reconstruction of the environment. In this approach, we employ our robust image retrieval method APGeM along with our robust local features R2D2. This method is implemented in our own visual localization pipeline KAPTURE.

Robust local features and robust image retrieval (R2D2 + APGeM)

To extract the local features, we use our sparse keypoint detector and descriptor R2D2, which uses a model trained on synthetic image pairs with known transformations. R2D2 performs detection and description jointly, but estimates keypoint reliability and repeatability separately.

Structure from motion performs localization by considering all possible image pairs, but this is a computationally expensive process. This makes it impractical for 3D reconstructions of large-scale environments such as big buildings or even cities. In such cases, image retrieval methods can be used to quickly retrieve only the most relevant images from a structure-from-motion model. This provides initial guesses about the location, which are then refined using a full feature correspondence search.

To retrieve similar images, we use our off-the-shelf deep visual representation APGeM, which consists of a generalized meanpooling (GeM) layer to aggregate feature maps into a compact, fixed-length representation. Mean average precision (mAP) was used to train the model to retrieve landmarks on the Google Landmarks dataset. Although retrieval representations designed for geolocalization have been used in other visual localization methods, we found APGeM to be more suitable for our purposes.

Finally, the COLMAP structure-from-motion library is used to perform the geometric verification of the matches, image registration, and point triangulation needed for 3D reconstruction.

The workflow approach consists of two major pipelines:

• Mapping: training images and poses of the region are first used to create a map of the environment using structure from motion (Figure 2).
• Localization: a query image is registered in the map using the same image retrieval and descriptor matching processes (Figure 3).

Although they accomplish different tasks, the pipelines employ APGeM, R2D2 and COLMAP in similar ways.

Introducing the KAPTURE pipeline

When running a visual localization pipeline on several datasets, one challenge is to convert those datasets into a format that the algorithm and all the tools can handle. To facilitate our experiments, we created KAPTURE, a flexible data format and processing pipeline for structure from motion and visual localization, which we plan to release publicly in the near future.

Many structure-from-motion formats exist, notably the ones from Bundler, VisualSFM, OpenMVG, and COLMAP, but none met all our requirements. In particular, we needed a format that could handle not only timestamps, shared camera parameters, and multi-camera rigs, but also reconstruction data (such as descriptors, global features, 3D points and matches). Moreover, we wanted a format that would be flexible and easy to use in localization experiments. Finally, it should be easy to convert data into other formats supported by major open-source projects such as OpenMVG and COLMAP.

Inspired by OpenMVG and COLMAP, KAPTURE started as a pure data format that provides a good representation of all the information we needed. It then grew into a Python toolbox and library for data manipulation (such as data format conversion, dataset merging/splitting, and trajectory visualization), and finally it became the basis for our mapping and localization pipeline.

Currently, the KAPTURE format can be used to store sensor data such as images, camera parameters, camera rigs, trajectories as well as other sensor data such as LiDAR and WiFi records. It can also be used to store reconstruction data, in particular local descriptors, keypoints, global features, 3D points, observations, and matches.

Results and Future Work

Our method obtains state-of-the-art performance on the various public datasets in the VisLocOdomMapCVPR2020 challenge (the Aachen Day-Night, Inloc, RobotCar Seasons, Extended CMU-Seasons, and SILDa Weather and Time of Day datasets). The images in these datasets include changes in the time of day, season of the year, and outdated reference representations. Images with occlusion, motion blur, extreme viewpoint changes, and low-texture areas are also included. These results demonstrate the excellent ability of our method to generalize and handle many possible changes in the appearance of scenes.

The top rankings in the results from the VisLocOdomMapCVPR2020 challenges further demonstrate its wide variety of potential applications. Specifically, we ranked first in challenge 1, which focuses on autonomous vehicles and provides a sequences of query images with motion limited to a plane. However, our method also performed competitively in challenge 2, which addresses a very different scenario that provides only single query images and unlimited 6-degrees of freedom motion.

Although we could have achieved the same results without KAPTURE, it would have taken much longer because the experiments would have had to be manually run. We believe that the KAPTURE format and tools could be useful to the community, so we plan to release them as an open-source project very soon. We’d also like to provide the major public datasets in the visual localization domain in this format to facilitate future experiments for everybody.

Finally, we believe that global as well as local image features can be further improved to increase the robustness of visual localization, so we plan to continue research in this direction. However, we also believe that large 3D point clouds are quite difficult to maintain in practice. So, we’re also investigating methods that provide image poses without the need for 3D reconstruction.

Research team who contributed to the technology and challenge (in alphabetical order):

Yohann Cabon, Gabriela Csurka Khedari, Nicolas Guérin, Vincent Leroy, Julien Morat, Noé Pion, Philippe Rerole, Jérôme Revaud, Cesar De Souza, Philippe Weinzaepfel