Dense and Unconstrained Stereo 3D Reconstruction (DUSt3R) is a novel breakthrough approach in 3D reconstruction. DUSt3R distinguishes itself by analysing image content directly which, contrary to existing methods, means it doesn’t require any camera parameters. It is robust and fast, works on just a few images (no overlap required) and is the first method that can do both 3D reconstruction and visual localization with no user intervention.
3D reconstruction models made easy
at
| Shuzhe Wang, Vincent Leroy, Yohann Cabon, Boris Chidlovskii, Jérome Revaud |
| 2024 |
Code: github.com/naver/dust3r
3D reconstruction
3D reconstruction is the task of creating a 3D virtual representation of an object or scene from images that are only two dimensional (2D). It is used in simulation, visualization or localization, most commonly in the fields of computer vision, robotics and virtual reality (VR). For example, if you provide a 3D reconstruction tool with a couple of photos of a room, it will give you a 3D representation of the room (see Figure 1).
Current state of the art methods in Multi-View Stereo Reconstruction (MVS) such as DeMoN [1] and DeepV2D [2] can accomplish this task if the camera parameters (intrinsics) for the two images are known as shown in Video 1. However, not only do the intrinsics need to be provided, making a good estimate can be cumbersome and prone to error and, if the camera parameters are unknown or poorly estimated, the 3D reconstruction fails as shown in Video 2.
Video 1: A 3D reconstruction method with a pair of images I1 and I2 as input and depth maps D1 and D2 as output plus a relative pose P between them.
Video 2: Existing state of the art methods fail to reconstruct a 3D scene if camera intrinsics are unknown or poorly estimated.
The main advantage of the simpler DUSt3R method is that, by operating directly on the image content, it can estimate the image depths and viewpoint poses to generate a complete 3D reconstruction without knowing the camera parameters.
Video 3: Illustration of how DUSt3R works with no knowledge of camera parameters, contrary to existing state of the art methods.
How DUSt3R works
DUSt3R is ‘unconstrained’ because it addresses the challenge of 3D reconstruction from images with no information on camera parameters or viewpoint poses. i.e. it doesn’t have that as a constraint. The system works by formulating pairwise reconstruction as a regression of pointmaps, which makes it very different from the standard projective camera models. The network efficiently decodes these pointmaps with rich geometric details from pairs of images, simplifying the process of extracting detailed scene geometry.
The network architecture is based on standard Transformer encoders and decoders which means that we can leverage existing powerful pre-trained models. Our formulation directly provides a 3D model of the scene as well as depth information, but we can also seamlessly recover pixel matches and relative and absolute camera poses.
Figure 2: The DUSt3R architecture shows how two views of a scene (I1 , I2) are first encoded in a Siamese manner with a shared ViT encoder. The resulting token representations F1 and F2 are then passed to two transformer decoders that constantly exchange information via cross-attention. Finally, two regression heads output the two corresponding pointmaps and associated confidence maps. Importantly, the two pointmaps are expressed in the same coordinate frame of the first image I1. The network is trained using a simple regression loss.
Reconstruction from two opposite images
Below are several examples of 3D reconstructions from nearly opposite viewpoints of an object. For each of the cases (bench, motorcycle and toaster), we show the two input images and the video for the animated visualizations. DUSt3R handles drastic viewpoint changes with apparent ease, even when there is either no or almost no overlapping visual content between images e.g. for the motorcycle.
Video 4: On the left are 2 input images of an object from nearly opposite viewpoints and on the right, the 3D visualization output produced by DUSt3R.
Reconstruction from one image
DUSt3R is able to reconstruct the 3D scene from even just a single image as shown in Figure 3 below.
Global optimization from multiple views
In the case where multiple views are provided, DUSt3R proposes a simple yet effective global optimization strategy that expresses all pairwise pointmaps in a common reference frame.
Video 5: Animation of the global optimization process that expresses all pairwise pointmaps in a common reference frame.
Integrating multiple tasks into a single pipeline
The most important feature of DUSt3R is its ability to unify various 3D vision tasks, traditionally handled separately, into a single, simplified pipeline. The DUSt3R architecture leverages pretrained models and a fully data-driven approach to learn powerful geometric and shape priors. This process results in direct 3D models of scenes that also lend themselves to tasks like depth and pose estimation, visual localization and multi-view 3D reconstruction.
Results
Experimentation results with datasets such as DTU, Tanks and Temples, and ETH3D demonstrate that DUSt3R works without known camera parameters. The network can handle monocular reconstruction and align multiple image pairs in a common reference frame. The evaluations position DUSt3R as setting new standards, achieving state of the art results across a range of tasks including multi-view/monocular depth estimation and camera pose estimation. Table 1 below compares DUSt3R to state of the art methods in the task of multi-view pose regression on the CO3Dv2 and RealEst10K datasets (see paper [3] for details).
3D reconstruction of ETH3D exhibition hall sequence
Video 6: 3D reconstruction of Exhibition hall sequence from ETH3D benchmark (free pose trajectory).
Video 7: Old tower at NAVER LABS Europe site: 3D reconstruction
Conclusion
DUSt3R represents a major advancement in 3D geometric vision, offering a substantial simplification over traditional methods. The results underline its potential and versatility in handling diverse 3D vision challenges without the exhaustive and meticulous steps of estimating and calibrating camera parameters, representing a significant step forward for future developments in the field of 3D reconstruction.
References
1: DeMoN: Depth and Motion Network for Learning Monocular Stereo, Benjamin Ummenhofer et al., IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, Hawaii, 21-26 July, 2017.
2: DeepV2D: Video to Depth with Differentiable Structure from Motion, Zachary Teed, Jia Deng, The Eighth International Conference on Learning Representations (ICLR), Virtual only conference, April 26-May 1, 2020.
3: DUSt3R: 3D geometric 3D vision made easy, Shuzhe Wang, Vincent Leroy, Yohan Cabon, Boris Chidlovskii, Jerome Revaud, arXiv 2312.14132V1, 2023.CVPR 2024 and arXiv
To make robots autonomous in real-world everyday spaces, they should be able to learn from their interactions within these spaces, how to best execute tasks specified by non-expert users in a safe and reliable way. To do so requires sequential decision-making skills that combine machine learning, adaptive planning and control in uncertain environments as well as solving hard combinatorial optimisation problems. Our research combines expertise in reinforcement learning, computer vision, robotic control, sim2real transfer, large multimodal foundation models and neural combinatorial optimisation to build AI-based architectures and algorithms to improve robot autonomy and robustness when completing everyday complex tasks in constantly changing environments.
VISION
The research we conduct on expressive visual representations is applicable to visual search, object detection, image classification and the automatic extraction of 3D human poses and shapes that can be used for human behavior understanding and prediction, human-robot interaction or even avatar animation. We also extract 3D information from images that can be used for intelligent robot navigation, augmented reality and the 3D reconstruction of objects, buildings or even entire cities.
Our work covers the spectrum from unsupervised to supervised approaches, and from very deep architectures to very compact ones. We’re excited about the promise of big data to bring big performance gains to our algorithms but also passionate about the challenge of working in data-scarce and low-power scenarios.
Furthermore, we believe that a modern computer vision system needs to be able to continuously adapt itself to its environment and to improve itself via lifelong learning. Our driving goal is to use our research to deliver embodied intelligence to our users in robotics, autonomous driving, via phone cameras and any other visual means to reach people wherever they may be.
This web site uses cookies for the site search, to display videos and for aggregate site analytics.
Learn more about these cookies in our privacy notice.
Cookie settings
You may choose which kind of cookies you allow when visiting this website. Click on "Save cookie settings" to apply your choice.
This content is currently blocked. To view the content please either 'Accept social media cookies' or 'Accept all cookies'.
For more information on cookies see our privacy notice.