A novel, plug and play model for human 3D shape estimation of the body or hands, in videos which is trained by mimicking the BERT algorithm from the natural language processing community.
PoseBERT
at
Code: github/naver/posebert
PoseBERT [1] is a new algorithm that takes as input the 3D poses of a person estimated in each frame of a video i.e. the position of his/her body joints, and predicts a sequence of 3D shapes. Although the estimations may be noisy due to motion blur, occlusions or ambiguities, PoseBERT returns a smooth sequence of 3D shapes. PoseBERT can also be plugged on top of any state of the art pose estimation method such as SPIN [2], our DOPE model [3] or our new MoCap-SPIN model also presented in [1].
PoseBERT is inspired by the BERT algorithm from the natural language processing (NLP) community. BERT (which stands for Bidirectional Encoder Representations from Transformers), is a method proposed by researchers at Google AI Language in 2018 that has had very good results on a wide variety of NLP tasks such as Question Answering or natural language inference. In their paper [4], the researchers detail, among other elements, a technique named Masked Language Model for bidirectional training of their models. Before feeding sentences into BERT, a percentage of the words in each sequence are masked and the model is trained to predict these masked words, based on the context provided by the other, non-masked, words of the sequence. PoseBERT adapts this learning process to human 3D poses. We mask, or perturb with noise, a percentage of poses in a sequence and PoseBERT attempts to predict the missing or noisy poses by using the context provided by the valid, untouched poses.
We trained two versions of PoseBERT. One model for the body and another one for the hand. In practice, we rely on the SMPL parametric model [5] developed by researchers at the Max Planck Institute in Germany, and train PoseBERT to predict the parameters of this model and not the thousands of vertices of the human 3D mesh. PoseBERT can be trained on Motion Capture data only, without requiring image annotations.
When combined with MoCap-SPIN, PoseBERT reaches state-of-the-art performance for human 3D pose estimation in videos on several 3D pose estimation benchmarks. We also combined PoseBERT with DOPE [3] to estimate the 3D shape of a hand in real-time and used these predictions to animate an ALLEGRO robot hand. This fun demo is given live at the 3DV 2021 conference demo session. We’ll be pursuing this work on pose retargeting and robots manipulating objects like humans so stay tuned to our blog and publications.
Code: github/naver/posebert
References:
- Leveraging MoCap Data for Human Mesh Recovery, Fabien Baradel, Thibault Groueix, Philippe Weinzaepfel, Romain Brégier, Yannis Kalantidis and Grégory Rogez. 9th International Conference on 3D Vision (3DV), virtual event, 1-3 December, 2021. 2min presentation and 10min presentation on Slideslive
- Learning to reconstruct 3D human pose and shape via model-fitting in the loop.Nikos Kolotouros, Georgios Pavlakos, Michael J Black and Kostas Daniilidis. International Conference on Computer Vision (ICCV), Seoul, South Korea, 27 October-2 November, 2019
- DOPE: Distillation Of Part Experts for whole-body 3D pose estimation in the wild. Philippe Weinzaepfel, Romain Brégier, Hadrien Combaluzier, Vincent Leroy and Grégory Rogez. European Conference on Computer Vision (ECCV), virtual event, 23-28 August, 2020
- Bert: Pre-training of deep bidirectional transformers for language understanding. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina ToutanovaarXiv:1810.04805, 2018
- SMPL: A skinned multi-person linear model Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll and Michael J. Black. ACM Transactions on Graphics (TOG) 34 (6), pp. 1-16, November 2015
Related Content
To make robots autonomous in real-world everyday spaces, they should be able to learn from their interactions within these spaces, how to best execute tasks specified by non-expert users in a safe and reliable way. To do so requires sequential decision-making skills that combine machine learning, adaptive planning and control in uncertain environments as well as solving hard combinatorial optimisation problems. Our research combines expertise in reinforcement learning, computer vision, robotic control, sim2real transfer, large multimodal foundation models and neural combinatorial optimisation to build AI-based architectures and algorithms to improve robot autonomy and robustness when completing everyday complex tasks in constantly changing environments.
VISION
The research we conduct on expressive visual representations is applicable to visual search, object detection, image classification and the automatic extraction of 3D human poses and shapes that can be used for human behavior understanding and prediction, human-robot interaction or even avatar animation. We also extract 3D information from images that can be used for intelligent robot navigation, augmented reality and the 3D reconstruction of objects, buildings or even entire cities.
Our work covers the spectrum from unsupervised to supervised approaches, and from very deep architectures to very compact ones. We’re excited about the promise of big data to bring big performance gains to our algorithms but also passionate about the challenge of working in data-scarce and low-power scenarios.
Furthermore, we believe that a modern computer vision system needs to be able to continuously adapt itself to its environment and to improve itself via lifelong learning. Our driving goal is to use our research to deliver embodied intelligence to our users in robotics, autonomous driving, via phone cameras and any other visual means to reach people wherever they may be.
This web site uses cookies for the site search, to display videos and for aggregate site analytics.
Learn more about these cookies in our privacy notice.
Cookie settings
You may choose which kind of cookies you allow when visiting this website. Click on "Save cookie settings" to apply your choice.
This content is currently blocked. To view the content please either 'Accept social media cookies' or 'Accept all cookies'.
For more information on cookies see our privacy notice.