Articulation Aware Canonical Surface Mapping


Nilesh Kulkarni1
Abhinav Gupta2,3
David F. Fouhey1
Shubham Tulsiani3

1University of Michigan
2Carnegie Mellon University
3Facebook AI Research

CVPR 2020

[pdf]
[code]
[video]
[slides]


We tackle the tasks of: a) canonical surface mapping (CSM) i.e. mapping pixels to corresponding points on a template shape, and b) predicting articulation of this template. Our approach allows learning these without relying on keypoint supervision, and we visualize the results obtained across several categories. The color across the template 3D model on the left and image pixels represent the predicted mapping among them, while the smaller 3D meshes represent our predicted articulations in camera (top) or a novel (bottom) view

We tackle the tasks of: 1) predicting a Canonical Surface Mapping (CSM) that indicates the mapping from 2D pixels to corresponding points on a canonical template shape , and 2) inferring the articulation and pose of the template corresponding to the input image. While previous approaches rely on leveraging keypoint supervision for learning, we present an approach that can learn without such annotations. Our key insight is that these tasks are geometrically related, and we can obtain supervisory signal via enforcing consistency among the predictions. We present results across a diverse set of animate object categories, showing that our method can learn articulation and CSM prediction from image collections using only foreground mask labels for training. We empirically show that allowing articulation helps learn more accurate CSM prediction, and that enforcing the consistency with predicted CSM is similarly critical for learning meaningful articulation.



Results


Short video explantion


Acknowledgements

We would like to thank the members of the Fouhey AI lab (FAIL), CMU Visual Robot Learning lab and anonymous reviewers for helpful discussions and feedback. We also thank Richard Higgins for his help with varied quadruped category suggestions and annotating 3D models. This webpage template was borrowed from some colorful folks.