Canonical Surface Mapping via Geometric Cycle Consistency


Nilesh Kulkarni
Abhinav Gupta*
Shubham Tulsiani*

Carnegie Mellon University
Facebook AI Research

In ICCV 2019

[pdf]
[code]
[video]


We study the task of Canonical Surface Mapping. This task is a generalization of keypoint estimation and involves mapping pixels to canonical 3D models. We learn CSM prediction without requiring correspondence annotations, by instead using geometric cycle consistency as supervision. This allows us to train CSM prediction for diverse classes, including rigid and non-rigid objects.

We explore the task of Canonical Surface Mapping (CSM). Specifically, given an image, we learn to map pixels on the object to their corresponding locations on an abstract 3D model of the category. But how do we learn such a mapping? A supervised approach would require extensive manual labeling which is not scalable beyond a few hand-picked categories. Our key insight is that the CSM task (pixel to 3D), when combined with 3D projection (3D to pixel), completes a cycle. Hence, we can exploit a geometric cycle consistency loss, thereby allowing us to forgo the dense manual supervision. Our approach allows us to train a CSM model for a diverse set of classes, without sparse or dense keypoint annotation, by leveraging only foreground mask labels for training. We show that our predictions also allow us to infer dense correspondence between two images, and compare the performance of our approach against several methods that predict correspondence by leveraging varying amount of supervision.

Approach Overview


Results


Paper

Kulkarni, Gupta, Tulsiani.

Cannonical Surface Mapping via Geometric Cycle Consistency.

In ICCV 2019

[pdf]
[bibtex]

Code


 [GitHub]

Acknowledgements

We would like the thank the members of the CMU Visual and Robot Learning group and the anonymous reviewers for helpful discussions and feedback. ST would also like to thank Alyosha Efros for his constant nudges to `work on cycles!'.This webpage template was borrowed from some colorful folks.