We introduce MultiScan, a scalable RGBD dataset construction pipeline leveraging commodity mobile devices to scan indoor scenes with articulated objects and web-based semantic annotation interfaces to efficiently annotate object and part semantics and part mobility parameters. We use this pipeline to collect 273 scans of 117 indoor scenes containing 10957 objects and 5129 parts. The resulting MultiScan dataset provides RGBD streams with per-frame camera poses, textured 3D surface meshes, richly annotated part-level and object-level semantic labels, and part mobility parameters. We validate our dataset on instance segmentation and part mobility estimation tasks and benchmark methods for these tasks from prior work. Our experiments show that part segmentation and mobility estimation in real 3D scenes remain challenging despite recent progress in 3D object segmentation.

This work was funded in part by a Canada CIFAR AI Chair, a Canada Research Chair, NSERC Discovery Grant, a research grant by Facebook AI Research, and enabled by support from WestGrid and Compute Canada. The iOS and Android scanning apps were developed by Zheren Xiao and Henry Fang. We thank Qirui Wu for his help in re-implementing PointGroup with the Minkowski engine and contributions to the minsu3D code repository. We also thank Zhen (Colin) Li for collecting additional scans, and Henry Fang, Armin Kavian, Han-Hung Lee, Zhen (Colin) Li, Weijie (Lewis) Lin, Sonia Raychaudhuri, and Akshit Sharma for helping to annotate data.

Last updated: 2022-10-23