SemanticFusion: dense 3D semantic mapping with convolutional neural networks
File(s)Semanticfusion_ICRA17_Final.pdf (3.31 MB)
Accepted version
Author(s)
McCormac, J
Handa, A
Davison, AJ
Leutenegger, S
Type
Conference Paper
Abstract
Ever more robust, accurate and detailed mapping using visual sensing has proven to be an enabling factor for mobile robots across a wide variety of applications. For the next level of robot intelligence and intuitive user interaction, maps need to extend beyond geometry and appearance — they need to contain semantics. We address this challenge by combining Convolutional Neural Networks (CNNs) and a state-of-the-art dense Simultaneous Localization and Mapping (SLAM) system, ElasticFusion, which provides long-term dense correspondences between frames of indoor RGB-D video even during loopy scanning trajectories. These correspondences allow the CNN's semantic predictions from multiple view points to be probabilistically fused into a map. This not only produces a useful semantic 3D map, but we also show on the NYUv2 dataset that fusing multiple predictions leads to an improvement even in the 2D semantic labelling over baseline single frame predictions. We also show that for a smaller reconstruction dataset with larger variation in prediction viewpoint, the improvement over single frame segmentation increases. Our system is efficient enough to allow real-time interactive use at frame-rates of ≈25Hz.
Date Issued
2017-07-24
Date Acceptance
2017-02-25
Citation
Robotics and Automation (ICRA), 2017 IEEE International Conference on, 2017, pp.4628-4635
ISBN
978-1-5090-4633-1
Publisher
IEEE
Start Page
4628
End Page
4635
Journal / Book Title
Robotics and Automation (ICRA), 2017 IEEE International Conference on
Copyright Statement
© 2017 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.
Sponsor
Dyson Technology Limited
Grant Number
PO 4500501004
Source
IEEE International Conference on Robotics and Automation (ICRA), 2017
Publication Status
Published
Start Date
2017-05-29
Finish Date
2017-06-03
Coverage Spatial
Singapore