SemanticFusion: dense 3D semantic mapping with convolutional neural networks

File Description SizeFormat 
Semanticfusion_ICRA17_Final.pdfAccepted version3.38 MBAdobe PDFView/Open
Title: SemanticFusion: dense 3D semantic mapping with convolutional neural networks
Authors: McCormac, J
Handa, A
Davison, AJ
Leutenegger, S
Item Type: Conference Paper
Abstract: Ever more robust, accurate and detailed mapping using visual sensing has proven to be an enabling factor for mobile robots across a wide variety of applications. For the next level of robot intelligence and intuitive user interaction, maps need to extend beyond geometry and appearance — they need to contain semantics. We address this challenge by combining Convolutional Neural Networks (CNNs) and a state-of-the-art dense Simultaneous Localization and Mapping (SLAM) system, ElasticFusion, which provides long-term dense correspondences between frames of indoor RGB-D video even during loopy scanning trajectories. These correspondences allow the CNN's semantic predictions from multiple view points to be probabilistically fused into a map. This not only produces a useful semantic 3D map, but we also show on the NYUv2 dataset that fusing multiple predictions leads to an improvement even in the 2D semantic labelling over baseline single frame predictions. We also show that for a smaller reconstruction dataset with larger variation in prediction viewpoint, the improvement over single frame segmentation increases. Our system is efficient enough to allow real-time interactive use at frame-rates of ≈25Hz.
Issue Date: 24-Jul-2017
Date of Acceptance: 25-Feb-2017
ISBN: 978-1-5090-4633-1
Publisher: IEEE
Journal / Book Title: Robotics and Automation (ICRA), 2017 IEEE International Conference on
Copyright Statement: © 2017 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.
Sponsor/Funder: Dyson Technology Limited
Funder's Grant Number: PO 4500285622
Conference Name: IEEE International Conference on Robotics and Automation (ICRA), 2017
Publication Status: Published
Start Date: 2017-05-29
Finish Date: 2017-06-03
Conference Place: Singapore
Appears in Collections:Computing

Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.

Creative Commons