Dual-modality endoscopic probe for tissue surface shape reconstruction and hyperspectral imaging enabled by deep neural networks.
File(s)MedIA2018_revision_v3 submitted.docx (2.74 MB)
Accepted version
Author(s)
Type
Journal Article
Abstract
Surgical guidance and decision making could be improved with accurate and real-time measurement of intra-operative data including shape and spectral information of the tissue surface. In this work, a dual-modality endoscopic system has been proposed to enable tissue surface shape reconstruction and hyperspectral imaging (HSI). This system centers around a probe comprised of an incoherent fiber bundle, whose fiber arrangement is different at the two ends, and miniature imaging optics. For 3D reconstruction with structured light (SL), a light pattern formed of randomly distributed spots with different colors is projected onto the tissue surface, creating artificial texture. Pattern decoding with a Convolutional Neural Network (CNN) model and a customized feature descriptor enables real-time 3D surface reconstruction at approximately 12 frames per second (FPS). In HSI mode, spatially sparse hyperspectral signals from the tissue surface can be captured with a slit hyperspectral imager in a single snapshot. A CNN based super-resolution model, namely "super-spectral-resolution" network (SSRNet), has also been developed to estimate pixel-level dense hypercubes from the endoscope cameras standard RGB images and the sparse hyperspectral signals, at approximately 2 FPS. The probe, with a 2.1 mm diameter, enables the system to be used with endoscope working channels. Furthermore, since data acquisition in both modes can be accomplished in one snapshot, operation of this system in clinical applications is minimally affected by tissue surface movement and deformation. The whole apparatus has been validated on phantoms and tissue (ex vivo and in vivo), while initial measurements on patients during laryngeal surgery show its potential in real-world clinical applications.
Date Issued
2018-06-15
Date Acceptance
2018-06-07
Citation
Medical Image Analysis, 2018, 48, pp.162-176
ISSN
1361-8415
Publisher
Elsevier
Start Page
162
End Page
176
Journal / Book Title
Medical Image Analysis
Volume
48
Copyright Statement
© 2018 Elsevier Ltd. All rights reserved. This manuscript is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International http://creativecommons.org/licenses/by-nc-nd/4.0/
Sponsor
Commission of the European Communities
Imperial College Healthcare NHS Trust- BRC Funding
Imperial College Healthcare NHS Trust- BRC Funding
Engineering & Physical Science Research Council (E
Deutsche Forschungsgemeinschaft ( German Research
Cancer Research UK
Identifier
https://www.ncbi.nlm.nih.gov/pubmed/29933116
PII: S1361-8415(18)30373-6
Grant Number
242991
RDB04 79560
RD207
EP/N50869X/1
637960
C24523/A25147
Subjects
3D reconstruction
Deep learning
Hyperspectral imaging
Intra-operative imaging
Structured light
Super-spectral-resolution
Publication Status
Published
Coverage Spatial
Netherlands