Reverse classification accuracy: predicting segmentation performance in the absence of ground truth

File Description SizeFormat 
rca.pdfAccepted version999.12 kBAdobe PDFView/Open
07902121.pdfPublished version1.67 MBAdobe PDFView/Open
Title: Reverse classification accuracy: predicting segmentation performance in the absence of ground truth
Authors: Valindria, V
Lavdas, I
Bai, W
Kamnitsas, K
Aboagye, E
Rockall, A
Rueckert, D
Glocker, B
Item Type: Journal Article
Abstract: When integrating computational tools such as au- tomatic segmentation into clinical practice, it is of utmost importance to be able to assess the level of accuracy on new data, and in particular, to detect when an automatic method fails. However, this is difficult to achieve due to absence of ground truth. Segmentation accuracy on clinical data might be different from what is found through cross-validation because validation data is often used during incremental method development, which can lead to overfitting and unrealistic performance expectations. Before deployment, performance is quantified using different metrics, for which the predicted segmentation is compared to a reference segmentation, often obtained manually by an expert. But little is known about the real performance after deployment when a reference is unavailable. In this paper, we introduce the concept of reverse classification accuracy (RCA) as a framework for predicting the performance of a segmentation method on new data. In RCA we take the predicted segmentation from a new image to train a reverse classifier which is evaluated on a set of reference images with available ground truth. The hypothesis is that if the predicted segmentation is of good quality, then the reverse classifier will perform well on at least some of the reference images. We validate our approach on multi-organ segmentation with different classifiers and segmentation methods. Our results indicate that it is indeed possible to predict the quality of individual segmentations, in the absence of ground truth. Thus, RCA is ideal for integration into automatic processing pipelines in clinical routine and as part of large-scale image analysis studies.
Issue Date: 17-Apr-2017
Date of Acceptance: 28-Jan-2017
ISSN: 1558-254X
Publisher: Institute of Electrical and Electronics Engineers (IEEE)
Start Page: 1597
End Page: 1606
Journal / Book Title: IEEE Transactions on Medical Imaging
Volume: 36
Issue: 8
Copyright Statement: This work is licensed under a Creative Commons Attribution 3.0 License. For more information, see
Sponsor/Funder: National Institute for Health Research
Engineering & Physical Science Research Council (EPSRC)
Funder's Grant Number: EME/13/122/01
Keywords: Science & Technology
Life Sciences & Biomedicine
Computer Science, Interdisciplinary Applications
Engineering, Biomedical
Engineering, Electrical & Electronic
Imaging Science & Photographic Technology
Radiology, Nuclear Medicine & Medical Imaging
Computer Science
image segmentation
machine learning
performance evaluation
08 Information And Computing Sciences
09 Engineering
Nuclear Medicine & Medical Imaging
Publication Status: Published
Appears in Collections:Faculty of Engineering
Division of Surgery
Division of Cancer
Faculty of Medicine

Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.

Creative Commonsx