Reverse classification accuracy: predicting segmentation performance in the absence of ground truth
File(s)
Accepted version
Published version
Author(s)
Type
Journal Article
Abstract
When integrating computational tools such as au-
tomatic segmentation into clinical practice, it is of utmost
importance to be able to assess the level of accuracy on new
data, and in particular, to detect when an automatic method
fails. However, this is difficult to achieve due to absence of ground
truth. Segmentation accuracy on clinical data might be different
from what is found through cross-validation because validation
data is often used during incremental method development, which
can lead to overfitting and unrealistic performance expectations.
Before deployment, performance is quantified using different
metrics, for which the predicted segmentation is compared to
a reference segmentation, often obtained manually by an expert.
But little is known about the real performance after deployment
when a reference is unavailable. In this paper, we introduce the
concept of
reverse classification accuracy
(RCA) as a framework
for predicting the performance of a segmentation method on
new data. In RCA we take the predicted segmentation from
a new image to train a reverse classifier which is evaluated
on a set of reference images with available ground truth. The
hypothesis is that if the predicted segmentation is of good quality,
then the reverse classifier will perform well on at least some of
the reference images. We validate our approach on multi-organ
segmentation with different classifiers and segmentation methods.
Our results indicate that it is indeed possible to predict the quality
of individual segmentations, in the absence of ground truth. Thus,
RCA is ideal for integration into automatic processing pipelines in
clinical routine and as part of large-scale image analysis studies.
tomatic segmentation into clinical practice, it is of utmost
importance to be able to assess the level of accuracy on new
data, and in particular, to detect when an automatic method
fails. However, this is difficult to achieve due to absence of ground
truth. Segmentation accuracy on clinical data might be different
from what is found through cross-validation because validation
data is often used during incremental method development, which
can lead to overfitting and unrealistic performance expectations.
Before deployment, performance is quantified using different
metrics, for which the predicted segmentation is compared to
a reference segmentation, often obtained manually by an expert.
But little is known about the real performance after deployment
when a reference is unavailable. In this paper, we introduce the
concept of
reverse classification accuracy
(RCA) as a framework
for predicting the performance of a segmentation method on
new data. In RCA we take the predicted segmentation from
a new image to train a reverse classifier which is evaluated
on a set of reference images with available ground truth. The
hypothesis is that if the predicted segmentation is of good quality,
then the reverse classifier will perform well on at least some of
the reference images. We validate our approach on multi-organ
segmentation with different classifiers and segmentation methods.
Our results indicate that it is indeed possible to predict the quality
of individual segmentations, in the absence of ground truth. Thus,
RCA is ideal for integration into automatic processing pipelines in
clinical routine and as part of large-scale image analysis studies.
Date Issued
2017-04-17
Date Acceptance
2017-01-28
Citation
IEEE Transactions on Medical Imaging, 2017, 36 (8), pp.1597-1606
ISSN
1558-254X
Publisher
Institute of Electrical and Electronics Engineers (IEEE)
Start Page
1597
End Page
1606
Journal / Book Title
IEEE Transactions on Medical Imaging
Volume
36
Issue
8
Copyright Statement
This work is licensed under a Creative Commons Attribution 3.0 License. For more information, see http://creativecommons.org/licenses/by/3.0/
License URL
Sponsor
National Institute for Health Research
Engineering & Physical Science Research Council (EPSRC)
Grant Number
EME/13/122/01
EP/N023668/1
Subjects
Science & Technology
Technology
Life Sciences & Biomedicine
Computer Science, Interdisciplinary Applications
Engineering, Biomedical
Engineering, Electrical & Electronic
Imaging Science & Photographic Technology
Radiology, Nuclear Medicine & Medical Imaging
Computer Science
Engineering
Abdominal
classification
image segmentation
machine learning
MRI
performance evaluation
MULTI-ATLAS SEGMENTATION
IMAGE SEGMENTATION
LESION SEGMENTATION
VALIDATION
QUALITY
FORESTS
cs.CV
08 Information And Computing Sciences
09 Engineering
Nuclear Medicine & Medical Imaging
Publication Status
Published