Can non-specialists provide high quality gold standard labels in challenging modalities?
OA Location
Author(s)
Type
Chapter
Abstract
Probably yes.—Supervised Deep Learning dominates performance scores for many computer vision tasks and defines the state-of-the-art. However, medical image analysis lags behind natural image applications. One of the many reasons is the lack of well annotated medical image data available to researchers. One of the first things researchers are told is that we require significant expertise to reliably and accurately interpret and label such data. We see significant inter- and intra-observer variability between expert annotations of medical images. Still, it is a widely held assumption that novice annotators are unable to provide useful annotations for use by clinical Deep Learning models. In this work we challenge this assumption and examine the implications of using a minimally trained novice labelling workforce to acquire annotations for a complex medical image dataset. We study the time and cost implications of using novice annotators, the raw performance of novice annotators compared to gold-standard expert annotators, and the downstream effects on a trained Deep Learning segmentation model’s performance for detecting a specific congenital heart disease (hypoplastic left heart syndrome) in fetal ultrasound imaging.
Editor(s)
Albarqouni, S
Cardoso, MJ
Dou, Q
Kamnitsas, K
Khanal, B
Rekik, I
Rieke, N
Sheet, D
Tsaftaris, S
Xu, D
Xu, Z
Date Issued
2021-09-21
Citation
Domain Adaptation and Representation Transfer, and Affordable Healthcare and AI for Resource Diverse Global Health, 2021, 12968, pp.251-262
ISBN
978-3-030-87721-7
Publisher
Springer Nature Switzerland AG
Start Page
251
End Page
262
Journal / Book Title
Domain Adaptation and Representation Transfer, and Affordable Healthcare and AI for Resource Diverse Global Health
Lecture Notes in Computer Science
Volume
12968
Copyright Statement
© 2021 Springer Nature Switzerland AG.
Identifier
https://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=PARTNER_APP&SrcAuth=LinksAMR&KeyUT=WOS:000722283000023&DestLinkType=FullRecord&DestApp=ALL_WOS&UsrCustomerID=a2bf6146997ec60c407a63945d4e92bb
Subjects
Annotations
Computer Science
Computer Science, Interdisciplinary Applications
Expert
Labels
Life Sciences & Biomedicine
Mathematical & Computational Biology
Novice
Science & Technology
Technology
Publication Status
Published
Date Publish Online
2021-09-21