Estimating model performance under domain shifts with class-specific confidence scores
File(s)
Accepted version
Author(s)
Li, Zeju
Kamnitsas, Konstantinos
Islam, Mobarakol
Chen, Chen
Glocker, Ben
Type
Conference Paper
Abstract
Machine learning models are typically deployed in a test setting that differs from the training setting, potentially leading to decreased model performance because of domain shift. If we could estimate the performance that a pre-trained model would achieve on data from a specific deployment setting, for example a certain clinic, we could judge whether the model could safely be deployed or if its performance degrades unacceptably on the specific data. Existing approaches estimate this based on the confidence of predictions made on unlabeled test data from the deployment’s domain. We find existing methods struggle with data that present class imbalance, because the methods used to calibrate confidence do not account for bias induced by class imbalance, consequently failing to estimate class-wise accuracy. Here, we introduce class-wise calibration within the framework of performance estimation for imbalanced datasets. Specifically, we derive class-specific modifications of state-of-the-art confidence-based model evaluation methods including temperature scaling (TS), difference of confidences (DoC), and average thresholded confidence (ATC). We also extend the methods to estimate Dice similarity coefficient (DSC) in image segmentation. We conduct experiments on four tasks and find the proposed modifications consistently improve the estimation accuracy for imbalanced datasets. Our methods improve accuracy estimation by 18% in classification under natural domain shifts, and double the estimation accuracy on segmentation tasks, when compared with prior methods (Code is available at https://github.com/ZerojumpLine/ModelEvaluationUnderClassImbalance).
Date Issued
2022-09-17
Date Acceptance
2022-09-01
Citation
2022, pp.693-703
ISBN
9783031164484
ISSN
0302-9743
Publisher
Springer Nature Switzerland
Start Page
693
End Page
703
Copyright Statement
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG. The final publication is available at Springer via https://doi.org/10.1007/978-3-031-16449-1_66
Sponsor
Commission of the European Communities
Identifier
https://link.springer.com/chapter/10.1007/978-3-031-16449-1_66
Grant Number
H2020 - 757173
Source
MICCAI 2022 25th International Conference
Subjects
cs.CV
cs.CV
cs.LG
Artificial Intelligence & Image Processing
Publication Status
Published
Start Date
2022-09-18
Finish Date
2022-09-22
Coverage Spatial
Singapore
Date Publish Online
2022-09-17