Bayesian analysis of the prevalence bias: learning and predicting from
imbalanced data
imbalanced data
File(s)
Working Paper
Author(s)
Type
Working Paper
Abstract
Datasets are rarely a realistic approximation of the target population. Say,
prevalence is misrepresented, image quality is above clinical standards, etc.
This mismatch is known as sampling bias. Sampling biases are a major hindrance
for machine learning models. They cause significant gaps between model
performance in the lab and in the real world. Our work is a solution to
prevalence bias. Prevalence bias is the discrepancy between the prevalence of a
pathology and its sampling rate in the training dataset, introduced upon
collecting data or due to the practioner rebalancing the training batches. This
paper lays the theoretical and computational framework for training models, and
for prediction, in the presence of prevalence bias. Concretely a bias-corrected
loss function, as well as bias-corrected predictive rules, are derived under
the principles of Bayesian risk minimization. The loss exhibits a direct
connection to the information gain. It offers a principled alternative to
heuristic training losses and complements test-time procedures based on
selecting an operating point from summary curves. It integrates seamlessly in
the current paradigm of (deep) learning using stochastic backpropagation and
naturally with Bayesian models.
prevalence is misrepresented, image quality is above clinical standards, etc.
This mismatch is known as sampling bias. Sampling biases are a major hindrance
for machine learning models. They cause significant gaps between model
performance in the lab and in the real world. Our work is a solution to
prevalence bias. Prevalence bias is the discrepancy between the prevalence of a
pathology and its sampling rate in the training dataset, introduced upon
collecting data or due to the practioner rebalancing the training batches. This
paper lays the theoretical and computational framework for training models, and
for prediction, in the presence of prevalence bias. Concretely a bias-corrected
loss function, as well as bias-corrected predictive rules, are derived under
the principles of Bayesian risk minimization. The loss exhibits a direct
connection to the information gain. It offers a principled alternative to
heuristic training losses and complements test-time procedures based on
selecting an operating point from summary curves. It integrates seamlessly in
the current paradigm of (deep) learning using stochastic backpropagation and
naturally with Bayesian models.
Date Issued
2021-12-07
Citation
2021
Publisher
ArXiv
Copyright Statement
©2021 The Author(s)
Identifier
http://arxiv.org/abs/2108.00250v1
Subjects
cs.LG
cs.LG
q-bio.QM
stat.AP
stat.ME
stat.ML
Publication Status
Published