Fisher consistency for prior probability shift
File(s)
Author(s)
Tasche, D
Type
Working Paper
Abstract
We introduce Fisher consistency in the sense of unbiasedness as a criterion
to distinguish potentially suitable and unsuitable estimators of prior class
probabilities in test datasets under prior probability and more general dataset
shift. The usefulness of this unbiasedness concept is demonstrated with three
examples of classifiers used for quantification: Adjusted Classify & Count,
EM-algorithm and CDE-Iterate. We find that Adjusted Classify & Count and
EM-algorithm are Fisher consistent. A counter-example shows that CDE-Iterate is
not Fisher consistent and, therefore, cannot be trusted to deliver reliable
estimates of class probabilities.
to distinguish potentially suitable and unsuitable estimators of prior class
probabilities in test datasets under prior probability and more general dataset
shift. The usefulness of this unbiasedness concept is demonstrated with three
examples of classifiers used for quantification: Adjusted Classify & Count,
EM-algorithm and CDE-Iterate. We find that Adjusted Classify & Count and
EM-algorithm are Fisher consistent. A counter-example shows that CDE-Iterate is
not Fisher consistent and, therefore, cannot be trusted to deliver reliable
estimates of class probabilities.
Date Issued
2017-01-19
Citation
2017
Copyright Statement
© 2017 The Author
Identifier
http://arxiv.org/abs/1701.05512v1
Subjects
stat.ML
cs.LG
stat.CO
62C10
Notes
26 pages, 2 figures, 8 tables