Uncertainty-aware classifier accuracy prediction under prior probability shift

Classifier accuracy prediction (CAP) is the task of estimat- ing the accuracy that a trained classifier will have on a set of datapoints characterized by dataset shift, i.e., data sampled from a distribution dif- ferent from the distribution from where the training data originated. We here propose three new algorithms for CAP under prior probability shift, an important type of dataset shift; our first two algorithms are built on top of surrogate quantifiers, while the third is a Bayesian algorithm. Im- portantly, these methods do not merely return a point estimate (i.e., the estimated accuracy of the classifier on the unlabelled data), but also re- turn a measure of their uncertainty in this estimate. Systematic exper- iments on a number of datasets and against a number of baseline CAP algorithms show that our methods are competitive with strong baselines, with one of them frequently achieving the best balance between point estimation accuracy and interval quality.