Scale Matters: on the many uses of calibration in machine learning
Peter Flach
Professor
University of Bristol
Abstract:
Calibration is the process of adjusting measurements to a standard
scale. In machine learning it is most commonly understood in relation to the
class probability estimates of a probabilistic classifier: we say that a
classifier is well-calibrated if among all instances receiving a probability
estimate p for a particular class, the proportion of instances having the
class in question is approximately p. The advantage of a well-calibrated
classifier is that near-optimal decision thresholds can be directly derived
from the operating condition (class and cost distribution). In this talk I
explore various methods for classifier calibration, including the isotonic
regression method that relates to ROC analysis. I will discuss how these
methods can be applied to single features, resulting in a very general
framework in which features carry class information and categorical features
can be turned into real-valued ones and vice versa. I will also discuss an
alternative notion of calibration whereby a classifier's score quantifies
the proportion of positive predictions it makes at that threshold. I will
introduce the ROL curve, a close companion of ROC curves that allow to
quantify the loss at a particular predicted positive rate. Rate-calibrated
classifiers have an expected loss that is linearly related to AUC, which
vindicates AUC as a coherent measure of classification performance (contrary
to recent claims in the literature).
Bio:
Professor Flach publishes widely and on a broad range of subjects. He
is an internationally leading researcher in the areas of mining highly
structured data and the evaluation and improvement of machine learning
models using ROC analysis. He has also published on the logic and philosophy
of machine learning, and on the combination of logic and probability. He is
author of Simply Logical: Intelligent Reasoning by Example (John Wiley,
1994) and Machine Learning: the Art and Science of Algorithms that Make
Sense of Data (Cambridge University Press, 2012). Prof Flach is currently
the Editor-in-Chief of the Machine Learning journal, one of the two top
journals in the field that has been published for 25 years by Kluwer and now
Springer. He was Programme Co-Chair of the 1999 International Conference on
Inductive Logic Programming, the 2001 European Conference on Machine
Learning, and the 2009 ACM Conference on Knowledge Discovery and Data
Mining. In 2012 he will co-chair the European Conference on Machine Learning
and Principles and Practice of Knowledge Discovery.