Image
Image
Image
Image
Image
Image
Image
Image
Image
Image



Search
»

Seminar abstract

Scale Matters: on the many uses of calibration in machine learning

Peter Flach
Professor
University of Bristol


Abstract: Calibration is the process of adjusting measurements to a standard scale. In machine learning it is most commonly understood in relation to the class probability estimates of a probabilistic classifier: we say that a classifier is well-calibrated if among all instances receiving a probability estimate p for a particular class, the proportion of instances having the class in question is approximately p. The advantage of a well-calibrated classifier is that near-optimal decision thresholds can be directly derived from the operating condition (class and cost distribution). In this talk I explore various methods for classifier calibration, including the isotonic regression method that relates to ROC analysis. I will discuss how these methods can be applied to single features, resulting in a very general framework in which features carry class information and categorical features can be turned into real-valued ones and vice versa. I will also discuss an alternative notion of calibration whereby a classifier's score quantifies the proportion of positive predictions it makes at that threshold. I will introduce the ROL curve, a close companion of ROC curves that allow to quantify the loss at a particular predicted positive rate. Rate-calibrated classifiers have an expected loss that is linearly related to AUC, which vindicates AUC as a coherent measure of classification performance (contrary to recent claims in the literature).

Bio: Professor Flach publishes widely and on a broad range of subjects. He is an internationally leading researcher in the areas of mining highly structured data and the evaluation and improvement of machine learning models using ROC analysis. He has also published on the logic and philosophy of machine learning, and on the combination of logic and probability. He is author of Simply Logical: Intelligent Reasoning by Example (John Wiley, 1994) and Machine Learning: the Art and Science of Algorithms that Make Sense of Data (Cambridge University Press, 2012). Prof Flach is currently the Editor-in-Chief of the Machine Learning journal, one of the two top journals in the field that has been published for 25 years by Kluwer and now Springer. He was Programme Co-Chair of the 1999 International Conference on Inductive Logic Programming, the 2001 European Conference on Machine Learning, and the 2009 ACM Conference on Knowledge Discovery and Data Mining. In 2012 he will co-chair the European Conference on Machine Learning and Principles and Practice of Knowledge Discovery.
  Name Size

Image
PoweredBy © LAMDA, 2022