2009年10月30日（星期五）14:30-15:30，蒙民伟楼404会议室
A framework for monitoring classifiers' performance: when and why failure occurs?¶
Nitesh Chawla
Assistant Professor
Computer Science and Engineering Department, University of Notre Dame, USA
Abstract :
Classifier error is the product of model bias and data variance. While understanding the bias involved when selecting a given learning algorithm, it is similarly important to understand the variability in data over time, since even the One True Model might perform poorly when training and evaluation samples diverge. Thus, the ability to identify distributional divergence is critical towards pinpointing when fracture points in classifier performance will occur. Contemporary evaluation methods do not take the impact of distribution shifts on the quality of classifiers’ predictions. In this talk, I present a comprehensive framework to proactively detect breakpoints in
classifiers’ predictions and shifts in data distributions through a series of statistical tests. I outline and utilize three scenarios under which data changes: sample selection bias, covariate shift, and shifting class priors.
Bio:
Nitesh Chawla is an Assistant Professor in the Department of Computer Science and Engineering at the University of Notre Dame. He directs the Data Inference Analysis and Learning Lab (DIAL) and co-directs the Interdisciplinary Center of the Network Science and Applications (iCenSA) at Notre Dame. His research is primarily focused on machine learning, data mining, and complex networks. His work has led to applications in various domains including climate data sciences, biology, medicine, finance, security, and social science. He is on the editorial board of IEEE Transactions on Systems, Man and Cybernetics Part B, and has served/serving on the program and organizational committees for a number of top-tier conferences. He has received various awards and honors, including the best dissertation, best papers, outstanding undergraduate teacher, and the NAE New Faculty Fellowship. His current research is supported form NSF, DOD, NWICG, NIJ, and industry sponsors.