Miracle of Regularization: A Robust Optimization Perspective
Huan Xu
Post-Doc research fellow
Department of Electrical and Computer Engineering, University of Texas at Austin, USA.
Abstract :
In Machine Learning, regularization is widely used to control overfitting. Its success is usually interpreted as coming from penalizing the solution complexity. In this talk, we explain it from a robust optimization perspective. That is, assuming that each sample has certain disturbance, we find the best decision under the most adversarial disturbance. We show that this recovers the solution obtained by penalizing complexity via regularization. SVM and Lasso are investigated in detail. This equivalence relationship between regularization and robustness gives a physical interpretation of the regularization process. Moreover, we are able to explain consistency and sparseness from a robustness perspective.
Bio:
Huan Xu obtained his Ph. D. degree in the Department of Electrical and Computer Engineering of McGill University under the supervision of Prof. Shie Mannor in Aug 2009. He is now a Post-Doc research fellow in the Department of Electrical and Computer Engineering of the University of Texas at Austin working with Prof. Constantine Caramanis. His main research interest lies in the intersection of optimization, control and statistical learning.