On Generalization and Implicit Bias ofGradient Methods in Deep Learning¶
Jian Li
Associate Professor
Institute for Interdisciplinary, Tsinghua University
Abstract: Deep learning has enjoyed huge empirical success in recent years. Although training a deepneural network is a highly non-convex optimization problem, simple (stochastic) gradientmethods are able to produce good solutions that minimize the training error, and moresurprisingly, can generalize well to out-of sample data, even when the number of parametersis significantly larger than the amount of training data. It is known that changing theoptimization algorithm, even without changing the model, changes the implicit bias, and alsothe generalization properties. What is the bias introduced by the optimization algorithms forneural networks? What ensures generalization in neural networks? In this talk, we attempt toanswer the above questions by proving new generalization bounds and investigating theimplicit bias of various gradient methods.
Bio: Jian Li is currently an associate professor at Institute for Interdisciplinary Information Sciences(NIS, previously ITCS), Tsinghua University, headed by Prof. Andrew Yao. He got his BScdegree from Sun Yat-sen (Zhongshan) University, China, MSc degree in computer sciencefrom Fudan University, China and PhD degree in the University of Maryland, USA. His majorresearch interests lie in algorithm design and analysis, machine learning, databases andfinance. He co-authored several research papers that have been published in major computerscience conferences and journals. He received the best paper awards at VLDB 2009 and ESA2010. He is also a recipient of the "221 Basic Research Plan for Young Faculties" at TsinghuaUniversity, the "new century excellent talents award" by Ministry of Education of China, andthe National Science Fund for Excellent Young Scholars.