Skip to main content
Colloquia
event
“Towards Understanding First Order Algorithms for Nonconvex Optimization in Machine Learning”
Add to Calendar 2019-11-21T20:30:00 2019-11-21T21:30:00 UTC “Towards Understanding First Order Algorithms for Nonconvex Optimization in Machine Learning” 201 Thomas Building
Start DateThu, Nov 21, 2019
3:30 PM
to
End DateThu, Nov 21, 2019
4:30 PM
Presented By
Tuo Zhao,Georgia Institute of Technology
Event Series:

Stochastic Gradient Descent-type (SGD) algorithms have been widely applied to many non-convex optimization problems in machine learning, e.g., training deep neural networks, variational Bayesian inference and collaborative filtering. Due to current technical limit, however, establishing convergence properties of SGD for these highly complicated practical non-convex problems is generally infeasible. Therefore, we propose to analyze the behavior of the SGD-type algorithms through two simpler but non-trivial non-convex problems – (1) Streaming Principal Component Analysis and (2) Training Non-overlapping Two-layer Convolutional Neural Networks. Specifically, we prove that for both examples, SGD attains a sub-linear rate of convergence to the global optima with high probability. Our theory not only helps us better understand SGD, but also provides new insights on more complicated non-convex optimization problems in machine learning.