TutORial: Bayesian Optimization

By Peter Frazier.

Bayesian optimization is widely used for tuning deep neural networks and optimizing other black-box objective functions that take a long time to evaluate. In this tutorial, we describe how Bayesian optimization works, including the Bayesian machine learning model it uses to model the objective function, Gaussian process regression, and three common acquisition functions: expected improvement, entropy search, and knowledge gradient. We then describe applications at Yelp and Uber, explain techniques important for making it work well in practice, and survey techniques for solving "exotic" Bayesian optimization problems.