Hyperparameter optimization (HPO) is crucial for getting the best performance possible out of your machine learning models. BOHB (Bayesian Optimization and Hyperband) is a recently developed algorithm that combines the best parts of two popular approaches to the HPO problem. It allows for very flexible declaration of the hyperparameter configuration space, parallel search across computational resources, and large numbers of hyperparameters. Best yet, there is a fantastic open source implementation in the Python package hpbandster.

Bayesian optimization methods create a model of the function that maps hyperparameter configurations to model performance. They use this model to choose new hyperparameter configurations to test and refine the model with the result. These methods focus on configuration selection.

In contrast, Hyperband is a bandit strategy that focuses on configuration evaluation. It uses an adaptive multi-resolution approach to get quick and dirty estimates of many more hyperparameter configurations than is possible in the Bayesian optimization framework. It uses this greater speed of evaluation to determine what seems promising and what to rule out as it randomly searches through configuration space. This results in a fast, flexible, and parallelizable algorithm.

BOHB fuses these two approaches by taking the adaptive multi-resolution evaluation strategy of Hyperband, but replacing the random search component with a principaled configuration selection algorithm based on Bayesian kernel density estimates. This talk will describe the algorithm in further detail, and walk through the open source Python package hpbandster to see how you can deploy BOHB to maximize the performance of your machine learning models.