Sunday 9:45 AM–10:30 AM in Room 1

Implementing distributed grid search for deep learning using scikit-learn and joblib

Mike Heilman

Audience level:


Grid search over hyperparameters is an important but computationally expensive process in machine learning, particularly for deep learning and tree ensembles. In this talk, I will describe how one can use joblib's recently added custom backend functionality to do distributed grid search on Amazon EC2 for a TensorFlow deep text classifier that follows the scikit-learn estimator API.


While sophisticated machine learning approaches such as deep learning and tree ensembles often produce excellent predictive models, they also typically require performing computationally expensive searches over hyperparameter settings (e.g., the number, sizes, and types of hidden layers for a neural network). It is desirable to parallelize this search across multiple machines, but such functionality is not readily available in scikit-learn. However, scikit-learn does provide functionality to parallelize grid search across multiple processes on a single machine, using the joblib package. Also, joblib recently added functionality that allows users to implement custom parallel backends to support parallel computation on their own infrastructure. In this talk, I will describe the implementation of a custom joblib backend for an EC2-based auto-scaling computation engine. I will also describe applying this backend to the problem of grid searching deep learning text classifiers, using TensorFlow models that follow the scikit-learn estimator API.