Data characterization, treatment, and cleaning are necessary (though not always glamorous) components of machine learning and data science projects. While there is no substitute for getting your hands dirty in the data, there are many data issues that repeat from project to project. In particular, there are pitfalls in properly dealing with missing data values, previously unobserved categorical values, and high-cardinality categorical variables.

In this talk, we will discuss using the vtreat package to prepare data for supervised machine learning. We will demonstrate vtreat on a real-world data set, with xgboost and Pandas. Vtreat automates the statistically sound treatment of common data problems, leaving the data scientist free to concentrate on problem-specific data and modeling issues.