As data scientist, my first instinct is to focus on using cool new packages to create incredibly powerful predictive models. However, I learned from hard experience that our most exciting models in development often totally failed us in production. Not taking into account the needs of a production system when developing our models caused us to lose lots of money and me to lose lots of sleep.

In this talk, I will outline the different ways that I failed while building production models, and how I learned to avoid these failures in the future. With relatively small changes to how we develop and evaluate models, we can build data science systems that we can depend on in high-stakes environments. Well suited to anyone wanting to go from one off analysis to automated data science, or who is tired of feeling nervous about what their models are doing.

I will give particular attention to:

• creating diagnostics that provide richer insight than a single accuracy score
• building models that are easy to monitor and maintain
• evaluating models within the wider context they operate in