Deploying Machine Learning models, is fundamentally about bridging the gap between the research and production environment. In the research environment, the data scientists develop the models and evaluate the benefit it will bring to the organisation. In the production environment, software developers re-build these models to make them available to both internal and external systems. Reproducibility of the models between the two environments is key to maximise the benefit of using the ML model.

What is reproducibility for Machine Learning? We define reproducibility as the ability to duplicate a model exactly such that given the same raw data as input, both models return the same output. This means that, models in the research and production environment should return the same prediction for the same input data. There are a series of measures in place to ensure reproducibility during research and deployment. However, before fully integrating our models, we typically run a battery of testing and reconciliation processes to confirm reproducibility.

In this talk, we will introduce various tests used in organisations to confirm that the models in the research and production environment are equivalent. We will cover the tests for reproducibility of both the feature engineering steps and the final model predictions. We will also cover the tests performed when a model is updated, including benchmark testing, to evaluate training time and API response latency. In our projects, we use python and the common python numerical libraries.