Hierarchical probabilistic models are an expressive and flexible way to build models that allow us to incorporate feature-dependent uncertainty and produce a range of credible values. We will walk through the process of applying one of these models to predict a customer’s budget from realized spend data using PyMC3.
To introduce the topic, we will provide a background on probabilistic modelling and how hierarchical models take advantage of the probabilistic framework. We will cover some well-known applications such as Raudenbush and Bryk’s (1985) on school effectiveness and Rossi et al.’s (1995) in marketing. We will then provide an introduction to PyMC3 and its syntax for defining deterministic and estimated parameters.
We will provide participants a simulated dataset structured to be used in the hierarchical application. The data will be at the customer level and will include two main levels for the hierarchy; industry and region. It will also contain a set of covariates and the realized spend. Participants will be expected to set up their python environments ahead of time and we will assist participants in loading the data. We will walk through several model designs including fully-pooled (non-hierarchical), fixed-effect model (i.e. region and industry intercepts) and random-effect model (i.e. region and industry-varying parameters).
Participants will run their own sampling processes and we will review the trace plots. We will show how to diagnose issues like divergence in the sampling process and typical methods for avoiding these issues (e.g. running a longer trace, multiple chains). We will also run posterior predictive checks and compare the resulting distributions to the actual/expected outcomes. We will discuss methods for interpreting the results for business stakeholders such as using the predicted ranges to estimate customer value.
This last part of the tutorial will be more open ended. We will share thoughts and experience with productionizing these models, allowing participants to contribute as well. We will then provide an outline of next step, including using libraries like Edward to implement deep probabilistic approaches in applications like the one presented in the tutorial.
Provide participants with probabilistic modeling skill: By the end of the tutorial, participants will have designed and applied a probabilistic model in the Python ecosystem. These models provide ranges to parameter estimates and posterior predictions that are readily interpretable. With increasing focus on interpretability in the machine learning field, this skillset will likely prove useful in participants’ work.
Pilot a method for indirect prediction of a business-relevant outcome: The focus of the tutorial will be to predict customer budget from realized spend. This is a difficult problem for traditional machine learning approaches, as the measured outcome is a fraction of the desired prediction. This is also an obstacle frequently encountered with real-world datasets (e.g. signal processing, survival analysis). The tutorial will provide one possible method for addressing this issue and quantifying uncertainty around predictions.
Add to the discussion about productionizing probabilistic models: While probabilistic models interpretability and quantification of uncertainty are attractive features, they are frequently not used in production environments. In the last portion of the tutorial, we will discuss barriers to productionizing these models and how these barriers may be overcome.
The target audience for this tutorial will have a background in statistics and machine learning and a basic understanding of Bayesian estimation. We will provide some theoretical grounding for the model, but the focus will be more on practical application. Experience of working with probabilistic models is not necessary. If they’ve encountered barriers to using them in their work or have had difficulty finding an appropriate application, we believe they will leave the tutorial much better equipped to again engage with these powerful methods.