A/B testing and other related experimental designs are familiar ways to determine the effect of an intervention or change. They do this by comparing a treatment group with a randomized control group. But often, we want to measure effects in cases where it's not possible to have a control group or to limit the exposure of the intervention. Some example scenarios include attributing sales to a national TV campaign, estimating the cost of a service outage, assessing the business impact of a PR event, and quantifying the public health consequences of a policy change.

In these situations, we can often still make a probabilistic inference about the effect by building a predictive model for what might have happened in the absence of the intervention (a counterfactual). In particular, the above examples all deal with time series data. We can take advantage of that property by using standard time series modeling techniques to create a baseline prediction, and comparing it with what we observed. This method is called interrupted time series analysis (ITS), and it has broad applications to real world data.

In this talk, we'll discuss the structure and principles of ITS analysis, and go over common use cases. We'll work through an implementation on an example data set and consider various practical considerations and modeling choices. We'll also address some caveats. ITS is considered a quasi-experimental method because the "control" is inferred, not observed. As a result, its estimates can only be trusted under certain assumptions. We'll discuss the main threats to validity for an ITS analysis and look at ways to mitigate them.

Interrupted time series analysis is a powerful and useful tool when used appropriately, and this talk aims to provide a theoretical and practical introduction to its application.