While A/B testing can be a powerful means of comparing the performance of two variants, it can be difficult in practice to ensure that control and treatment groups are being determined in an unbiased manner. An additional layer of complexity is introduced in cases where we would like to find the incremental effect of a treatment: since this requires us to estimate a counterfactual (what would have happened if the treatment had not been applied), a simple comparison between test and control is no longer sufficient.

Geo experiments and CausalImpact provide a framework to handle two of the main areas of complexity involved in incrementality testing: experimental design and causal inference. Geo experiments randomly assign geographies to control and test groups. CausalImpact, a methodology developed by Google with packages available in Python and R, uses Bayesian structural time series models that not only provide estimates of the treatment's incremental impact on our outcome of interest, but confidence bounds as well. In this talk, I will cover a real example from HelloFresh, where we used these two methods to estimate the incremental CAC (customer acquisition cost) of a YouTube campaign.