If an A/B test's statistical evaluation is based on the number of trials and successes for each variation, what happens if you don't know the number of trials for your experiment? Although this question sounds rather theoretical, we faced it in practice. This talk is about our search for the right question to ask.
GetYourGuide is the leading online platform for touristic in-destination activities such as tours, excursions, and pub crawls. An important marketing channel for us is search engine marketing (SEM). The question of how much to pay for showing a potential customer a paid ad is highly complex. It does not only involve some estimate about the likelihood that the user will follow our ad to our website and finally book with us; the price is also influenced by our competitors since the decision about what ad is shown is made in an auction. A lot of room for improving our bidding strategies.
In early 2016, Google Adwords, one of the big players in SEM, rolled out a new feature called Campaign drafts and experiments. It enabled advertisers for the first time to test different bidding strategies in proper A/B tests. We thought they were proper, at least, until we figured out that Adwords wouldn't supply us with all necessary information to evaluate our tests in a decent statistical way.
The simple question of statistical significance became a search for the right question to ask.
In this talk, we will explain the problem in some more detail and report about our various approaches of how to circumvent it. On our journey, we will touch the foundations of SEM, A/B testing as well as time series.