So you want to run an A/B test! How long should you run it for? How will you know if an observed difference is real? Does it matter what my A/B test is measuring? All of these questions and more can be answered by learning about Hypothesis Testing in SciPy. You might have heard of T-Tests, Chi Squared, and ANOVA in your high school stats class, but you've never heard them explained like this!
Students should be familiar with basic Python, Jupyter notebooks, and loading and selecting data using Pandas.
Learners will be able to define an "A/B Test" and give examples of how they are used by companies to aide in data-driven decision-making.
Learner will be able to distinguish between categorical and continuous measurements
Learner will be able to use SciPy to generate normal (for continuous measurements) and binomial (for categorical measurements) distributions
Learner will be able to simulate different A/B Tests and understand how sample size and effect size can cause misleading A/B test results
Learner will be able to explain how hypothesis testing can be used to validate A/B testing results
Learner will able to define "null hypothesis" and explain it's importance for hypothesis testing
Learner will be able to estimate a sample size for an A/B Test using simulations
Learner will be able to select one of the following hypothesis tests based on the type of metric they are reporting: Chi Squared, Student's T Test, or ANOVA
Learner will be able to estimate the necessary sample size for an A/B test using simulations
Learner will be able to use online tools for estimating sample size
Learner will be able to use online tools performing hypothesis testing