Binomial test
Contents
Binomial test¶
In this application, we cover the topic of A/B testing for an online site which wants to investigate the effectiveness of different colored buttons to increase the amount of users that sign up for a premium account (this is our conversion).
We test the changes in conversion rates associated with the different colors of a button.
Outline for A/B Tests
Set up the experiment.
Choose the right statistical test (usually binomial test or t-test)
Run the test and calculate the statistical significance of the difference between conditions A and B.
Test example¶
Imagine that we split a random sample of 2000 website members into two groups using random assignment. Random assignment ensures that, on average, everything else is held constant between the two groups.
Let’s formulate a nullhypothesis (\(H_0)\):
\(H_0\) “There is no difference in the time spent on site between the two groups.”
Suppose in one condition, we use a red button on our website and sign up 90 out of 1000 randomly selected users who are offered a premium account:
Group A (control group): sees red button
Conversion rate: 9% (90/1000).
In the other condition, we use a green button and measure the outcome. We get the following results:
Group B (test group): sees green button
Randomly selected users on our site: 1000
Users who signed up: 106
Conversion rate: 10.6%
We then compare the conversion rates between the two groups. To determine wether the 1.6% increase in conversion rate is statistically significant, we need to perform a statistical test.
Since we measured a binary event (conversion yes or conversion no), we have a categorical variable. In this situation, we can use the binomialtest.
Null hypothesis (\(H_0)\): there is no difference between the two conversion rates
Alternative hypothesis (\(H_1)\): there is a difference between the two conversion rates
import scipy.stats as stats
stats.binom_test(106, n=1000, p=0.09, alternative='two-sided')
0.08638821078879608
The p-value of 0.086 is greater than the critical value of alpha = 5%.
The null hypothesis cannot be rejected at the 5% level of significance
Different results
Now, suppose that instead of 106 sign ups, we only received 70 in condition B. Everything else remains the same.
stats.binom_test(70, n=1000, p=0.09, alternative='two-sided')
/Users/jankirenz/opt/anaconda3/lib/python3.9/site-packages/scipy/stats/_discrete_distns.py:67: RuntimeWarning: divide by zero encountered in _binom_pdf
return _boost._binom_pdf(x, n, p)
0.026919752379179443
In this case the null hypothesis can be rejected at the 5% level of significance because the returned p-value of 0.0269 is smaller than the critical value of 5%.
However, note that the conversion rate in condition A is actually higher then in condition B.
Optional further information: Review the SciPy documention about binomialtests for more information about the test. Note that we always use a two-sided test in A/B-Testing since the estimated value may be more than or less than the reference value.