Binomial test

Binomial test

In this application, we cover the topic of A/B testing for an online site which wants to investigate the effectiveness of different colored buttons to increase the amount of users that sign up for a premium account (this is our conversion).

  • We test the changes in conversion rates associated with the different colors of a button.

Outline for A/B Tests

  • Set up the experiment.

  • Choose the right statistical test (usually binomial test or t-test)

  • Run the test and calculate the statistical significance of the difference between conditions A and B.

Test example

Imagine that we split a random sample of 2000 website members into two groups using random assignment. Random assignment ensures that, on average, everything else is held constant between the two groups.

Let’s formulate a nullhypothesis (\(H_0)\):

  • \(H_0\) “There is no difference in the time spent on site between the two groups.”

Suppose in one condition, we use a red button on our website and sign up 90 out of 1000 randomly selected users who are offered a premium account:

  • Group A (control group): sees red button

  • Conversion rate: 9% (90/1000).

In the other condition, we use a green button and measure the outcome. We get the following results:

  • Group B (test group): sees green button

  • Randomly selected users on our site: 1000

  • Users who signed up: 106

  • Conversion rate: 10.6%

We then compare the conversion rates between the two groups. To determine wether the 1.6% increase in conversion rate is statistically significant, we need to perform a statistical test.

Since we measured a binary event (conversion yes or conversion no), we have a categorical variable. In this situation, we can use the binomialtest.

  • Null hypothesis (\(H_0)\): there is no difference between the two conversion rates

  • Alternative hypothesis (\(H_1)\): there is a difference between the two conversion rates

import scipy.stats as stats

stats.binom_test(106, n=1000, p=0.09, alternative='two-sided')
0.08638821078879608
  • The p-value of 0.086 is greater than the critical value of alpha = 5%.

  • The null hypothesis cannot be rejected at the 5% level of significance

Different results

Now, suppose that instead of 106 sign ups, we only received 70 in condition B. Everything else remains the same.

stats.binom_test(70, n=1000, p=0.09, alternative='two-sided')
/Users/jankirenz/opt/anaconda3/lib/python3.9/site-packages/scipy/stats/_discrete_distns.py:67: RuntimeWarning: divide by zero encountered in _binom_pdf
  return _boost._binom_pdf(x, n, p)
0.026919752379179443
  • In this case the null hypothesis can be rejected at the 5% level of significance because the returned p-value of 0.0269 is smaller than the critical value of 5%.

However, note that the conversion rate in condition A is actually higher then in condition B.

Optional further information: Review the SciPy documention about binomialtests for more information about the test. Note that we always use a two-sided test in A/B-Testing since the estimated value may be more than or less than the reference value.