×

AB Test Calculator


Group 1

Group 1 can't have a greater conversion rate than 100%. Group 1 can't have a greater conversion rate than 100%.

Group 2

Group 2 can't have a greater conversion rate than 100%. Group 2 can't have a greater conversion rate than 100%.

How to Use AB Test Calculator?

Follow the below steps to calculate the most significant 
sample space
  • Select the Confidence interval from the drop-down menu
  • Enter the sample size
  • Enter the number of possible results
  • Enter the conversion rate
  • Hit the “Calculate” button to check whether it is statistically significant or not.
  • Click on the “Show Steps” button to see the step-by-step solution
  • Click on the “reset” button to erase all inputs. 





AB Test Calculator


The AB test Calculator calculates the statistical significance of two sample spaces to find out whether the samples are statistically significant or not.


What is AB Test?


Statistical significance test in which two populations are related to checking if they differ significantly on a characteristic. it is basically a kind of Z-test. The objective of the AB test is to show that two proportions are not the same.


What is a Z-score?


A Z-score expresses a value's relationship to a group of values from the same distribution. The Z-score is calculated in the number of standard deviations from the group's mean. A higher Z-score means the value is more away from the distribution's mean.


Method of calculation:


  1. Determine their sample sizes (n1 and n2)
  2. Determine the number of positive results in each group (t1 and t2)
  3. Calculate the population's proportions using the formulas

                             p1 = t1 / n1 and p2 = t2 / n2

  1. Calculate the overall sample proportion

                             p = t1 + t2 / n1 + n2

  1. Calculate the Z-score

                         Z score formula            

  1. Compare the absolute values of the results. If the absolute value of the Z-score is greater than or equal to the absolute value of the alpha level’s Z-score then the samples are statistically significant.

Example section:


In this section, we have given the step-by-step calculation of the AB test.

Example 1:

Compare the following groups and find out whether these are statistically significant or not,

if the confidence interval is 90%

Group 1

Sample size = 40

No. of positive results = 2

Conversion rate = 5%

Group 2

Sample size = 12

No. of positive results = 5

Conversion rate = 41.7%

Solution:

Step 1: Extract the data

Sample size = n1 = 40

No. of positive results = t1 = 2

The conversion rate in percentage (%) = 5

Group 2

Sample size = n2 =12

No. of positive results = t2 = 5

The conversion rate in percentage (%) = 41.7

Step 2: Apply the formula

p1 = t1 / n1

p1 = 2 / 40

p1 = 0.05

p2 = t2 / n2

p2 = 5 / 12

p2 = 0.417

Step 3: Calculate “p”

p = t1+t2 / n1+n2

p= 2+5 / 40 + 12

p = 0.135

Step 4: Calculate the Z-score

Z-score = p1 - p2 / √{p * (1 - p) * (1/n1 + 1/n2)}

Z-score = 0.05 - 0.417 / √{0.135 * (1 – 0.135) * (1 / 40 + 1 / 12)}

Z-score = -3.263

Alpha level’s z-score = 1.645

Step 5: Comparison

For the comparison, compare the absolute value of both the Z-score and Alpha level’s Z-score.

The absolute value of Z-score = 3.263

Absolute value Alpha level’s z-score = 1.645

As the absolute value of “Z-score” is greater than the absolute value of “Alpha level’s z-score” the samples are “statistically significant”.

 


 References:


A/B testing statistics: An intuitive guide for non-mathematicians | Conversion Sciences.

A/B testing - A complete guide to statistical testing. | Towards Data Science.