AB Test Part1

AB test

• An expriment where two or more variants of a page are shown to users at random.
• statistical anlysis is used to determine which variation perform better for a give conversion goal and whether the improvment is statistically significant.
• conversion (the process of changing) goals can be around clicks, subscriptions, purchases etc.
• the current page, or the current business as usual scenario is also reffered to as the Control.
  • the new Page / the new business is often reffered to as Test.

When to use AB test

1. Test the differing impacts of alternate campaifn / web page desgins or approaches
2. Test one element on the page.

Why are statistic important to AB test?

• AB test is an example of statistical hypothesis testing.

Concept of Sampling Error

Regression Towards the Mean

• regression toward the mean is “the phenomenon that if a variable is extream on its first measurement, it will tends to be closer to the average on its 2nd measurement” and vice vase.

• As we continue increasing sample size, and the length of observation, the mean of our observations will get closer and closer to true mean population.

Confidence Interval

Example

• Original Page ```

  of visitors = 1’000

  of conversions = 100

Conversion Rate of Page = 100/1000 = 10% ``` 10% simply the mean of our sample. we want to know the mean of population

What affect the width of the confidence intervals

1. variation in population
2. size of sample
3. confidence level

Effect of variation = population with low variation.