Sample size calculation

Sample size determines how many users you need to make reliable A/B test decisions. Using too few users can lead to false conclusions, while testing with too many unnecessarily delays results. The calculation balances statistical confidence with practical time constraints.

The essential factors in sample size calculation include your baseline conversion rate, the minimum improvement you want to detect, and your desired statistical confidence level. For example, if your homepage has a 20% conversion rate and you want to detect a 5% improvement with 95% confidence, you might need 25,000 visitors per variation to make a valid conclusion. Power calculators like Optimizely simplify this process, but understanding the variables matters. Higher baseline rates need smaller samples — a page with 50% conversion rate requires fewer test visitors than one converting at 5%.

Similarly, detecting smaller improvements requires larger sample sizes than spotting major changes. For example, testing a major change, like extending a free trial from 7 to 30 days, might need only 5,000 visitors per variation because it could drive a big improvement. However, testing a button color change might require more visitors per variation since it may only create a small difference.