Sample size calculation

Sample size tells you how many users need to see your test before you can trust the results. Too few users means your winner might just be lucky. Too many users means waiting forever when you already have a clear answer.^[1]

Online calculators like Optimizely help you find the right number.^[2] The calculation needs 3 things: your baseline conversion rate or other metric you want to improve, the minimum improvement you want to detect, and your desired statistical confidence level. For example, if your product’s desired subscription level has a 20% conversion rate and you want to detect a 5% improvement with 95% confidence, you might need 25,000 visitors per variation to make a valid conclusion.

Your daily traffic determines how long tests run. Amazon can test new features in hours because millions visit daily. A small B2B tool might need weeks to get enough data. This is why you should calculate sample size first. Otherwise you might end tests too early or let them drag on too long.