Best Practices: Making the Most of Your Tests
A test is only as good as the practices behind it. To make sure you're getting the most out of your A/B and multivariate testing, I have put together three best practices. Anything else you would add?
-
Make sure your sample size is sufficient – If you're trying to sell 10,000 widgets, you can't expect to learn much from an A/B test that only reaches 30 people. Also, make sure your sample size is representative. If your friends and family tell you your marketing email is effective, what have you learned? Whenever possible, do your testing in the real world. If something sells, it sells.
-
Make sure your test lasts long enough – In the online poker world, they use the word variance to distinguish one lucky tournament run from months of consistent profits. One day of record-breaking sales tells you nothing. You need to run your tests long enough to decide whether you're looking at a real trend or a blip that relates more to luck than skill.
-
Control for extremes – Unless you're specifically testing for the impact of a holiday on your sales, remember to control for extremes. Don't let a Cyber Monday throw you off. Your tests should be as close to real science as they can be if you want them to work.