In our last blog post, we talked about the basics of A/B split testing - what it is and what to test. More crucial than knowing the basics though is knowing what to avoid. Mistakes in A/B split testing make it more likely that you will misinterpret results from your experiment. It also often times leads to more testing that may not have even been necessary to begin with. Below we present to you the top three mistakes people make when A/B split testing and how to avoid them.
Randomly Testing for Random Results
We briefly mentioned the importance of using a hypothesis before you delve into A/B split testing, and sure enough - less than one month later - it still holds true. Random testing with minor changes to see if there are any positive results will prove futile if you do not actually know what you are looking for.
Your hypothesis should have two parts - a what and a why. You should have both noticeable changes and know the motivations behind the changes. Are you changing your “Go” button from blue to green because you read somewhere it increases sales? Well then, go for it! It is important not to worry if your hypothesis is correct or not. Many successful business owners have been proven wrong and made a killing on it! At the end of the day, you are simply looking for a solid answer, backed by concrete data and actual results.
Small Sample Size
Patience plays a key role in A/B split testing for many reasons, the first being your sample size. A large enough sample size is essential for proper A/B split testing; however you’ll notice that finding an estimate of exactly how large your sample size should be is tougher than it sounds. Some programs claim to get accurate results after hundreds of visits, while others recommend 1,000 to 6,000 visits.
Each website is unique, so base the sample size you will need on the number of visitors you normally get. Your web developer will have the best idea of how many visits you need with any given experiment before you make a conclusion.
Jumping the Gun
Another patience related problem is reading your customer’s choices too soon. A/B split testing is exciting, no doubt, but that does not mean you should sit by your computer every night clicking the refresh button on your statistics. A/B split testing does not take into consideration every possible variable, so there may be some unknown outside factors that influence your results.
Waiting it out means that you will have a better chance of glossing over any anomalies, whereas stopping too early may lead to a false positive. So even if you see a definite winner, hold off until you make a change. In the end, allowing a test to run its course will lead to larger sample sets with more complete and accurate data.
In the End, Trust Your Gut
It is very important to give the people what they want, but make sure not to completely ignore your designer’s advice just because more people clicked on the blue button you made. It is also very important not to ignore your accountant. If your A/B split test resulted in more people clicking “Submit Order,” but your overall sales is trending downward, you may want to rethink things.
