When is A/B testing a good idea? When is it a bad idea?
A/B testing most commonly fails because the test itself has unclear goals, so you’ve got to know what you’re testing. Use A/B testing to test a theory, for example — would adding a picture to this landing page increase conversions? Are people more likely to click a red button or a blue button? What if I change the headline to stress the time-limit of the offer? These are all changes that can be easily quantified. People run into trouble with A/B testing when their theories are too vague, like testing two entirely different designs with multiple variants. While it can be done, unless there is a clear landslide winner, testing different designs can lead to softer conclusions and an uncertainty about what actually caused the increase in conversions.
How many variations should I have in A/B testing?
Let’s say you’ve brainstormed as a marketing team and you have four great ideas for a landing page design. It can be tempting to run all four treatments at once to declare a winner, but similar to the variations issue above, it’s not a true A/B test if you have multiple different treatments running at once. A number of factors from each different design can get in there and muddy the test result waters, so to speak. The beauty of an A/B test is that its results are straightforward and concrete. We suggest running two versions against each other, and then running a second test afterwards to compare the winners. Think of it as a really techy basketball bracket.
What is a null hypothesis?
A null hypothesis is the hypothesis that any difference in outcomes is the result of a sampling error or standard variation. Think about flipping a coin. While you have 50/50 odds for the coin to land on heads, sometimes the outcome in practice is 51/49 or some other variation due to chance. The more you flip the coin, though, the closer you shoud get to a 50/50 result. In statistics, the way you prove or disprove an idea is to dispute the null hypothesis. Disputing a null hypothesis is a matter of running the experiment long enough to rule out an incidental outcome. This concept is also referred to as reaching statistical significance.
How many visits to a page do I need to get good results with A/B testing?
Before you can test the results of an A/B test, you have to be sure the test has reached statistical significance — the point at which you can have 95% confidence or more in the results.
The good news is, many A/B testing tools have statistical significance built right in so you get an indication as to when your test is ready for interpretation. If you don’t have that, however, there are also a number of free calculators and tools out there for understanding the statistical significance. HubSpot’s is below, and you can also check out a more detailed excel spreadsheet over on the Occam’s Razor blog.
What’s multivariate testing, and how does it compare to A/B testing?
A/B testing is typically used for redesigns to test out the effectiveness of a single design direction or theory against a goal (like driving conversions). Multivariate testing tends to be used for smaller changes over a longer period of time. It will take a number of elements of your site and test out all possible combinations of these elements together for ongoing optimization. In a post in January, my colleague Corey Eridon explained the differences between when you’d use one test over the other in detail, saying:
A/B testing is a great testing method if you need meaningful results fast. Because the changes from page to page are so stark, it will be easier to tell which page is most effective. It is also the right method to choose if you don’t have a ton of traffic to your site. Because of the multiple variables being tested in a multivariate test, you’ll need a highly trafficked site to get meaningful results with MVT.
If you do have enough site traffic to pull off a successful multivariate test (though you can still use A/B testing if you’re testing brand new designs and layouts!) a great time to use the testing method is when you want to make subtle changes to a page and understand how certain elements interact with one another to incrementally improve on an existing design.
Does A/B testing negatively affect SEO?
There’s a myth that A/B testing hurts search engine rankings because it could be classified as duplicate content, which search engines don’t look kindly upon. This myth is most definitely false. In fact, Google’s Matt Cutt advises running A/B tests to improve the functionality of your site. Website Optimizer has a good breakdown of the myth too, and why it doesn’t hold up. If you’re still concerned, you can always add a “no index” tag to your variation page. Detailed instructions on adding a “no index” tag can be found here.
How and when do I interpret my split test results?
The test starts. The results begin to roll in. You scramble to check who’s winning. But the early stages of a test are not the right time to start interpreting your results. Wait until your test has reached statistical significance (see question 4 above) and then revisit your original hypothesis. Did the test definitively prove or disprove your hypothesis? If so, you can start to draw some conclusions. When you interpret your test, try to stay disciplined about attributing your results to the specific changes made. Make sure there are clear connections between the change and the outcome, and there aren’t any other forces at play.
How many variables should I test?
You want your A/B test to be conclusive — you’re investing time in it, so you want a clear and actionable answer! The problem with testing multiple variables at once is you aren’t able to accurately determine which of the variables made the difference. So while you can say one page performed better than the other, if there are three or four variables on each, you can’t be certain as to why or if one of those variables is actually a detriment to the page, nor can you replicate the good elements on other pages. Our advice? Do a series of basic one-variable tests to iterate your way to a page you know is more effective.
What should I test?
- Calls-to-Action: Even with the single element of a call-to-action, there are a number of different things you can test. Just make sure you’re clear on what aspect of the CTA you’re testing. You could test the text — what the CTA compels the viewer to do; the location — where the CTA is positioned on the page; the shape and style — what the CTA looks like. In the example below, for instance, HubSpot tested the shape and style of our demo CTA to see which performed better. The CTA shaped like a button (on the right) rather than the CTA that included a sprocket image (left) performed signficantly better, giving us a 13% increase in conversions.
- Headline: It’s typically the first thing a viewer reads on your site, so the potential for impact is significant. Try out different styles of headlines on your A/B tests. Make sure that the difference between each headline’s positioning is clear rather than some simple wordsmithing so you can be certain as to what caused the change.
- Images: What’s more effective, an image of a person using your product, or the product on its own? Test different versions of your pages with alternate supporting images to see if there’s a difference in action.
- Copy length: Does shortening the text on your page result in a clearer message, or do you need the extra text to explain your offer? Trying out different versions of your body text can help you determine what amount of explanation a reader needs before converting. To make this test work, try to keep the text similar and just test the volume of it.
Can you run A/B tests on things other than web pages?
Yes! In addition to landing pages and webpages, many marketers run A/B tests on emails, PPC campaigns, and calls-to-action.
- Email: Email testing variables include the subject line, personalization features, and sender name, among others.
- PPC: For paid search ad campaigns, you can A/B test the headline, body text, link text, or keywords.
- CTAs: With CTAs, try altering the text on the CTA, its shape, color, or placement on the page.
Need more ideas? This post has 28 different tests you can run.
How often should I run A/B testing?
Perspectives vary on this one. There’s a good case to be made for always testing and iterating on your site. Just be sure that each test has a clear purpose and will result in a more functional site for your visitors and company. If you’re running a lot of tests that are resulting in minimal outcomes or minor victories, reconsider your testing strategy.