A/B testing is one of the most powerful tools in a digital marketer's toolbox. While it's easy to speculate about which logo looks better or which email subject lines get the most clicks, it's hard to argue in the face of cold, hard statistics.
By seeing which versions of your web page or marketing materials have the highest performance in terms of conversion rate, you can make smarter, data-driven decisions about the best way to bring in customers.
While it's difficult to deny the utility of A/B testing for your business, however, it can be tempting to go overboard in the effort to tweak and fine-tune your advertisements until perfection.
Unfortunately, marketing budgets aren't unlimited (especially for small businesses), so you need to figure out how you can get the most bang for your buck.
In this article, we'll discuss how much you should be spending on A/B testing, so that you can strike the right balance between receiving valuable results and ruining your return on investment (ROI).
The Return on Investment of A/B Testing
The average cost per click of a Google AdWords campaign is between $1 and $2, and some keywords may cost upwards of $50 for a single click.
With the associated expenses so high, you need to make sure that your digital marketing campaigns are well-oiled machines designed to convert as many people as possible – and you can do that with A/B testing.
Done right, A/B testing has resulted in some amazing victories for marketers.
When President Barack Obama was running for office, his digital marketing team used A/B testing to pick among four different buttons for user signups. The team estimates that the experiment helped pick up 2.8 million additional email addresses, or roughly $60 million in extra donations.
There's also the (in)famous story about Google manager Marissa Mayer (later CEO of Yahoo), who used A/B testing to decide between 41 different shades of blue to use for Google's navigation bar. Today, Google's simple interface with bright primary colors is one of the most recognizable brands in the world.
Despite these successes, you need to make sure that A/B testing is right for your own company's situation.
It's estimated that only 1 in 8 A/B tests actually result in significant change for the organization.
How Much Your Company Should Spend on A/B Testing
In order to decide how much you should be spending on A/B testing, you first need to settle on a set of metrics and KPIs to monitor, and then weigh them to ensure that the benefits of testing outweigh the costs.
A/B Testing Metrics
There is no single set of metrics and KPIs that will work for every organization when doing A/B testing. However, some of the most common metrics are:
• Bounce rate:
The percentage of visitors who leave the web page without visiting the rest of the site.
• Conversion rate:
The percentage of visitors who perform a desired action, such as making a purchase or signing up for your mailing list.
• Return rate:
The percentage of visitors who return to your website at a later time.
• Engagement rate:
The percentage of visitors who engage with a given website feature.
• Time on site:
The average length of time that a user spends on your website.
The average predicted net profit that a new customer will generate for your business.
Evaluating Your A/B Testing Metrics
Once you've chosen the metrics that are most meaningful to your business, you can do a little math to figure out whether A/B testing makes sense for you.
For example, suppose that you've selected conversion rate and lifetime value as your two most important metrics:
• By performing A/B testing, you find that one of the website designs improves conversion rates by 2 percent, and you estimate that 100 new customers will sign up as a result.
• What's more, you project that these customers will have an average lifetime value of $200.
This means that you can spend a maximum of 100 x $200 = $20,000 on A/B testing to ensure it's still worth your while.
When A/B Testing Isn't Worth It
Although there are clear benefits to A/B testing, there are also certain cases where it's simply not worth the effort. Below, we'll describe two of these situations.
Not Enough Visitors
Every unique impression on your website counts, which means that you might feel pressured to A/B test all of your visitors. If your sample size isn't large enough, however, your results won't be statistically meaningful.
Generally speaking, you want to have a p-value of less than 0.05 in order to say that a test proves or disproves a given hypothesis. This depends on both the sample size of your survey and the conclusiveness of your results.
If you find that 55 percent of your first 100 visitors prefer a given logo, for example, that's likely not enough users to come to a definitive conclusion yet.
If 95 percent of them prefer the logo, however, you may be able to conclude which logo is preferable without having to survey hundreds of additional users.
What's more, as you test more minor elements, the traffic you need to arrive at a conclusion should increase.
Deciding whether a button should be green or blue will require a great deal more testing than picking between two landing pages for your website.
A/B testing is an application of the scientific method: you come up with a hypothesis, run some tests, and then modify your assumptions if need be.
For example, you might notice that an unusually high number of visitors are abandoning their online purchases in their shopping cart. After making this observation, you hypothesize that this is due to a confusing checkout process that causes people to lose interest.
In order to test this hypothesis, you can use A/B testing on your customers: one group sees your original checkout screen, and the second group sees a new screen that you've tried to streamline and make more clear. The results of this experiment will help you decide whether you should change the process.
Without a hypothesis to test, however, you'll be flying blind, testing without a greater purpose to guide your efforts.
As a result, you'll be more likely to make rookie mistakes such as making too many changes at once, or not adding a control group to the experiment.
A/B testing is a powerful tool that can provide you incisive insights into your customers' most intimate thoughts and behaviors. Like all tools, however, A/B testing is only as good as the person who's currently using it.
By learning which situations A/B testing is best suited for and applying it in the right ways, you can cut your marketing costs while improving your ROI. Most importantly, remember that A/B testing should be combined with other tools in your marketing toolbox.