The Recommended A/B Testing Stack for Advanced Digital Marketers

The concept behind A/B testing is simple. You compare two versions of content against each other to determine which one performs better.

As a result of running split tests, your business and website will benefit from more leads and higher conversion rates.

My name is Lars Lofgren and before I started running a company at Quick Sprout, I started my career in Growth. As the Director of Growth at KISSmetrics and I Will Teach You To Be Rich, I was personally responsible for hitting our lead goals and ran our A/B testing programs to improve conversions across each website.

I’ve conducted hundreds of A/B tests over my career.

At both companies, I doubled and tripled conversion rates. More specifically, I quadrupled the monthly lead flow at KISSmetrics and built lead-generation machine yielding 40,000 leads per month at I Will Teach You To Be Rich.

During my marketing career, I’ve tried just about every software related to testing available. Some were amazing, and some were not. If I were to ramp up an A/B testing program again, these are the 11 tools that I would trust.

Best tools for A/B testing

There’s quite a few different types of tools that you’ll need for A/B testing. It goes way beyond a single tool for running tests. Let’s go through all the applicable categories.

Transactional and custom emails

To properly test transactional emails, you’ll need complete control over your email creation and analytics. SendGrid offers this type of control. More popular email tools like MailChimp just aren’t robust enough, even many of the Marketing Automation tools aren’t flexible enough. It’s better to have an email tool that your developers can control completely.

Sendgrid does have some A/B testing built-in but it’s limited:

The magic of Sendgrid doesn’t come from the built-in features. It comes from your developers being able to A/B test entire emails and funnels since they have complete control. When you have full control, you can run any tests that you want.

First, I personally never bother testing subject lines or click-through rates. They don’t have a material impact on the business. Second, testing content is okay, but I prefer to test completely different email funnels. That’s where I found my biggest wins. That’s why I ignore the built-in testing UI and use Sendgrid for entire emails.

Most of my A/B email tests are actually hold-out tests. Send your email to 50% of the test group, send nothing to the other 50%. You’ll be able to see the true impact of that email.

Surveys

SurveyMonkey is the industry standard for online surveys. Almost all of you reading this have taken a survey through their platform at one point or another.

I like SurveyMonkey because it’s very easy to use. Since nearly everyone is familiar with it, users are comfortable taking surveys powered by their software.

You may be asking, “Why do I consider surveys to be part of my A/B testing stack?”

Survey responses fuel the insights that lead to A/B testing wins. Before running any tests, I start with qualitative research. The better I understand the customers, the higher the odds I’ll find a winner with A/B tests.

To collect valuable qualitative insights, keep these tips in mind:

• Don’t overcomplicate your surveys. Focus on asking 5 to 10 questions that you really need to know.
• Focus the bulk of your questions on the problems that customers are having.
• Start with open-ended questions until you have a really good intuition for the respondent.

For more guidance, there’s a bunch of great survey tips here.

User tests

Another great way to find qualitative insights for A/B testing wins is with user tests. The best tool for these experiments is UserTesting.

The first thing you’ll need to do is to select your audience. You can use your own audience or connect with a specific target through UserTesting’s panel. The platform allows you to be precise with demographics and other factors to find the best users for your experiments.

Next, select 5 or 10 users to go through your onboarding process or other conventional user flow actions on your website. UserTesting will provide videos of these users interacting with your content. As a result, you can identify the major points of friction to give you wins for A/B testing.

This is a gold mine for A/B testing wins. I strongly recommend running a few of these on your signup or onboarding flows. I always find a major point of friction that leads to big wins later. At KISSmetrics, we learned how difficult it was for folks to map out all the data they needed to track. Then we revamped our onboarding, making it more structured and walking people through every piece of data step-by-step.

Business intelligence

For those of you who want a serious A/B testing program, you’ll need to bring all of your data and reporting in-house. To do this effectively, you’ll need to have a data warehouse of some kind. Once this is set up, then you can implement the best tool (in my opinion) for business intelligence reporting — Tableau.

I prefer Tableau because it gives me complete control and it will generate any report that you want. Since I know the exact queries, reports, and visualizations that I need, I can get everything built just the way I want it.

However, the unfortunate news is that someone must manage it and build the reports that you need, which is a full-time job. You’ll also need the expertise at the executive level to know what reporting is required. A traditional BI analyst can build the reports but won’t know which reports and metrics matter the most for the business. All this gets expensive fast.

This solution is not for everyone. Only go this route if you have the expertise and budget to make sure that it’s executed properly.

Statistical significance calculator

Depending on your skill level, two tools are recommended for statistical significance calculations.

Easy calculator
The A/B Significance Calculator from Neil Patel is your most straightforward option. All that’s required is to manually insert the number of visitors and conversions for each variation:

The calculator then determines if the test was significant enough to draw conclusions.

Advanced calculator
For a more advanced option, the AB Testguide calculator is your best solution. This tool gives you more parameters to include in the calculation.

With this calculator, you can choose a hypothesis and select your confidence level. Even the results of the test are more advanced. Not only does the tool show you a graph, but other statistical data as well like the observed power, p-value, z-score, and standard error of the difference:

It’s a bunch of extra details that only make sense if you’ve done a lot of testing or have a background in statistics. If this is too complicated, opt for the simpler calculator.

Why do I use these?

Don’t most testing tools calculate statistical significance?

If there’s one lesson I’ve learned, it’s not to trust new tools blindly. Double-check statistical significance calculations until you’re confident an A/B testing tool can be trusted. There are lots of ways to calculate statistical significance and it’s also easy to mess this stuff up.

So whenever I’m working with new testing software, I always use one of the calculators above until I know for sure that their calculations are solid.

Web analytics

Look no further than Google Analytics, the king of traffic analysis.

To A/B test a website, you’ll need to understand the traffic dynamics, in particular:

• What pages get the most traffic?
• Which traffic sources are growing or declining?
• Which pages bring in the most traffic?

Even if you only use Google Analytics to confirm you have enough traffic on the pages you intend to A/B test, it’s easily worth the effort to install. It’s also completely free and a world-class tool.

There’s a reason that it’s installed on nearly every site out there. The quality of data, the volume of reports and insights, and the fact that it’s completely free make Google Analytics an unbeatable deal.

Previously, I used to customize Google Analytics profiles heavily. Currently, though, I rarely touch the settings and rely on the built-in reports. So I suggest you do the same.

User analytics

Why get a second analytics tool? Isn’t Google Analytics enough?

Even after all these years, Google Analytics excels at traffic analysis. Yes, it has user tracking now, but it was never designed for that. You need to be a Google Analytics power user to use it for true user analysis.

For that, my favorite tool for user analytics and customer data is Amplitude because you can easily track user funnels:

Most A/B testing programs typically focus on signup and onboarding funnels. Not only is there plenty of volume to test, but small improvements to these funnels can have a major impact on the business.

As you test your funnels, you’ll want to monitor them every month to make sure they’re steadily improving. A/B tests may report one result, but your monthly KPIs won’t match up perfectly.

If you’ve decided to go all-in with a data warehouse and business intelligence tool like Tableau, you can skip the user analytics tool entirely. I find that a user analytics tool is a good in-between step when a business isn’t ready to build out the entire infrastructure.

Post-click landing pages

For post-ad clicks, you’ll want a dedicated solution that specializes in landing page experiences — not an all-in-one software that happens to have “A/B testing” or a “builder.” This is especially true when each campaign deserves a unique post-click landing page and A/B testing each page in the process. Not only does Instapage help you scale landing page production for every campaign, it has A/B testing built right in. When you find wins, you’ll be able to launch them immediately:

My marketing programs always end up with tons of landing pages. If you’re not A/B testing the pages with the most traffic, you’re leaving revenue on the table.

Heatmaps

In customer research, there’s a common piece of advice: “don’t focus on what people say, focus on what they do.” That’s why heatmaps are another great data collection resource for qualitative insights. Heatmaps are perfect for getting a better idea of how people behave. Plus, they’re super easy to understand.

Knowing what people click on, how far they scroll, and where their mouse moves spur all sorts of ideas for optimization — it’s another gold mine of ideas for A/B testing.

Instapage has heatmaps built into their platform that can do all three for post-click landing pages (image below is only the click map):

You’ll also want to run heatmaps on your main site, too. In that case, I’d recommend Crazy Egg:

Run heatmaps on your homepages, pricing pages, and onboarding flows. Look for things that get a lot of clicks that you didn’t expect, and items you thought would get more attention but don’t. Then come up with test ideas that highlight what people want while de-emphasizing the stuff they don’t care about.

CXL has a lot of great tips on how to use heatmaps here.

A/B testing

What about the A/B testing tool itself? What’s the best option for testing homepages, onboarding flows, and products? In my opinion, VWO:

VWO has a solid A/B testing product that starts at $199/month. Compared to other tools that can easily be tens of thousands of dollars per year, VWO is much more accessible. The value per dollar is exceptionally high.

With VWO, you’ll be able to run multiple split tests simultaneously. There’s also a visual editor that allows you to run A/B tests without editing any code. Perfect for testing elements like headlines. You can also use their code editor to make more advanced changes to your tests. The code editor is a huge improvement.

WordPress A/B testing

WordPress is the most popular CMS in the world, but running A/B tests across an entire WordPress site can be a pain. The platform is definitely not designed for this.

That’s why you should use a plugin like Nelio for WordPress A/B testing:

Be warned; you want to get your site in order before doing this testing. If you don’t get everything cleaned up, you’ll run into countless plugin conflicts and your site won’t be able to serve testing variants fast enough for valid testing. Since WordPress was never designed for site-wide testing, it’s easy for A/B tests to conflict with plugins, themes, and poorly built sites.

First, make sure you get a good web host. The last thing you want is a slow host that will delay serving the variant, so the user notices the change. Then clean up your WordPress theme and remove any unnecessary plugins. This will reduce any plugin conflicts and serve your tests fast enough to avoid being noticed.

Once your WordPress site cleaned up and has a fast load time, install Nelio, and use that to run site-wide A/B tests. I used to run a lot of tests on our blog to increase conversions to email subscriptions.

Conclusion

A true A/B testing program needs a suite of tools to run effectively. After testing hundreds of tools myself across multiple companies and hundreds of tests, I prefer the stack above because they focus on different areas of digital marketing, and not one is “best” in all categories.

If there’s some hot, new tool that you’ve been considering, feel free to swap one of these out. My general rule is to experiment often with tools that are easy to start and stop like user recording or survey tools. But for infrastructure tools like analytics, I pick the one that’s reliable and that I’ll be able to depend on for years.

About the Author
Lars is currently the CEO of QuickSprout. Before that, he was the Director of Growth at I Will Teach You To Be Rich and KISSmetrics.