Conversion Rate Optimisation - The Best Tools for A/B Testing

CRO & UX Article

TL;DR:

Choosing the right A/B testing tool can be challenging, but here we have reviewed some popular options compared based on cost, user interface, test versatility, and reporting:

Google Optimize: Offers a free plan with intuitive UI and clear reporting, but its future beyond September 2023 is uncertain.
VWO: A popular alternative with a free plan, Google Analytics integration, and various paid plans for more advanced features. Test versatility depends on the chosen plan.
Split: Geared towards software and development teams, with Google Analytics integration and customizable plans. Good for testing changes that impact website functionality.
Convert.com: Affordable and offers multiple price plans, but no fully free option. Popular among small businesses and provides familiar UI, ease of setup, and sufficient test versatility.

Remember, the choice ultimately depends on your specific needs and preferences and how you plan to carry out your conversion rate optimisation. If we missed any tools, feel free to reach out for a review.

Share it with friends

Choosing the best A/B testing tool

When met with the task of putting your CRO insights and analysis into practice, it is not immediately obvious which tool is best for the job. An A/B testing tool that is sufficient for one person might be the complete opposite for another. There are so many different options out there, all with their own draws and perks. This short analysis presents a few of the more popular choices with some distinct variations between them and will hopefully help point you towards the tool that is right for you, or at least get you thinking about what you want to prioritise in your CRO testing!

The a/b tools considered here are:

Google Optimize
VWO
Convert.com
Split

and are compared based on the following aspects:

Cost
User Interface (UI)
Test versatility
Reporting

Google Optimize

Yes, it has been announced that Optimize is sunsetting, coinciding approximately with Google Analytics 4 (GA4) becoming the sole analytics tool from Google. However, will Google simply abandon its A/B testing capabilities, or launch a shiny new tool that incorporates the new designs and tracking features of GA4? The feeling amongst digital marketers is that it would be foolish for Google to give up such a popular source of data and force its users to find other platforms.

In terms of free plans, Optimize is probably one of the easiest tools to get to grips with. Google does offer a paid plan, Optimize 360, which is mainly geared towards larger organisations and reflected in its high price tag. It offers the ability to run many more tests simultaneously, with more variants and objectives; the free version of Optimize is limited to five A/B tests at one time.

When it comes to assessing Optimize as an A/B testing tool, keep in mind that it may not exist beyond September 2023 and, if it does, we are assuming that there will be a clear continuity and consistency with the current set up.

Cost: £0; ~$150,000 (~£120,000) per year
UI: Intuitive, easy to use, although can be slow to navigate through
Test versatility: Sufficient in most situations (free version)
Reporting: Clear results and useful view of modelled conversion rates over time, although cannot visualise distributions at a given time

Visual Web Optimiser (VWO)

VWO is one of the most popular alternatives to Optimize. Perhaps more recently this is due to their discount on migration from Optimize 360 users. There is also the added benefit of its own Google Analytics integration, meaning that there is no need for separate data tracking. It has a free plan, although paid plans offer arguably essential features. For example, for deeper dives into the data you’d want to download csv files of test reports, which requires at least the ‘growth’ (£313 per month) paid plan.

A bit more versatility in testing parameters such as more refined targeting (based on channel source, location, cookies etc.) and multivariate tests are likely the other considerations you need to make, but may see you requiring to spend more. Multivariate tests in particular require the second price plan, which may be a deal-breaker for some. The alternative would be to run multiple tests consecutively, taking much longer to gain insights.

Note that the price plans are quoted monthly, but are billed annually and a slight change in feature requirements may change your annual bill quite drastically.

This is all not to say you cannot run effective tests with the free plan. All the basics are there to get you testing for smaller CRO requirements.

Cost: £0; £313 per month; £720 per month; £1260 per month
UI: Quite easy to navigate around, a lot of test features for keeping track of test ideas, hypotheses, notes etc.. Its unique design may a bit of time to get used to
Test versatility: Highly dependent on the cost plan
Reporting: High visual clarity, options for different views of conversion rates and probability distributions

Split

Similar to VWO, Split also offers Google Analytics integration. A/B testing is only one of Split’s many features. It is mainly geared towards software and development considerations, and its technical data tracking means that it is a great option for testing changes that potentially have a large impact on your website’s functionality. However, there is still the capability to perform simple A/B experiments, such as colour changes.

Hence, while Split has the potential to run traditional CRO for marketing purposes, it seems best suited for development teams looking to test methods when creating new components and functions.

A drawing point for this easy-to-use alternative is that you can highly customise a plan for your own needs. Dependencies are how many users need access, the extent of different data integrations and the number of testing configurations. Their free plan is good for trying the tool out and has a decent level of options for getting started. Similar to other tools, Split is quoted per month but is billed annually.

Cost: Starts at £0 and customised plans range from $330 (~£260) per month to $54,000 (~£43,000) per month
UI: Minimalist in design, but doesn’t include unnecessary content. Clarity in different terminologies
Test versatility: Highly versatile, but likely depends on the development skills of the user
Reporting: Not as visual as other testing tools, but required statistics (e.g. confidence levels) can be tailored for a given purpose

Convert.com

Convert.com offers a slightly more affordable alternative when it comes to standard A/B testing platforms. Like the couple mentioned above, there are several different price plans available. An advantage here is that they can all be trialed for free, alleviating the difficulty of choosing. The downside is that there is no fully free plan, although the cheapest option has quite a lot to offer as a baseline. There is also a small degree of customisation available in the case of choosing a plan as the maximum number of users can be increased for an increased price. Flexibility is also a bonus for the lower value plans as they are billed monthly.

The difference between plans appears to be more on the maximum allowances (test users, goals, deploys etc.) rather than testing parameters and options.

From Convert.com’s user base, it seems to be more of a small business tool compared to some of the more up-market options out there. For those coming from Google Optimize, this one would likely feel the most familiar, in terms of UI, format and its functionalities. It rates highly in user feedback, particularly in meeting A/B testing requirements, as well as its ease of set up and general use.

Cost: $99 (~£80) per month; $199 (~£160) per month; $699/999 (~£560/800) per month; $1,119/1,599 (~£895/1280) per month. The latter two values depend on whether the plan is billed annually or monthly.
UI: Not overly-complicated and fairly simple setup. It does not contain unnecessary content
Test versatility: Sufficient in most situations (across plans)
Reporting: Quick overviews and summaries

Summary

This blog has covered some of the most common and popular A/B testing tools that marketers, data analysts and developers will encounter. There is not one definitive option that will always be the correct tool, as exact choice will also depend on the individual’s particular needs and preferences. While not an exhaustive list, we have identified some of the main distinguishing features that separate different tools to get you thinking about how to make your choice.

Think we have missed one out, drop us a message and we will review it.