In a perfect world, we would always know exactly which marketing strategies, website arrangements, and e-Commerce campaigns will be most effective. There is often a plethora of possible but untested solutions and approaches that could feasibly provide the best method. The difficulty is determining which of these strategies is the most optimal for any given sales, marketing, or e-Commerce situation.
Wouldn’t it be great if we could compare, in practice, the effectiveness of a variety of different methods? Well, it just so happens that we can, using a procedure known as A/B testing.
This article defines A/B testing, what it can be used to test, and the steps for conducting A/B testing.
BigCommerce defines A/B testing, also called split testing, as a procedure where two versions of the same thing are deployed to each half of the intended audience. It can be a makeover of the e-Commerce store; a new product, feature, or a service; or a new marketing campaign. It can also be about deploying a small fix and testing whether it improves our business or not.
Often the effectiveness of a new version is measured using its conversion rate. Below are a few types of commonly tested conversion rates:
If more than two versions need to be tested, you can use A/B/n testing, which is a modified form of A/B testing for testing more than two versions of a strategy at the same time..
There are several benefits to A/B testing besides being able to test whether or not a feature you want to implement or change will work. According to Taplytics, A/B testing also allows you to:
These benefits will come from a detailed analysis of the resulting data from A/B testing. In order to take the most advantage of this information, you will need a system for storing the data and doing further analysis.
You can use A/B testing to test features for all steps in the marketing funnel. ClickFunnels defines the marketing funnel as “the entire journey from a person being aware of your business to that person becoming a committed customer.”
There are three main steps in the marketing funnel, with the corresponding insights you can gain from investigating them:
There are a wide range of digital assets and tools that we use for each of the steps of the marketing funnel. These include marketing campaigns, ads, and the e-Commerce site where orders can be placed. Each of these items can undergo A/B testing. In fact, it is the small but significant features that are often tested. BigCommerce lists the following features as often undergoing A/B testing:
As you can see, items which are tested can range from small details in the e-Commerce website up to entire promotional campaigns. They all play a role in the marketing funnel. A/B testing, therefore, is the main method of improving and optimizing your marketing funnel.
The number one consideration in designing your A/B testing is that you need to make sure that the only difference between version A & version B is the feature you are testing.
According to Harvard Business Review, the principles behind A/B testing started from experimental sciences and have been adopted by various industries. One of the most important industries that uses this principle is medicine, especially for testing new treatments and medications. In medicine, the new treatment or medication is tested on two groups: the control group and the treatment group. The treatment group gets the new treatment or medication while the control group gets the standard treatment which serves as a benchmark for comparison. They control almost all other details so that the only difference is the new treatment or medicine being tested. The difference in the variables related to can then be used to conclude whether the new treatment or medicine being tested is effective or not.
Similarly, A/B testing involves controlling all factors and variables, isolating a single feature we want to test.
Let’s say you plan to make changes to your e-Commerce website, and you want to make sure that it works by doing A/B testing. What are the steps?
When we say initial research, we are referring to the process of looking for new trends and results that have been published regarding digital marketing in general and in the specific industry your business is part of. This is a good start; even if your business is doing well, you still want to stay ahead of your competition. For example, if you want to update the look and feel of your e-Commerce website, your initial research will help you look for the best ways to improve the look-and-feel of your website.
An initial assessment, however, will allow you to determine if your metrics are getting better or getting worse. For your e-Commerce website, you also need to regularly check your performance through its metrics and change whatever negatively affects the growth of your business. Initial research and assessment will help you narrow down what can be changed in order to improve your metrics. These will serve as your goals in A/B testing.
You would discover via your initial research and assessment that there are several possible solutions to your problem, whether that be to further improve the performance of your e-Commerce website or to modify your ads. You will need to choose which solutions are the best to test. What you get here is what is called a hypothesis.
After identifying your hypothesis, you need to implement the best changes as recommended by your initial research and prepare to set them up in your e-Commerce website, for example. Additionally, you need to know how long the experiment should run. This depends on the amount of data points you need, or what is called sample size. One tool to help you calculate the sample size is A/B Test Sample Size Calculator (Optimizely).
Make sure you have adequately set up your experiment such that the users can randomly get the original setup or the variation that you want to test. This is called randomization. Randomization is important because it ensures that the circumstances of the users won’t affect the results of the testing, and that all significant differences are due to the changes implemented.
You also need to prepare to group the users into blocks depending on varying factors and user characteristics.This is called blocking. For example, one way to group the users into blocks is by separating desktop/laptop and mobile users into their own groups. Randomization is then applied within these groups.
After setting up everything for the experiment, you can then launch the test. You should wait for some time since it is important that you reach the optimal sample size. With the optimal sample size, you can make reasonable conclusions from the data.
In the case of an e-Commerce website, half of its visitors will get the original website while the other half will get the website with modifications. While the users visit the website, various relevant metrics are being measured and recorded, such as bounce rate, conversion rate, and revenue.
One common misconception on testing is that you can only test them one-at-a-time. Harvard Business Review made it clear that sequential A/B tests don’t capture the whole picture as they often leave out the possible effects of interactions between two or more changes implemented at the same time. More complex tests are then recommended (A/B/n testing), which may involve changing two or more features at the same time. A/B/n Testing can take longer than simple A/B tests as it will take more time to collect the optimal sample size, but the interactions between the modified features are recorded.
After gathering a sufficient amount of samples, we can now analyze the data. Since A/B testing involves testing whether our proposed changes will work, then we essentially apply hypothesis testing, which we discussed in another article entitled Data Analysis 101: The types of analysis you can conduct. You can use existing tools to run A/B testing with automatic analysis of data. For example, Google has its own built-in A/B testing setup.
There is one thing you need to be aware of, and they are called external validity threats. External validity threats are external events or factors that can impact, or more accurately, disturb your results. Shopify gives the following as examples:
These events can affect the data and the resulting conclusions from it. You need to run the tests so that you can minimize the effect of these events to your data and your conclusions.
Using the results of data analysis, you can make the decision whether to implement the proposed changes or not.
Even once you have made the decision to implement the proposed changes, you still need to be aware of unexpected changes in the future which can either enhance or reduce the effect of the implemented changes.
As search engines regularly crawl the Internet, they can easily record snapshots of websites even in the midst of A/B testing. Fortunately, there are technical changes we can do to preserve the integrity of our e-Commerce website and their ranking in search engines. Some of them are the following, as recommended by Google and compiled by Optimizely, partially verbatim:
Using automated A/B testing tools can help in handling the technical aspects of A/B testing that may affect your SEO rankings.
A/B testing is the main process of testing whether a planned set of changes in our digital marketing strategy will work, whether it be features of our e-Commerce site, the content of our ads, or even our online services such as customer service.
In doing A/B testing, we first perform research and conduct a thorough assessment in order to identify which features would benefit most from changes. The chosen changes are then deployed to half of the users to see how they fare. Changes that improve our metrics will then be permanently implemented. We hope you have enjoyed our Ultimate Lido A/B Testing Guide.
What is A/B testing? How online businesses increase…
“A Refresher on A/B Testing” HBR
A/B Testing – Optimization Glossary
The Complete Guide to A/B Testing: Expert Tips from Google, HubSpot and More
More Marketers Are Running 'Turn-Off' Experiments. Should You?
How to Measure the Success of an A/B Test Plan
Which Metrics You Must Track While A/B Testing
Choosing the right optimization KPI for your A/B tests By addressing some common A/B
