Basics of A/B Testing

This blog will discuss A/B Testing and its uses. Learn what A/B Testing is, why to use it, and how to use it.

TABLE OF CONTENTS
  1. What is A/B testing?
  2. Why use A/B testing?
  3. What can be tested through A/B testing?
  4. What is the main consideration for A/B testing?
  5. How is A/B testing conducted?
  6. What are some technical details needed in implementing A/B testing?
  7. Summary
  8. References 

Table of contents
Chapter 1
Chapter 2
Chapter 3
  1. What is A/B testing?
  2. Why use A/B testing?
  3. What can be tested through A/B testing?
  4. What is the main consideration for A/B testing?
  5. How is A/B testing conducted?
  6. What are some technical details needed in implementing A/B testing?
  7. Summary
  8. References 

10 Minutes

In a perfect world, we would always know exactly which marketing strategies, website arrangements, and e-Commerce campaigns will be most effective. There is often a plethora of possible but untested solutions and approaches that could feasibly provide the best method. The difficulty is determining which of these strategies is the most optimal for any given sales, marketing, or e-Commerce situation. 


Wouldn’t it be great if we could compare, in practice, the effectiveness of a variety of different methods? Well, it just so happens that we can, using a procedure known as A/B testing.


This article defines A/B testing, what it can be used to test, and the steps for conducting A/B testing.

What is A/B testing?

A/B testing lets you compare a proposed set of changes to your digital marketing strategy with your existing strategy. Image source

BigCommerce defines A/B testing, also called split testing, as a procedure where two versions of the same thing are deployed to each half of the intended audience. It can be a makeover of the e-Commerce store; a new product, feature, or a service; or a new marketing campaign. It can also be about deploying a small fix and testing whether it improves our business or not. 


Often the effectiveness of a new version is measured using its conversion rate. Below are a few types of commonly tested conversion rates:

  • Product sales
  • Subscription to service
  • Newsletter subscriptions
  • Survey responses


If more than two versions need to be tested, you can use A/B/n testing, which is a modified form of A/B testing for testing more than two versions of a strategy at the same time.. 

Why use A/B testing?

There are several benefits to A/B testing besides being able to test whether or not a feature you want to implement or change will work. According to Taplytics, A/B testing also allows you to:

  • Gather valuable messaging feedback
  • Gather detailed user data
  • Increase conversions and engagement
  • Increase user trust
  • Limit negative impact of projects


These benefits will come from a detailed analysis of the resulting data from A/B testing. In order to take the most advantage of this information, you will  need a system for storing the data and doing further analysis. 

What can be tested through A/B testing?

The summary of the marketing funnel. Image source

You can use A/B testing to test features for all steps in the marketing funnel. ClickFunnels defines the marketing funnel as “the entire journey from a person being aware of your business to that person becoming a committed customer.”


There are three main steps in the marketing funnel, with the corresponding insights you can gain from investigating them:

  1. Lead generation: generating awareness of your brand. This is how your customers initially discover your product or business. 
  2. Lead nurturing: converting interest to intent. Customers often ask for additional information before they decide to buy your product or pay for your service. 
  3. Sales: fulfilling the intent to buy by actually paying for the product and/or service. At this point, the customers make the critical decision to buy your product or pay for your service. There are several things going on in their mind when making that decision. 


There are a wide range of digital assets and tools that we use for each of the steps of the marketing funnel. These include marketing campaigns, ads, and the e-Commerce site where orders can be placed. Each of these items can undergo A/B testing. In fact, it is the small but significant features that are often tested. BigCommerce lists the following features as often undergoing A/B testing:

  • Calls to action
  • Images
  • Body content, down to individual words
  • Social proof
  • The navigation structure
  • The check-out process
  • Prices
  • Promotional offers
  • Trial-offer lengths
  • Free delivery


As you can see, items which are tested can range from small details in the e-Commerce website up to entire promotional campaigns. They all play a role in the marketing funnel. A/B testing, therefore, is the main method of improving and optimizing your marketing funnel.

What is the main consideration for A/B testing?

The number one consideration in designing your A/B testing is that you need to make sure that the only difference between version A & version B is the feature you are testing. 


According to Harvard Business Review, the principles behind A/B testing started from experimental sciences and have been adopted by various industries. One of the most important industries that uses this principle is medicine, especially for testing new treatments and medications. In medicine, the new treatment or medication is tested on two groups: the control group and the treatment group. The treatment group gets the new treatment or medication while the control group gets the standard treatment which serves as a benchmark for comparison. They control almost all other details so that the only difference is the new treatment or medicine being tested. The difference in the variables related to  can then be used to conclude whether the new treatment or medicine being tested is effective or not.


Similarly, A/B testing involves controlling all factors and variables, isolating a single feature we want to test. 


How is A/B testing conducted?

The quality of the data determines how useful your A/B testing will be. Image source

Let’s say you plan to make changes to your e-Commerce website, and you want to make sure that it works by doing A/B testing. What are the steps?

Step 1: Initial research and assessment 

When we say initial research, we are referring to the process of looking for new trends and results that have been published regarding digital marketing in general and in the specific industry your business is part of. This is a good start; even if your business is doing well, you still want to stay ahead of your competition. For example, if you want to update the look and feel of your e-Commerce website, your initial research will help you look for the best ways to improve the look-and-feel of your website.


An initial assessment, however, will allow you to determine if your metrics are getting better or getting worse. For your e-Commerce website, you also need to regularly check your performance through its metrics and change whatever negatively affects the growth of your business. Initial research and assessment will help you narrow down what can be changed in order to improve your metrics. These will serve as your goals in A/B testing. 


Step 2: Preparing what to test 

You would discover via your initial research and assessment that there are several possible solutions to your problem, whether that be to further improve the performance of your e-Commerce website or to modify your ads. You will need to choose which solutions are the best to test. What you get here is what is called a hypothesis. 


After identifying your hypothesis, you need to implement the best changes as recommended by your initial research and prepare to set them up in your e-Commerce website, for example. Additionally, you need to know how long the experiment should run. This depends on the amount of data points you need, or what is called sample size. One tool to help you calculate the sample size is A/B Test Sample Size Calculator (Optimizely).


Make sure you have adequately set up your experiment such that the users can randomly get the original setup or the variation that you want to test. This is called randomization. Randomization is important because it ensures that the circumstances of the users won’t affect the results of the testing, and that all significant differences are due to the changes implemented. 


You also need to prepare to group the users into blocks depending on varying factors and user characteristics.This is called blocking. For example, one way to group the users into blocks is by separating desktop/laptop and mobile users into their own groups. Randomization is then applied within these groups.



Step 3: Launch the test 

After setting up everything for the experiment, you can then launch the test. You should wait for some time since it is important that you reach the optimal sample size. With the optimal sample size, you can make reasonable conclusions from the data. 


In the case of an e-Commerce website, half of its visitors will get the original website while the other half will get the website with modifications. While the users visit the website, various relevant metrics are being measured and recorded, such as bounce rate, conversion rate, and revenue.


One common misconception on testing is that you can only test them one-at-a-time. Harvard Business Review made it clear that sequential A/B tests don’t capture the whole picture as they often leave out the possible effects of interactions between two or more changes implemented at the same time. More complex tests are then recommended (A/B/n testing), which may involve changing two or more features at the same time. A/B/n Testing can take longer than simple A/B tests as it will take more time to collect the optimal sample size, but the interactions between the modified features are recorded.


Step 4: Analyze the data 

After gathering a sufficient amount of samples, we can now analyze the data. Since A/B testing involves testing whether our proposed changes will work, then we essentially apply hypothesis testing, which we discussed in another article entitled Data Analysis 101: The types of analysis you can conduct. You can use existing tools to run A/B testing with automatic analysis of data. For example, Google has its own built-in A/B testing setup


There is one thing you need to be aware of, and they are called external validity threats. External validity threats are external events or factors that can impact, or more accurately, disturb your results. Shopify gives the following as examples:

  • Black Friday Cyber Monday (BFCM) sales
  • A positive or negative press mention
  • A major paid campaign launch
  • The day of the week
  • The changing seasons


These events can affect the data and the resulting conclusions from it. You need to run the tests so that you can minimize the effect of these events to your data and your conclusions.



Step 5: Make the decision 

Using the results of data analysis, you can make the decision whether to implement the proposed changes or not. 


Even once you have made the decision to implement the proposed changes, you still need to be aware of unexpected changes in the future which can either enhance or reduce the effect of the implemented changes. 



What are some technical details needed in implementing A/B testing?

As search engines regularly crawl the Internet, they can easily record snapshots of websites even in the midst of A/B testing. Fortunately, there are technical changes we can do to preserve the integrity of our e-Commerce website and their ranking in search engines. Some of them are the following, as recommended by Google and compiled by Optimizely, partially verbatim:

  • No cloaking - Cloaking is the practice of showing search engines different content than a typical visitor would see. Cloaking can result in your site being demoted or even removed from the search results.
  • Use rel="canonical" - If you run a split test with multiple URLs, you should use the rel="canonical" attribute to point the variations back to the original version of the page
  • Use 302 redirects instead of 301s - This tells search engines such as Google that the redirect is temporary, and that they should keep the original URL indexed rather than the test URL.
  • Run experiments only as long as necessary - Running tests for longer than necessary, especially if you are showing one variation of your page to a large percentage of users, can be seen as an attempt to deceive search engines. Google recommends updating your site and removing all test variations of your site as soon as a test concludes and to avoid running tests for excessive periods of time.


Using automated A/B testing tools can help in handling the technical aspects of A/B testing that may affect your SEO rankings.


Summary

A/B testing is the main process of testing whether a planned set of changes in our digital marketing strategy will work, whether it be features of our e-Commerce site, the content of our ads, or even our online services such as customer service. 


In doing A/B testing, we first perform research and conduct a thorough assessment in order to identify which features would benefit most from changes. The chosen changes are then deployed to half of the users to see how they fare. Changes that improve our metrics will then be permanently implemented.


References


What is A/B testing? How online businesses increase… 

“A Refresher on A/B Testing” HBR

A/B Testing – Optimization Glossary

The Complete Guide to A/B Testing: Expert Tips from Google, HubSpot and More

More Marketers Are Running 'Turn-Off' Experiments. Should You?

How to Measure the Success of an A/B Test Plan

Which Metrics You Must Track While A/B Testing

Choosing the right optimization KPI for your A/B tests By addressing some common A/B

Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.
SECTIONS
  1. What is A/B testing?
  2. Why use A/B testing?
  3. What can be tested through A/B testing?
  4. What is the main consideration for A/B testing?
  5. How is A/B testing conducted?
  6. What are some technical details needed in implementing A/B testing?
  7. Summary
  8. References