Getting Started With A/B Testing

At Reflective Data, we believe that A/B testing is one of the best uses for all the data you gather in your digital analytics system. Of course, there are unlimited other uses like spotting when something is broken or calculating the ROI for your latest campaign but these are mostly just acknowledging what has happened. A/B testing is (should be) taking real actions based on real data.

In this article, we are going to give you a comprehensive overview of how to get started with A/B testing, including the prerequisites, tools, and methodologies. We are even going to give you some test ideas to get you started! In case you are already running A/B tests, I’d still suggest you take a look how others (we) approach the problems involved.

How A/B testing works

A/B testing or split testing is a method of comparing two versions of a webpage against each other to determine which one performs better. Yes, it’s that simple. Or is it?

AB testing is essentially an experiment where two (or more) variants of a page are shown to users at random, and statistical analysis is used to determine which variation performs better for a specific conversion goal. In e-commerce, the most common goals are the number of purchases and of course, the revenue. Although, you could also measure the micro-conversions or whatever KPI is relevant to your business.

The main goal is to take guessing out of your website optimization process, and eventually, from your business decisions. It is much wiser to take actions based on real evidence, statistically significant test results in this case.

To put it very simply, the process of A/B testing is as follows:

Analyzing information you have

First, you have to know the website you are working with. The main KPI-s, the funnel, the audience etc.

Next step is taking a look at the analytics in order to spot any trends, issues or other useful insights that you could use for coming with the ideas (hypotheses) for the experiments.

If you don’t have enough data or you have squeezed everything out of it, get more data. Take a look at tools like Reflective Data Platform that will give access to features like Form Analytics, Heatmaps and On-Site Polls.

Coming up with the hypotheses

In this step, the main goal is to figure out what should be tested. We do that based on the information gathered in the previous step.

The more hypothesis the better, but keep in mind that they should be backed up by real data.

A hypothesis should be formulated similarly to this example: People don’t click the main call to action (CTA) because it’s too low on the page. Moving it above the fold would raise the CTR.

Putting hypotheses into practice

You have a hypothesis, now it’s time to bring it to life. The first step here is usually to draw a wireframe of the solution. In our example, it would show where the CTA would be in the variation.

If it’s a more complicated test, a design file might be needed.

When you have figured out exactly how your experiment is going to look like, it is time to build the actual test, using an A/B testing tool such as Google Optimize. Once the setup is done, it is important to make sure it looks as it should on every device and browser that is being targeted. For that, most tools have a preview mode.

If all the bugs are fixed, it’s time to launch the test to the public!

Analyze & document the results, then repeat the process

Here is a nice visualization of a single A/B test and how it directly affects the revenue.

Once the test has reached the statistical significance, it can be stopped and the process of making conclusions may begin.

It is also important to document the hypotheses, experiments, and their results. That gives us an opportunity to evaluate the entire testing program and make it more effective over time.

If the test is a winner, meaning that the main KPI is up in the variation, I would suggest considering implementing the change.

If it was a loser, try to find out why your hypothesis didn’t work, maybe find a better form of execution and run it again.

In every experiment, there should be a learning point and every upcoming test a bit better than the previous ones.

Are you and your website ready for A/B testing?

While I believe that A/B testing is the best way of turning analytics data into real-life decisions, unfortunately not all websites are in the phase where they could actually run meaningful experiments.

Website requirements

In A/B testing, a winner is determined by statistical models. Some of the most popular ones in use today are Bayesian and Frequentist, while different in many ways, they both have something in common – the sample size.

Sample size means the number of visitors that are part of your experiment, in an A/B test that sample would be equally split between original and the variation. You could lower the sample size by running experiments that have a bigger effect (but how could you be sure) or getting satisfied with lower confidence rates (not a good idea).

Take a look at this calculator to make sure if your website can conclude an A/B test in a reasonable timeframe (less than 8 weeks).

There is one more calculator available by Optimizely.

Skillset requirements

While getting started is not too difficult, there is a good set of skills you need to achieve before you become a really good tester that will bring a real value to whichever business they are working with.

If your website has enough traffic (see the calculators above), take a look at the skills you should consider learning. And if your website is not eligible yet, work on both – getting the traffic and learn the skills, so you would be ready once the site is.

PS! These skills could be all in one person, in a team or even outsourced.

Analytical skills

In order to drive actionable insights from your data, you would have to know the tool that you are going to use. In most cases that tool will be Google Analytics.

Being able to pull out all sorts of reports is not enough, you need to know what it all means. Try looking for trends, segments that have extremely high or low performance numbers, and figure out what are the points in your funnel that seem to cause the most friction so that many visitors drop there.

Besides building custom reports, you will also need customized dashboards, ones that are the most relevant to your business.

Here’s a screenshot of a general performance dashboard for one of our test pages.

Design skills

Quite often aA/B test is much more than just a simple copy, image or color change. To pull these changes off, there has to be a proper design.

Many teams already have a designer, if yours does, ask him/her to help you. If you don’t have a designer to help you, you are going to have to outsource or try your best.

Coding skills

To set an experiment live there is almost always some coding involved. Yes, most tools come with a visual editor feature, but that is only good for most basic changes.

Coding skills required for setting up an A/B test are pretty similar to those needed from a front-end web developer. Keywords are Javascript, jQuery, HTML and CSS.

If you can’t code and don’t know someone who could, play around with the visual editor and start learning now! Or hire someone that can.

Presenting and documenting skills

If you work in a company then most probably you will have to report your experiments and results to your team. Good presenting skills come in extra handy here. You are going to have some winning tests, some neutral tests and probably a good amount of losing tests. You have to communicate that there’s value in each and every one of those tests.

Documenting is a very important part of every A/B testing program. Even if you run a single test every month it will be extremely hard to remember all the important details a year after. There are good tools created specifically for that purpose but for beginners, an Excel document is just fine.

How to the right tool?

The market for A/B testing tools has become much wider in the past years. That means that everyone has to work hard to make theirs the best one but it also means that for a beginner it can be rather difficult to make a right pick.

And picking the right tool is important because just like with most software, switching them later is a pain in the ass.

Google Optimize (free version)

It is free. That means that if you’re new to A/B testing and you are just figuring out if it fits your business needs etc. then there is almost no risk involved.

The number one problem with Google Optimize is that the free version is limited to three simultaneous experiments. That means you can’t run more than three tests at a time, of course, more can be in draft or ended.

Here’s a full guide that will help you get started with Google Optimize.

And this is what it feels like to use Google Optimize for three moths.

Click here to sign up and create your first experiment today, for free!

Optimizely

Optimizely is a great tool, it has been in the business since 2010. It was founded by Dan Siroker, a former Director of Analytics for the Obama campaign.

Optimizely has been my personal favorite tool for years, their approach is right in so many ways. It is logical, easy to use and the most developer-friendly tool out there.

Here’s a list of things I like about Optimizely

• Easy to install, add a snippet and voilà.
• Developer friendly, everyone who is using code instead visual editor will like it.

Some of the issues I’ve seen people have are:

• It is expensive, for a smaller company I’d suggest starting with something free and once you are sure that A/B testing is for you, move on to more advanced tools.
• Somewhat slow support, I believe it’s faster for enterprise clients but for medium-sized companies it can be frustrating.

Learn more on their Website.

VWO

Visual Website Optimizer, another very popular A/B testing platform that is being used by thousands of companies across the world.

In many ways, VWO is similar to Optimizely but it also has some user behavior analysis features built-in – heatmaps, surveys and website reviews.

I used to like Optimizely a lot more but in a past year I find the two being almost equal when it comes to general A/B testing related features.

Things I like about VWO

• Nice interface, it is very pleasant to use. Everything is where you’d expect it to be.
• Very good support, their support team is top notch!

Reasons I would consider not getting VWO as my main testing tool.

• Expensive, for starters I’d suggest a free tool such as Google Optimize.

Take a look at their website.

Wasabi (open source)

Wasabi is an open-source A/B testing platform for large companies and seasoned developers.

There are two cases where I’d suggest Wasabi:

• Full control, if your company can’t afford to have their data in someone else’s servers.
• If you love coding, if you’re a developer that likes challenges, try getting Wasabi up and running 😉

Read more and get started on their Github.

DIY

The logic of basic A/B testing is not a rocket science. If your developer has free time or if you want a tool that is tailored to your business, build your own tool!

You could build it on the front-end or on the server side, all of the results should go directly to the analytics platform of your choice. A good combo would be Google Tag Manager and Google Analytics.

Take a look at GTM Testing and take it from there.

Sentient Ascend

If regular A/B testing is not enough, take a look at Sentient Ascend. It leverages the power of AI and evolutionary algorithms to detect a winner out of thousands of combinations.

If this sounds interesting, take a look at our comprehensive overview.

Coming up with new test ideas

First of all, test ideas should be backed up by real data. One of the best sources for insights is your analytics platform, Google Analytics for most websites.

Analytics

If you haven’t looked at your data lately, there is a good chance that you will find something that is broken. No need to A/B test that, just fix it ASAP. You will probably also find a good amount of things that are so-so, these you should test!

Example: You see that a lot of people drop in the last step of your checkout, the part where they have to insert their credit card info. Now, you have to think of the reasons why that could be. Maybe that area is not trustworthy enough? Test adding trust badges next to CC fields!

Real users

Take a look at your website, try acting like a real visitor, go through the funnel and try to detect what could be improved. Even better, ask someone not related to your business to do so.

Example: You ask your friend to go visit your e-commerce website and buy a specific product. Stand behind them and see what kind of issues they have. Maybe they can’t find the right navigation item, or perhaps the search box is not prominent enough? Try making it more visible!

Others

Other tools that will provide you with the useful insights are:

• Form analytics tools
• Heatmaps (click, move, scroll)
• On-Site Polls
• Session recordings

Example: When taking a look at your heatmaps report you notice that a lot of visitors click on a picture on your website. The problem is, that picture is not meant for clicking and nothing happens. Try giving it a relevant click action, a zoom maybe?

Heatmaps, form analytics and on-site polls are included in Reflective Data Platform.

Conclusion

A/B testing is becoming more and more popular every day. Still, there are a lot of websites that aren’t running a single experiment, that is a lost opportunity.

If you have a website with a decent amount of traffic, start testing! There is nothing to lose, start with a free tool such as Google Optimize.

In case you have a website with traffic which is too low for testing, get yourself familiar with the basics while working on getting more traffic. It is worth it, you will be making tons more money!

When picking a tool, take a look at each of their websites, ask for a demo, find someone who is using them and get a real-life overview. If you are from a smaller business, go with a free option.