This came as an unwelcome surprise for anyone working in the experimentation industry. Even if you didn’t use the tool itself, it was the first tool most newcomers used to get themselves into experimenting on their websites.

For a while, Google Optimize going away was everything people talked about on Twitter and LinkedIn.

Exporting Experiment Data From Google Optimize

Since several of our existing clients asked for it, we built Google Optimize Data Exporter to store your experiment data for as long as you need it. It supports almost any data destination, including popular ones like Google BigQuery, Amazon S3 and Snowflake.

Google Optimize Data Exporter runs on the same robust and scalable Reflective Data Infrastructure that you hopefully already know and love.

We’ve made the process of exporting your Google Optimize data as simple as possible. Here’s a quick overview.

1. Planning and scoping

Get in touch with one of our data analysts to plan your Google Optimize Data export. The main questions to answer are the list of experiments, dimensions, metrics, time frames and the data destination you wish to use for your Optimize data export.

2. Data export and storage

Executing the plan. Our data analyst will configure Reflective Data Export System to pull the requested data from your Google Optimize instance and store at your chosen data storage destination.

Most exports use Google BigQuery as a data destination.

3. Reporting and consultation

The post Export Experiment Data From Google Optimize – While You Still Can appeared first on Reflective Data.

In this article, we are going to explore some of the ways we can leverage Google Analytics to track churn, LTV and other really useful metrics.

Depending on the software you’re using, there may be some off-the-shelf solutions that you can install. For example, if you’re on Shopify then you can use something like Littledata to send a more accurate LTV value into a custom dimension in Google Analytics. More often than not, though, there is no good solution available or you just need more control over the setup.

One common misconception is that such long-term retention metrics are relevant for a few specific business types only. Yes, metrics like churn are vital for SaaS and subscription products but any company that gets return business should have their long-term KPIs in place. And I don’t mean simply tracking them but actually analyzing them and optimizing the business with those metrics in mind.

Acquiring a new customer is anywhere from five to 25 times more expensive than retaining an existing one. It makes sense: you don’t have to spend time and resources going out and finding a new client — you just have to keep the one you have happy.

Harvard Business Review

So, if you’ve been focusing on getting new customers and metrics like revenue or transactions, this article is just for you!

How to measure retention metrics like LTV and churn

The long-term retention metrics most relevant to you depend on the type of business you’re working with but the most common ones are customer lifetime value (LTV) and churn. Below is a list of other popular retention KPIs. Think about the ones that would be relevant for your business.

Common Customer Retention Metrics

1. Customer Churn
2. Revenue Churn
3. Existing Customer Growth Rate
4. Repeat Purchase Ratio
5. Product Return Rate
6. Days Sales Outstanding
7. Net Promoter Score
8. Time Between Purchases
9. Loyal Customer Rate
10. Customer Lifetime Value

Source

Almost all retention metrics require a proper User ID implementation. This means you’d have to identify the user over time and even if they’re using multiple devices or browsers. Luckily, in most cases, actions like completing a purchase or signing up for a subscription do involve some kind of authentication.

While it’s possible to track retention metrics with Google Analytics alone, in most cases you’d get much better (more accurate) results when combining it with some other technology. Let’s explore two of the more popular options.

Sending retention data into Google Analytics

This solution involves sending retention data into a custom dimension or a custom metric in Google Analytics.

The exact workflow depends on the software (CRM, CMS, database etc.) your site uses but the general process would look something like this.

1. Create a custom dimension in Google Analytics (should be user-scoped)
   
2. For logged-in/identified users, pull/calculate the values for the relevant retention metrics from a database or other system (CRM, CMS etc.)
   Something like this if your order data is stored in BigQuery.
   
3. Make the retention metrics available in the data layer
   
4. Use Google Tag Manager to send your retention metrics to Google Analytics, using the custom dimension or metrics slots/indices according to how you configured them in step #1

Now, having this data available in Google Analytics, you do whatever you want with it. Here are a few examples.

Using LTV in a Google Analytics custom report

LTV in the Google Analytics user explorer report

Notice the difference between LTV that Google Analytics is reporting by default ($439) and the value we see in our custom dimension ($2,016). This is because Google Analytics can’t keep a track of the user as accurately as your backend system or an e-commerce platform you’re using. The same goes with other retention metrics, getting accurate metrics requires some custom work.

The list of possible use cases for this kind of data is unlimited. For example, think about creating custom segments in Google Analytics for customers that are in the top 10% in terms of LTV and see what differentiates them from the rest of the visitors. Besides making more/larger purchases, of course. Things like their traffic source, what pages they landed on, what A/B test variants they saw etc. can be quite insightful.

Talking about ways you can analyze your data and the insights it will give you. Let’s take measuring retention metrics to a whole new level by sending data into a data warehouse.

Storing data in a data warehouse

If you’re just getting started with retention metrics and you still mostly optimize for generic metrics like leads, total transactions and total revenue, then you’ll still be better off with having them in Google Analytics. Compared to not having them at all, that is. If you’re serious about analyzing and optimizing for retention and customer lifetime value then you need a data warehouse.

Here’s a quick step-by-step guide that will lead you in the right direction.

1. Send all Google Analytics data into a data warehouse (i.e. BigQuery). Tools using the Reporting API (most of them) can get you started but for true unsampled hit-level data you need something like Parallel Tracking.
2. Send, pull, push data from other relevant sources into your data warehouse. This should include your database, CRM, CMS, marketing tools, ads platforms, customer support, live chat and every other tool that has data about your customers and their interactions with your brand. Self-service tools like Stitch will get you started but we’d recommend more flexible managed solutions.
3. Solution for accessing data stored in your data warehouse. You’d need something (could be separate tools) that can handle ad-hoc queries, dashboarding, automated reports, and building data models. Tools like Google Data Studio will get you started. Something like Looker or Tableau would be better. Our recommendation is to go with a managed service that will put together the best set of tools for you and configure the rest as well.

If having retention metrics in Google Analytics enabled you to do all sorts of useful new reports and analysis then your options with the setup above are truly limitless.

Having a proper data warehouse will be your competitive advantage. Not only does it give you an option to get a very good overview of the current status of your business and your customers, but it will also allow for truly optimizing the user experience and user journey. Leading to a better user experience and improved retention metrics. Remember, acquiring a new customer is anywhere from five to 25 times more expensive than retaining an existing one!

One way to describe the usefulness of a data warehouse is by giving you a few sample questions. Questions that would be very difficult to answer without a data warehouse.

• Purchases from which traffic channels are most likely to be refunded at some point in the future? Might lead to revising your marketing budget.
• Which traffic sources have the highest retention/LTV?
• What is the correlation between subscription value ($) and churn rate?
• What is the long-term impact of your campaigns or A/B experiments? Do quick wins lead to higher churn or lower LTV?
• Does data from different sources add up? Maybe Google Analytics is missing some transactions that are in Shopify or perhaps some of them are duplicates?

Here’s an example of the last one on the list above.

As you can see, Google Analytics is missing a good amount of transactions and this requires further investigation. Definitely something you should include in your Google Analytics dashbaord.

This was just a short list of ideas to get you thinking about what is possible with a proper data warehouse. You can trust me when I say that I’ve seen companies become truly data-driven after they’ve implemented a tailor-made data warehouse and started digging for insights they couldn’t before.

Working with automatically recurring events

It is important to keep in mind that some retention metrics can change without the user themselves taking any action. You need to make sure that those cases are being tracked and taken into account. Here are a few examples.

• Recurring orders/payments
• Subscription expirations
• Payment method expirations
• Orders being changed/cancelled (i.e. due to missing item)

If your data warehouse was configured properly, you should have this data already available. Just make sure to include it in your analysis and reports.

In case you don’t have a data warehouse and you’re trying to solve this with Google Analytics alone then you need to use the Measurement Protocol. Some of the more common subscription platforms like ReCharge for Shopify have this built-in or solvable with some 3-rd party solutions but quite often custom development is required. In which case, you should think about implementing a data warehouse instead.

Conclusion

If you’re in a business where customers are expected to generate value more than once (repeat purchase, subscription etc.) then you need to start focusing on your retention metrics.

Google Analytics can get you started with the basic metrics and limited accuracy. A much better setup would be Google Analytics combined with Parallel Tracking and if you’re serious about optimizing for those metrics then you need a custom-built data warehouse where all marketing data is pulled together.

Feel free to post your ideas and questions in the comments below. If you’d like to get some consultation and discuss your ideas further, get in touch with us.

The post Measure Long-Term Metrics Like Customer Lifetime Value (LTV) Using Google Analytics appeared first on Reflective Data.

About the client

Our client, in this case study, is a large global e-commerce website, headquartered in the US.

Their main focus is on men’s clothing and they’re specialising in dress shirts, as well as ties, suits, casualwear, shoes and accessories.

Depending on a season, their monthly online revenue is between 10 and 15 million USD. This comes from around 45k monthly transactions.

Due to NDAs, we are not allowed to disclose the name of the client. Everything else in this case study remains unchanged.

The challenge

Working with skewed data can be worse than having no data at all. This is why we’re always promoting all sorts of analytics audits and making sure all data sources agree with each other. At the very least, you should know why the numbers in different tools don’t match (i.e. analytics doesn’t include offline sales but backend does).

Our client in this case study contacted us with a quite specific problem. They were running a decent CRO program with 4-6 A/B experiments running every month. The problem they had with the program, though, was that the numbers they saw in their testing tool Optimizely, Google Analytics and backend didn’t match. In fact, there was a ~35% discrepancy overall.

While it is almost impossible to reach 100% accuracy with Javascript based tools due to ad-blockers, cookie deletion etc. and some tools have different logic for calculation metrics like conversions, sessions, users — we knew that this discrepancy is way off the limits and immediate action is needed.

This kind of discrepancy was even more concerning as the client was actively running their CRO program. It wasn’t rare for them to see test results flip based on where they got their numbers (Google Analytics vs Optimizely). This put their entire CRO effort under question internally and the management wasn’t far from shutting it down. The results just weren’t trustworthy enough.

The solution

Even though the client described some rather clear symptoms, it is necessary to always start with a proper diagnosis of the problem. This way we can be sure that there indeed is a discrepancy between all these tools – in some cases, the problem can be in how you understand/read the numbers. Another goal for the diagnosis was to understand the size/reach of the problem.

Diagnosis

The very first thing we did to diagnose and measure the discrepancy was to get access to all data sources. That included the testing tool (Optimizely), Google Analytics and the client’s backend numbers. Next, we made sure the metrics are comparable (i.e. all the transactions we see in the backend are indeed supposed to show up in Google Analytics as well). We then put those numbers side-by-side in a spreadsheet and grouped them by date.

After analyzing the daily numbers, we came to the conclusion that there is indeed an average of 35% discrepancy between Optimizely and Google Analytics (less transactions in the testing tool). The discrepancy between Google Analytics and backend, on the other hand, was rather small — just below 3%. This can be considered OK for a Javascript-based analytics tool.

The discrepancy between Google Analytics and Optimizely meant we had to dig deeper on that side. Unfortunately, Optimizely doesn’t give access to the raw underlying transaction ID level data by default. Luckily, though, we were able to get this data from their support team. With Google Analytics, exporting transaction-level data was doable with a custom report and CSV export. For the Google Analytics dataset, we included dimensions that would give us some extra context — payment gateway, products, device category, country etc.

After pulling transaction-ID-level data from Optimizely and Google Analytics in spreadsheets, we wrote a quick function that would mark the transactions in Google Analytics that didn’t show up in Optimizely. Then it was time to start looking for some patterns. Is there something in common for all missing transactions?

This is how one version of this spreadsheet looked like. This was to highlight transactions in the backend that didn’t show up in Google Analytics.

Implementing the fixes

Since the website and payment processing system for this website were rather complex, it turned out there are more than one such patterns. They were all related to the use of various payment gateways, device categories and local sub-sites for specific countries.

In order to come up with a proper solution, we had to dig even deeper into how transaction tracking was set up for both Google Analytics and Optimizely. Luckily, we were able to detect the shortcomings quite fast and fixing them wasn’t rocket science either. After a few rounds of testing and a couple of A/A tests run in Optimizely, we were able to get the discrepancy number from 35% down to 2.5%.

The main reason for this mismatch was that while Google Tag Manager was in use to unify transaction tracking for Google Analytics, tracking snippets for Optimizely were all over the place. The solution involved moving all Optimizely tracking into Google Tag Manager and use the same triggers that were being used for Google Analytics.

Going forward

Since the site has a rather complicated structure with multiple payment gateways and different pages for different countries/markets, we wanted to make sure that after implementing the fixes, the discrepancy would not start sneaking in again.

The solution here was a proper monitoring system that would get data from Google Analytics, Optimizely and the backend. Since the client was already on Google Cloud, we recommended using BigQuery as their data warehouse technology. Besides being really fast and cost-effective, BigQuery has some of the best integrations which makes implementing and using it as a marketing data warehouse much easier.

Getting data into BigQuery

For Google Analytics data, we used a technology called Parallel Tracking. This allowed us to send raw hit-level unsampled Google Analytics into BigQuery. In real-time and for much cheaper than using Google Analytics 360.

To get data from Optimizely into BigQuery, we built a custom connector using Optimizely API. This solution allowed us to pull only the information that was necessary and make sure it’s always up to date.

Since the client had a custom backend infrastructure, we built a custom connector between that and BigQuery as well.

All connectors are now running on Google Cloud.

Reporting and alerts

In order for the client to be able to check the numbers from all three sources in one place, we built a dashboard using Google Data Studio which has a native connector with BigQuery.

To detect any discrepancies as soon as possible, we also created an automated reporting system with some custom code on Google Cloud Platform. This will send an email to relevant parties every time a certain threshold is passed. So far, it has already detected a few issues the client was able to fix before larger damage was done.

Conclusion

Overall, we were quite happy with the results of this project but I believe it’s better to end with a quote from the client.

We hadn’t worked with Reflective Data before but they were recommended by our friends in the industry. We needed a quick but reliable solution to our problem. Our management was questioning the results of our CRO program and we weren’t far from getting shut down.

Team at Reflective Data was able to immediately understand our problem and jumped right into solving it. I think all together it took just around 2 weeks for them to diagnose and together with out IT team fix the problems behind this rather large discrepancy.

After great success with our first project together, we’ve continued working with them on various data-related projects around our data warehouse, reporting and dashboards.

Director of e-commerce

We are so glad about these words from our client and it’s moments like this that give meaning to our work.  By the way, their CRO program has only grown since we were able to fix the mismatch. So, in a way, we were part of saving the jobs of an entire team.

Should you have similar problems with discrepancy or other data related requirements, please contact us and I’m quite sure we can work together and find a solution.

The post Case Study: Solving The Discrepancy Between A/B Testing Tool, Google Analytics and Backend Data For a Large E-Commerce Business appeared first on Reflective Data.

In case your tool of choice is Google Optimize, you should be using their official anti-flicker snippet to minimize the flicker effect.

<!-- Anti-flicker snippet (recommended) -->
<style>.async-hide { opacity: 0 !important} </style>
<script>(function(a,s,y,n,c,h,i,d,e){s.className+=' '+y;h.start=1*new Date; h.end=i=function(){s.className=s.className.replace(RegExp(' ?'+y),'')}; (a[n]=a[n]||[]).hide=h;setTimeout(function(){i();h.end=null},c);h.timeout=c; })(window,document.documentElement,'async-hide','dataLayer',4000, {'OPT_CONTAINER_ID':true});</script>

Looking at the anti-flicker snippet (previously known as page-hiding snippet), you see that the default timeout is set to 4000ms. That means that, potentially, some of your visitors may see a blank white page for 4 seconds.

Since four seconds is a long time, many companies reduce it. A common timeout I’ve seen is 2 seconds. Changing the maximum timeout to a shorter period, although, means that there can be a good amount of people to whom the snippet occasionally times out.

When the anti-flicker snippet times out, no experiments will be loaded on that page load (people will see the control for all experiments and personalizations) which can lead to a user experience where a single user is seeing different variations in single session.

What you probably want to know is the optimal timeout period for your website. To figure that out, you’d need to know how many timeouts occur with your current setup. Let’s take a look how to set that up using Google Tag Manager and Google Analytics.

Before moving forward, you should read about how the anti-flicker snippet works.

Catching the timeout

To make this a little bit easier for us, Google has added a variable in the data layer that indicates whether the snipped has timed out or not.

You can access this variable like so:

window.dataLayer.hide["GTM-xxxxxx"] where GTM-xxxxxx stands for your Optimize container ID.

To make sure it’s not too easy, though, they don’t push this variable into the data layer as you normally would. This means you can’t access it using GTM Data Layer variable nor the Custom Javascript variable.

What you need is a Custom Javascript function variable in GTM to check the value.

Here’s the function you can use

function () { 
    if ( window.dataLayer.hide ) { 
        return window.dataLayer.hide["GTM-xxxxxxx"]; 
    } 
}

And here is the same function implemented in Google Tag Manager.

PS! In the beginning, the value is always true, it turns to false if the timeout period is over and it managed to load Optimize before this happened (the snippet did not time out).

Our recommendation is to check the value on GTM’s default DOM Ready event.

Sending data to Google Analytics

This part is quite straightforward for anyone that has used GTM to send custom events to Google Analytics so let’s get right to it.

1. Create the DOM Ready trigger

In case you haven’t configured this default trigger in GTM, go ahead do it now.

2. Create a new Tag for sending the event to Google Analytics

Create a new Tag with a tag type of Google Analytics: Universal Analytics and choose track type of Event.

These are the recommended event parameters but you can use whatever works for you.

Category: optimize

Action: timeout

Label: true/false (using the Custom Javascript function variable you created before)

Non-Interaction Hit: True (because it’s not related to a user action)

That’s it. This is all you need to start tracking Optimize anti-flicker snippet timeouts using Google Tag Manager and Google Analytics.

How to use this data?

Your goal should be to find the optimal timeout duration for your website and audience. The percentage of users who experience the timeout should be kept under 10% and you might consider excluding those with timeouts from your post-analysis process using a custom segment in Google Analytics.

Any questions or ideas? As always, feel free to post them in the comments below.

The post Tracking Optimize Anti-Flicker Snippet Timeout in Google Analytics appeared first on Reflective Data.

At Reflective Data, when we start working with a new client, we always start by figuring out what their current system consists of — and in many cases, it’s a real headache. Not to mention, when we ask the client about a specific tag or goal that they set up 6 months ago, they don’t remember anything — and the name they chose isn’t helping much either.

For several clients, we’ve really pushed the need for getting a proper naming structure across the company. And while it hasn’t always been easy, we’ve managed to fix it for almost all of them. And according to our clients, the new naming conventions made their work so much more effective.

Renaming everything is one thing but, unfortunately, it won’t provide a long-term solution. What’s needed is nomenclature guidelines and educating the team on how to use them, and the importance of it.

Some (bad) real-life examples from Google Tag Manager that we’ve seen recently:

All from a single setup with over 170 tags

1. Sending Google Pay click event
2. jquery-scrolldepth
3. UA event tracking for checkout
4. Hotjar for blog
5. Live chat widget
6. Conversion tag

Now, if your tag manager has 10 tags you could probably get away with names like this. But when you start adding more tags and/or if more than one person is working with the setup, things start getting messy.

So, how should you name your tags? Let’s start by how we renamed the very same tags.

1. GA – Event – Click – Google Pay
2. JS – Event – Scroll Depth
3. GA – Event – Clicks – Checkout
4. HJ – Main Snippet – Blog
5. Olark – Main Snippet
6. AdWords – Conversion – Purchase

There are several ways how you could approach naming your tags (or anything else) but we generally like the space-dash-space method. The key is to pick one and stick to it. Of course, formatting alone just the facade.

Here are a few guidelines that will hopefully help you a bit further:

1. Always have a documentation

So you are working on a new nomenclature for your business. That’s great, but not everyone might understand things the way you do. Even if they look at a few examples they might come up with something totally different. That is why you must have documentation for explaining it.

2. Educate your team

This goes with the previous point. You need to inform your people about the new nomenclature and explain to them how it works and why it’s really important. You must also make sure everyone is actually following the new rules and friendly remind them every time they don’t. Trust me, in the beginning, you will have to do it guide often.

3. Keep your names short

Include as little as possible but as much as needed. It is okay to use shortenings and acronyms (you should include them in the docs, though) and lose the words that are not absolutely needed.

4. Consistency is really important

Make a plan and stick to it. If you are using an acronym, always use the same acronym. Don’t change the order of the elements in your names.

• GA – Event – Click – Add to Cart
• GA – Event – Scroll Depth
• Event – Blog Comment – GA
• UA – Mobile menu click

5. Add some context

Make sure people looking at your names would understand what they are, what are they doing and where are they doing it. For Google Analytics events, you might want to have “GA” and “Event” in your name, followed by what kind of event it is and on which pages this event is coming from.

I hope these you find these guidelines helpful and they get you a step closer to having proper naming conventions at your company, too.

As a bonus, I’ve added some examples that I think are pretty good. Divided by where they’re used.

Google Tag Manager Triggers

• PV – Homepage (PV stands for page view)
• Click – Outbound Link
• CE – VWO – Data Push (CE stands for custom event)

Google Tag Manager Variables

• VWO – Experiment ID
• EC – Transaction ID (EC stands for enhanced ecommerce)

Google Analytics Goals

• Click – Complete Purchase
• PV – Newsletter Signup Complete (PV stands for page view)
• Event – Submit Comment – Blog

A/B Testing Experiments

• RD – Homepage – Benefits Bar – Desktop (RD means this test was built by Reflective Data)
• Sitewide – Change Main Menu Order – Desktop & Mobile

Do you have any further ideas for improving analytics nomenclature? Share them in the comments below!

The post Good Nomenclature Is More Important Than You Probably Think appeared first on Reflective Data.

In this post, we are covering the common statistics terminology and models, and taking a closer look at the different methods of calculating A/B testing related metrics.

After going through everything, you will have a good understanding of what your testing tool does and how to calculate the metrics yourself.

Three main topics of this post:

• Frequentist statistics
• Bayesian statistics
• Revenue and Other Non-Binomial Metrics

Frequentist vs Bayesian

No, we are not going to debate on which one you should prefer. There is a bunch of articles and threads on the pros and cons of each. Long story short – there is no right and wrong, go with whatever your A/B testing tool of choice is using and get to know how it works.

In the broadest sense, A/B testing is detecting which variation of your website has the highest probability of performing the best based on set criteria (actually it’s a bit more complicated but more on that later in the post). We can determine four types of probability.

1. Long-term frequencies
2. Physical tendencies/propensities
3. Degrees of belief
4. Degrees of logical support

Frequentist inference is based on the first definition, Bayesian, on the other hand, is rooted in definitions 3 and 4.

Therefore, based the frequentist definition of probability, only repeatable random events (like the flipping of a coin) have probabilities. Furthermore, these probabilities are equal to the long-term frequency of occurrence of the events in question. It is important to understand that Frequentists don’t attach probabilities to hypotheses or to any fixed but unknown values in general. Ignoring this fact is what often leads to misinterpretations of frequentist analyses.

On the other hand, Bayesian approach views probabilities like a more general concept. Following the Bayesian technique, you can use probabilities to represent the uncertainty in any event or hypothesis. Hence, it’s perfectly acceptable to assign probabilities to non-repeatable events, like the result of your new product launch campaign. Therefore, many frequentists would say that such probabilities don’t actually make sense because the event is not repeatable. You can’t run your product launch campaign an infinite number of times.

Frequentist approach

Tools using frequentist-type statistics

• Optimizely (leveraging some Bayesian wisdom)
• Convert

You can export data from your testing tool or analytics platform and perform Frequentist tests, like Z test, yourself in tools like Excel or using programming languages like Python and R.

To interpret the results from a tool that is using Frequentist statistics, there are a few concepts that we need to understand.

Null hypothesis

A default position where conversion rate for control is equal to a conversion rate for a variation, there is no significant difference. Usually presented as H0.

Alternative hypothesis

An alternative hypothesis is a statement that is being tested against the null hypothesis is. Often presented as an H1 or Ha.

In A/B testing, an alternative hypothesis generally claims that a certain variation is performing significantly better than the control.

P-value

One of the most commonly misunderstood concepts of A/B testing and in statistics in general. Generally, people new to A/B testing tend to believe it describes the probability of a variation being a winner or a loser, compared to the control.

That, of course, is not true. P-value describes a probability of observing the observed (or greater) difference between the control and variation if you assume that your hypothesis is wrong and there should actually be no difference between the groups.

Another way to put this is that p-value describes the probability of seeing the observed difference randomly, say, in an A/A test.

P-value is one of the most important metrics in Frequentist statistics, therefore, it is not about detecting the probability of your null or alternative hypothesis of being true or false nor is it detecting the probability of your variation being better than the control. It is about rejecting or not rejecting the null hypothesis. Here’s how it goes:

1. You come up with some reasonable threshold for rejecting the null hypothesis. The notation used for this threshold is α (the Greek letter alpha). This threshold is a real number between 0 and 1 (usually very close to 0).
2. You promise to yourself in advance that you will reject the null hypothesis if the calculated p-value happens to be below α (and not reject it otherwise).

In A/B testing, the most common threshold α is 0.05, although sites with more traffic that want to minimize the risk of falsely rejecting their null hypothesis often pick 0.01. And sites with little traffic that are looking for quick learnings might go with 0.1.

It is a good practice to wisely choose a suitable threshold and stick to it until there’s a strong enough reason to change it. That way you can determine how many of your null hypothesis were falsely rejected.

I recommend using a statistical significance calculator to see how the chosen threshold affects your needed sample size.

Type I error

False positive – Rejecting a true null hypothesis.

In A/B testing this would mean that you call your variation a winner when it actually is worse, equal or less good than your test showed.

So, the p-value, along with the prespecified α, directly controls the type I (false positive) error rate.

Rejecting a null hypothesis (calling your variation a winner) is a result that triggers an action, most commonly ending in a change on your website (implementing your variation). Therefore, you must think wisely about what is the per cent of such false decisions that you can live with!

Type II error

False negative – Not rejecting a false null hypothesis.

In A/B testing it usually describes a situation where your variation is a winner but your test shows it is not significantly better than the control.

Statistical power

Simply put, statistical power is the probability that you will reject a null hypothesis if it is false.

The false negative rate will depend on these 3 factors:

• The size of the actual difference between the groups (which, by definition, is nonzero when the null hypothesis is false)
• The variance of the data with which you’re testing the null hypothesis
• The number of data points (your sample size)

In the real world, you only have control over the last factor, so you see why controlling the type II error is much trickier.

Source: An Intuitive Explanation Of P-Values

Confidence intervals

Confidence intervals are the frequentist way of doing parameter estimation. The technical details behind calculating and interpreting confidence intervals are beyond the scope of this post, but I’m going to give you the general overview.

Once you’ve calculated a confidence interval using 95% confidence level, it’s incorrect to say that it covers the true mean with a probability of 95% (this is a common misinterpretation). You can only say in advance that, in the long-run, 95% of the confidence intervals you’ve generated by following the same procedure will cover the true mean.

In A/B testing, let’s say you’ve calculated a confidence interval using a 95% confidence level for the conversion rate of a given variation [i.e 4.5% – 5.8%]. This means that after implementing this variation, its conversion rate will between 4.5% and 5.8% with 95% confidence.

Two-tailed test

If you are using a significance level of 0.05, a two-tailed test allots half of your alpha to testing the statistical significance in one direction and half of your alpha to testing statistical significance in the other direction. This means that .025 is in each tail of the distribution of your test statistic. When using a two-tailed test, regardless of the direction of the relationship you hypothesize, you are testing for the possibility of the relationship in both directions. A two-tailed test will test both if the mean is significantly greater than x and if the mean significantly less than x. The mean is considered significantly different from x if the test statistic is in the top 2.5% or bottom 2.5% of its probability distribution, resulting in a p-value less than 0.05.

One-tailed test

If you are using a significance level of .05, a one-tailed test allots all of your alpha to testing the statistical significance in the one direction of interest. This means that .05 is in one tail of the distribution of your test statistic. When using a one-tailed test, you are testing for the possibility of the relationship in one direction and completely disregarding the possibility of a relationship in the other direction. A one-tailed test will test either if the mean is significantly greater than x OR if the mean is significantly less than x, but not both. Then, depending on the chosen tail, the mean is significantly greater than or less than x if the test statistic is in the top 5% of its probability distribution or bottom 5% of its probability distribution, resulting in a p-value less than 0.05. The one-tailed test provides more power to detect an effect in one direction by not testing the effect in the other direction.

Here’s a good conclusion of Frequentist A/B testing, provided by Michael Frasco

In frequentist A/B testing, we use p-values to choose between two hypotheses: the null hypothesis — that there is no difference between variants A and B — and the alternative hypothesis — that variant B is different. A p-value measures the probability of observing a difference between the two variants at least as extreme as what we actually observed, given that there is no difference between the variants. Once the p-value achieves statistical significance or we’ve seen enough data, the experiment is over.

Source

Bayesian approach

Tools using Bayesian-type statistics

• Google Optimize
• VWO
• Adobe Target
• AB Tasty
• Dynamic Yield

Although the picture was a lot different just a few years ago, Bayesian has quickly overtaken Frequentist in terms number of A/B testing tools using it as the main logic behind their stats engines.

There’s a good amount of shorter and longer articles describing why Bayesian is a better choice for those running A/B tests, for example, “The Power of Bayesian A/B Testing“, they all seem to contain the following reasoning. And of course, Frequentists would argue on several of them.

• Bayesian gets reliable results faster (with a smaller sample)
• Bayesian results are easier to understand for people without the background in statistics (Frequentist results are often misinterpreted)
• Bayesian is better at detecting small changes (Frequentist favoring the null hypothesis).

Chris Stucchio has written a comprehensive overview of Bayesian A/B testing and the following section is mostly based on his white-paper.

Important variables of Bayesian testing:

α – underlying and unobserved true metric for variant A

β – underlying and unobserved true metric for variant B

Therefore, If we choose variant A when α is less than β, our loss is β – α. If α is greater than β, we lose nothing. Our loss is the amount by which our metric decreases when we choose that variant.

ε – the threshold of expected loss for one of the variants, under which we stop the experiment

This stopping condition considers both the likelihood that β — α is greater than zero and also the magnitude of this difference. Consequently, it has two very important properties:

1. It treats mistakes of different magnitudes differently. If we are uncertain about the values of α and β, there is a larger chance that we might make a big mistake. As a result, the expected loss would also be large.
2. Even when we are unsure which variant is larger, we can still stop the test as soon as we are certain that the difference between the variants is small. In this case, if we make a mistake (i.e., we choose β when β < α), we can be confident that the magnitude of that mistake is very small (e.g. β = 10% and α = 10.1%). As a result, we can be confident that our decision will not lead to a large decrease in our metric.

Source

Prior – one of the key differences between Frequentist and Bayesian is that the latter can take prior information into account. Hence, it doesn’t have to learn all the data points itself and can, therefore, reach the conclusions faster.

For example, let’s say we use a Beta(1, 1) distribution as the prior for a Bernoulli distribution. After observing 40 successes and 60 failures, our posterior distribution is a Beta(41, 61)⁶. However, if we had started with a Beta(8, 12) distribution as our prior, we would only need to observe 32 successes and 48 failures in order to obtain the same distribution as before.

Source

In general, it is suggested to choose priors that are a bit weaker than what the historical data suggest.

Most Bayesian-based A/B testing tools, like VWO, present their results using three key metrics

• Relative improvement VS control – a range by which the observed metric for the variation is better or worse than the same metric for the control. The range is calculated for a 99% probability. The more data the test collects, the smaller this range gets.
• Absolute potential loss – the potential loss is the lift you can lose out on if you deploy A as the winner when B is actually better.
• Chance to beat control/all – probability of the variation being better than the control/all other variations.

More on How VWO Calculates a Winning Variation

Just like with many Frequentist-based A/B testing tools, several Bayesan-based tools will let you choose some version of significance for the results your test is going to generate. In Frequentist, this is usually the p-value or confidence level, with Bayesian, you are likely to see some options to choose from. For example, this is what VWO gives you:

I think VWO’s approach is better for people without a background in statistics – it’s quite easy to mistakenly choose too weak confidence level without realizing the consequences.

Working with Revenue and Other Non-Binomial Metrics

How you (or the machine) calculate the results for an A/B test depends heavily on whether you are testing a binomial or non-binomial metric.

Here are some common non-binomial metrics used in A/B testing:

• Average order value
• Average revenue per user
• Average sessions per user
• Average session duration
• Average pages per session

The key difference between binomial and non-binomial metrics is that former can have only two possible values: conversion or no conversion, true or false etc. Non-binomial metrics, on the other hand, have a range of possible values, i.e from zero to infinity when measuring revenue.

This is a big difference, and without going into too much detail, it is quite clear that you cannot use the same algorithms for calculating the results for both. Mainly because of the lack of normal distribution for non-binomial metrics.

Further reading on whether or not you can use the same types of tests for both types of metrics:

• Testing Differences in Revenue? You’re Probably Not Using the Correct Statistics
• Your Average Revenue Per Customer is Meaningless

One of the most commonly used tests for testing non-binomial metrics is a Mann-Whitney-Wilcoxon rank-sum test. Unlike the t-test, it does not require the assumption of normal distributions.

BTW, if you have some experience with Python, setting one up for yourself is not too difficult. Here’s what you’ll need.

Another option is to use the following process suggested by Georgi Georgiev in his blog post.

• Extract user-level data (orders, revenue) or session-level data (session duration, pages per session) or order-level data (revenue, number of items) for the control and the variant
• Calculate the sample standard deviation of each
• Calculate the pooled standard error of the mean
• Use the SEM in any significance calculator / software that supports the specification of SEM in calculations

The key difference from binomial metrics is that no matter which method you choose, you will be working with user/session/order-level data, that is, you must feed the algorithm all the rows instead of totals.

Statistics plays a huge role in A/B testing and it is absolutely a must to know at least the basics of Frequentists, Bayesian and non-binomial metrics. That way you can choose the right tools and, hopefully, learn to know them (and stats they use) in depth.

I hope this post gave you a good starting point and hopefully you learned something new.

Did we miss something important? Suggestions are welcome in the comments below – so are the questions.

The post Comprehensive Guide to Statistics in A/B Testing appeared first on Reflective Data.

It has been a known problem for as long as front-end A/B testing solutions have been around. In this article, we take a look at why you should care and if it’s somehow possible to solve it.

In the example above, you can see the FOOC effect of a very simple A/B test that is changing the hero copy of the services page.

Why FOOC is a problem?

First of all, it is annoying for the visitors. No one likes when the look of the website is flickering too much while the page loads.

Second, and the most important reason is that it could easily skew the results of your A/B tests (and not in your favor).

Imagine this: You run a nice experiment on the checkout page of your website. Users actually like the change itself and would be willing to buy more but as there’s some flicker going on, they are not sure if that page is secure – flicker is often thought to be related to some suspicious activities such as stealing users credit card information.

And just like this, your test may loose just because of the flicker, even though the change itself would’ve worked!

By default, all of the most popular A/B testing tools have some sort of FOOC when they apply the changes on your website. Fortunately, there are some workarounds that could make the effect less noticeable. Let’s take a closer look at some of them!

How to get rid of FOOC?

If your A/B testing runs entirely on the client-side of the web, there will always we some downsides performance-wise. It could be the FOOC effect, slower load times, or both. Luckily, there are still some ways you can reduce the possible negative causes of A/B testing.

Install you A/B testing tool as recommended

Take a look at the guide for installing the A/B testing tool you are using. The most common recommendations are:

• Have the snippet as high in the <head> tag as possible
• Put it straight in the code, don’t use tag managers
• When using a tag manager, switch it from async to sync mode

Hide content before the changes have been applied

Many A/B testing tools have this feature built in. Some of them hide the entire page, some of them only the content. Contact your tool provider details.

When trying to achieve something similar on your own, here’s what to keep in mind:

• Hide the element as soon as possible (before the variation code runs)
• Have a fallback of ~2 seconds to show the content if something goes wrong
• Show the content immediately when the changes are done

Follow the general front-end development best practices

As Javascript based A/B testing is in many ways similar to front-end web development, we recommend knowing the at least the basics before starting your next experiment. If not, ask an expert to help you!

Some of the practices to get you started:

• When possible, use CSS over jQuery
• Cache your DOM elements in JS variables
• Prefer vanilla JS whenever possible
• Write your code in the order of DOM

Consider back-end A/B testing solutions

If you want to take the absolute maximum from your testing program, take a look at some of the server-side A/B testing tools.

These days, most of the popular A/B testing tools that are best known for their JS based solutions also offer a server-side version.

There’s also a popular open-source A/B testing tool called Wasabi, although setting it up requires a rather experienced developer.

Conclusion

FOOC and other loading-time-related issues will not leave Javascript-based A/B testing anytime soon. There are ways to lower the chance of letting it skew your results but to be 100% sure you should consider moving to one of the server-side A/B testing solutions.

Nevertheless, if you are able to reduce the flicker of your Javascript-based solution using the tips listed above, you shouldn’t be too worried. Just keep in mind that noticable FOOC might lower the performance of your variation and the results might be even better when the change is finally implemented.

The post What Is FOOC and How to Get Rid of It? appeared first on Reflective Data.

In this article I am going to share some of the best tips for Optimizely preview links that are going to make your work easier and save you a good amount of time.

While this article focuses on Optimizely, most of the tips can be transformed for other tools, too. For example, VWO’s links are quite similar.

Why bother?

You might think that they are just preview links, just see your preview and share it with others if needed.

True, you don’t need these tips to make your work possible, you want them because they can make it smoother and faster!

If you run a lot of A/B tests, you probably also work with a lot of preview URL-s. You open them on multiple devices and share with your team/clients.

Preview link components

Let’s take a closer look at Optimizely preview links.

Here’s what a normal preview link looks like:

https://www.optimizelypreview.com/http://reflectivedata.com/services/?optimizely_snippet=s3-197524926&optimizely_show_preview=true&optimizely_token=14010adhfte6ce482443b5845875ff0bdd8675b48f57c22b298567af8e18bc5d&optimizely_x8490154875=1

We can divide this link into 6 separate components:

• Optimizely prefix
• Your actual URL
• Optimizely snippet ID
• Show preview boolean
• Optimizely token
• Experiment ID and variation number

Now, let’s see what you need to know about each of these components.

Optimizely prefix

https://www.optimizelypreview.com/

This component is not always present, but it is needed for seeing a preview on pages that don’t have an Optimizely snippet installed yet.

It can happen that you have the snippet installed but Optimizely still adds it to your preview links, if that’s the case, just go ahead and remove it!

When the traffic is directed through https://www.optimizelypreview.com/ it can make some functions of your website broken. It is most likely the case with forms and some Javascript functions.

If your page has Optimizely snippet installed, it is best to remove this component from your Optimizely preview links.

Your actual URL

http://reflectivedata.com/services/

This component determines what page you will see in your preview.

What many users don’t know is that you can actually change it.

For example, you have an experiment that runs on more than one page and you’d like to see how it looks like on other pages. Just go ahead and change this part of the link and you are good to go (as long as this page has Optimizely snippet installed).

Optimizely snippet ID

?optimizely_snippet=s3-197524926

In most cases just ignore this component, there’s no need to change it.

One thing to check when having issues with preview mode is that the snippet with the same ID has to be present in your website’s source code!

In this case, the snippet has to be <script src="//optimizely.s3.amazonaws.com/js/197524926.js"></script>

Show preview boolean

&optimizely_show_preview=true

If this component is set to true (default) your pages will load with Optimizely preview overlay which covers a big part of your web page, sometimes even breaks it. Especially on mobile devices.

Here’s what a preview overlay looks like on mobile:

The best practice here is: if you don’t absolutely need the preview overlay, disable it like this: &optimizely_show_preview=false

Optimizely token

&optimizely_token=14010adhfte6ce482443b5845875ff0bdd8675b48f57c22b298567af8e18bc5d

Nothing interesting to know about this component. Just don’t change it.

Experiment ID and variation number

&optimizely_x8490154875=1

This component of the Optimzile preview links determines which experiment and variation you are going to see.

This is extra useful if your experiment has more than one variation, just change the last number to see a specific variation. If you wish, you can also change the experiment ID.

Conclusion

Try using these tips in your everyday work and, believe me, you are going to save some time very soon.

When sending a link to a colleague or client just use&optimizely_x8490154875=1as they (usually) don’t want to see the preview overlay. Same goes for testing on mobile devices.

If your experiment has more than variation just change the last number in &optimizely_x8490154875=1 and quickly switch between variations.

In case your preview looks broken or doesn’t work at all go see if the snippet ID matches the one in the link or try removing the https://www.optimizelypreview.com/ component.

If you have any more useful tips related to Optimizely preview links, let us know in the comments so we could update this post.

The post Must Know Tips For Optimizely Preview Links appeared first on Reflective Data.

In this article, we are going to give you a comprehensive overview of how to get started with A/B testing, including the prerequisites, tools, and methodologies. We are even going to give you some test ideas to get you started! In case you are already running A/B tests, I’d still suggest you take a look how others (we) approach the problems involved.

How A/B testing works

A/B testing or split testing is a method of comparing two versions of a webpage against each other to determine which one performs better. Yes, it’s that simple. Or is it?

AB testing is essentially an experiment where two (or more) variants of a page are shown to users at random, and statistical analysis is used to determine which variation performs better for a specific conversion goal. In e-commerce, the most common goals are the number of purchases and of course, the revenue. Although, you could also measure the micro-conversions or whatever KPI is relevant to your business.

The main goal is to take guessing out of your website optimization process, and eventually, from your business decisions. It is much wiser to take actions based on real evidence, statistically significant test results in this case.

To put it very simply, the process of A/B testing is as follows:

Analyzing information you have

First, you have to know the website you are working with. The main KPI-s, the funnel, the audience etc.

Next step is taking a look at the analytics in order to spot any trends, issues or other useful insights that you could use for coming with the ideas (hypotheses) for the experiments.

If you don’t have enough data or you have squeezed everything out of it, get more data. Take a look at tools like Reflective Data Platform that will give access to features like Form Analytics, Heatmaps and On-Site Polls.

Coming up with the hypotheses

In this step, the main goal is to figure out what should be tested. We do that based on the information gathered in the previous step.

The more hypothesis the better, but keep in mind that they should be backed up by real data.

A hypothesis should be formulated similarly to this example: People don’t click the main call to action (CTA) because it’s too low on the page. Moving it above the fold would raise the CTR.

Putting hypotheses into practice

You have a hypothesis, now it’s time to bring it to life. The first step here is usually to draw a wireframe of the solution. In our example, it would show where the CTA would be in the variation.

If it’s a more complicated test, a design file might be needed.

When you have figured out exactly how your experiment is going to look like, it is time to build the actual test, using an A/B testing tool such as Google Optimize. Once the setup is done, it is important to make sure it looks as it should on every device and browser that is being targeted. For that, most tools have a preview mode.

If all the bugs are fixed, it’s time to launch the test to the public!

Analyze & document the results, then repeat the process

Here is a nice visualization of a single A/B test and how it directly affects the revenue.

Once the test has reached the statistical significance, it can be stopped and the process of making conclusions may begin.

It is also important to document the hypotheses, experiments, and their results. That gives us an opportunity to evaluate the entire testing program and make it more effective over time.

If the test is a winner, meaning that the main KPI is up in the variation, I would suggest considering implementing the change.

If it was a loser, try to find out why your hypothesis didn’t work, maybe find a better form of execution and run it again.

In every experiment, there should be a learning point and every upcoming test a bit better than the previous ones.

Are you and your website ready for A/B testing?

While I believe that A/B testing is the best way of turning analytics data into real-life decisions, unfortunately not all websites are in the phase where they could actually run meaningful experiments.

Website requirements

In A/B testing, a winner is determined by statistical models. Some of the most popular ones in use today are Bayesian and Frequentist, while different in many ways, they both have something in common – the sample size.

Sample size means the number of visitors that are part of your experiment, in an A/B test that sample would be equally split between original and the variation. You could lower the sample size by running experiments that have a bigger effect (but how could you be sure) or getting satisfied with lower confidence rates (not a good idea).

Take a look at this calculator to make sure if your website can conclude an A/B test in a reasonable timeframe (less than 8 weeks).

There is one more calculator available by Optimizely.

Skillset requirements

While getting started is not too difficult, there is a good set of skills you need to achieve before you become a really good tester that will bring a real value to whichever business they are working with.

If your website has enough traffic (see the calculators above), take a look at the skills you should consider learning. And if your website is not eligible yet, work on both – getting the traffic and learn the skills, so you would be ready once the site is.

PS! These skills could be all in one person, in a team or even outsourced.

Analytical skills

In order to drive actionable insights from your data, you would have to know the tool that you are going to use. In most cases that tool will be Google Analytics.

Being able to pull out all sorts of reports is not enough, you need to know what it all means. Try looking for trends, segments that have extremely high or low performance numbers, and figure out what are the points in your funnel that seem to cause the most friction so that many visitors drop there.

Besides building custom reports, you will also need customized dashboards, ones that are the most relevant to your business.

Here’s a screenshot of a general performance dashboard for one of our test pages.

Design skills

Quite often aA/B test is much more than just a simple copy, image or color change. To pull these changes off, there has to be a proper design.

Many teams already have a designer, if yours does, ask him/her to help you. If you don’t have a designer to help you, you are going to have to outsource or try your best.

Coding skills

To set an experiment live there is almost always some coding involved. Yes, most tools come with a visual editor feature, but that is only good for most basic changes.

Coding skills required for setting up an A/B test are pretty similar to those needed from a front-end web developer. Keywords are Javascript, jQuery, HTML and CSS.

If you can’t code and don’t know someone who could, play around with the visual editor and start learning now! Or hire someone that can.

Presenting and documenting skills

If you work in a company then most probably you will have to report your experiments and results to your team. Good presenting skills come in extra handy here. You are going to have some winning tests, some neutral tests and probably a good amount of losing tests. You have to communicate that there’s value in each and every one of those tests.

Documenting is a very important part of every A/B testing program. Even if you run a single test every month it will be extremely hard to remember all the important details a year after. There are good tools created specifically for that purpose but for beginners, an Excel document is just fine.

How to the right tool?

The market for A/B testing tools has become much wider in the past years. That means that everyone has to work hard to make theirs the best one but it also means that for a beginner it can be rather difficult to make a right pick.

And picking the right tool is important because just like with most software, switching them later is a pain in the ass.

Google Optimize (free version)

It is free. That means that if you’re new to A/B testing and you are just figuring out if it fits your business needs etc. then there is almost no risk involved.

The number one problem with Google Optimize is that the free version is limited to three simultaneous experiments. That means you can’t run more than three tests at a time, of course, more can be in draft or ended.

Here’s a full guide that will help you get started with Google Optimize.

And this is what it feels like to use Google Optimize for three moths.

Click here to sign up and create your first experiment today, for free!

Optimizely

Optimizely is a great tool, it has been in the business since 2010. It was founded by Dan Siroker, a former Director of Analytics for the Obama campaign.

Optimizely has been my personal favorite tool for years, their approach is right in so many ways. It is logical, easy to use and the most developer-friendly tool out there.

Here’s a list of things I like about Optimizely

• Easy to install, add a snippet and voilà.
• Developer friendly, everyone who is using code instead visual editor will like it.

Some of the issues I’ve seen people have are:

• It is expensive, for a smaller company I’d suggest starting with something free and once you are sure that A/B testing is for you, move on to more advanced tools.
• Somewhat slow support, I believe it’s faster for enterprise clients but for medium-sized companies it can be frustrating.

Learn more on their Website.

VWO

Visual Website Optimizer, another very popular A/B testing platform that is being used by thousands of companies across the world.

In many ways, VWO is similar to Optimizely but it also has some user behavior analysis features built-in – heatmaps, surveys and website reviews.

I used to like Optimizely a lot more but in a past year I find the two being almost equal when it comes to general A/B testing related features.

Things I like about VWO

• Nice interface, it is very pleasant to use. Everything is where you’d expect it to be.
• Very good support, their support team is top notch!

Reasons I would consider not getting VWO as my main testing tool.

• Expensive, for starters I’d suggest a free tool such as Google Optimize.

Take a look at their website.

Wasabi (open source)

Wasabi is an open-source A/B testing platform for large companies and seasoned developers.

There are two cases where I’d suggest Wasabi:

• Full control, if your company can’t afford to have their data in someone else’s servers.
• If you love coding, if you’re a developer that likes challenges, try getting Wasabi up and running

Read more and get started on their Github.

DIY

The logic of basic A/B testing is not a rocket science. If your developer has free time or if you want a tool that is tailored to your business, build your own tool!

You could build it on the front-end or on the server side, all of the results should go directly to the analytics platform of your choice. A good combo would be Google Tag Manager and Google Analytics.

Take a look at GTM Testing and take it from there.

Sentient Ascend

If regular A/B testing is not enough, take a look at Sentient Ascend. It leverages the power of AI and evolutionary algorithms to detect a winner out of thousands of combinations.

If this sounds interesting, take a look at our comprehensive overview.

Coming up with new test ideas

First of all, test ideas should be backed up by real data. One of the best sources for insights is your analytics platform, Google Analytics for most websites.

Analytics

If you haven’t looked at your data lately, there is a good chance that you will find something that is broken. No need to A/B test that, just fix it ASAP. You will probably also find a good amount of things that are so-so, these you should test!

Example: You see that a lot of people drop in the last step of your checkout, the part where they have to insert their credit card info. Now, you have to think of the reasons why that could be. Maybe that area is not trustworthy enough? Test adding trust badges next to CC fields!

Real users

Take a look at your website, try acting like a real visitor, go through the funnel and try to detect what could be improved. Even better, ask someone not related to your business to do so.

Example: You ask your friend to go visit your e-commerce website and buy a specific product. Stand behind them and see what kind of issues they have. Maybe they can’t find the right navigation item, or perhaps the search box is not prominent enough? Try making it more visible!

Others

Other tools that will provide you with the useful insights are:

• Form analytics tools
• Heatmaps (click, move, scroll)
• On-Site Polls
• Session recordings

Example: When taking a look at your heatmaps report you notice that a lot of visitors click on a picture on your website. The problem is, that picture is not meant for clicking and nothing happens. Try giving it a relevant click action, a zoom maybe?

Heatmaps, form analytics and on-site polls are included in Reflective Data Platform.

Conclusion

A/B testing is becoming more and more popular every day. Still, there are a lot of websites that aren’t running a single experiment, that is a lost opportunity.

If you have a website with a decent amount of traffic, start testing! There is nothing to lose, start with a free tool such as Google Optimize.

In case you have a website with traffic which is too low for testing, get yourself familiar with the basics while working on getting more traffic. It is worth it, you will be making tons more money!

When picking a tool, take a look at each of their websites, ask for a demo, find someone who is using them and get a real-life overview. If you are from a smaller business, go with a free option.

The post Getting Started With A/B Testing appeared first on Reflective Data.

So, what’s our verdict? Is it going to replace other tools like Optimizely or VWO? In this article, we are going to tell you exactly what we think about Google Optimize.

Seamless integration with other Google products

If you (like we) are using Google Analytics as your main analytics platform and Google Tag Manager as your main tag manager then it’s logical step to start using Google Optimize as your main testing platform?

Well, the integration is truly simple. The installation with Google Tag Manager is easy, here’s how it’s done. Next, just link Optimize with your Analytics property and view and that’s it.

All your experiments will automatically show up in Google Analytics, take a look at Behavior -> Experiments.

For more advanced analysis we recommend creating segments or filters for each experiment and variant. That way you will be able to really see how your experiments are changing the way your website’s visitors behave.

Here’s how to create a new segment for a specific experiment and variant.

So, if it works so well with the tools that you are already using, doesn’t it make sense to just start using it as your main A/B testing platform? Okay, let’s take closer a look at Google Optimize’s pluses and minuses.

The bad

I am going to start with some of the aspects that I don’t like so much about Google Optimize.

Just a little notice that besides Google Optimize, I am also actively using Optimizely and VWO, plus I have been experimenting with an open-source testing tool called Wasabi and even built an A/B testing solution based on Google Tag Manager.

Three experiment limit

Today, Google Optimize allows optimizers to run only three experiments at the same time. You can have more as a draft or ended but not running.

That can be a real issue for companies who are really into testing, especially for those who like to run separate tests for mobile and desktop.

For those just starting off with their A/B testing game, three experiments at once could be just enough!

Limited code length

That’s right, with Google Optimize, your scripts are limited to 10240 characters.

This might not be an issue if you are just using the visual editor for some copy changes but if you are building a more advanced experiment this can become really annoying. Quite often I end up splitting my code into three or four sections.

No image upload functionality

There is no way to upload images into Optimize’s servers. You will have to upload them to your own server or use third-party services.

While this might not be a big issue for some, I found it really bothering. All of the other testing tools that I use have this feature.

Preview mode is a pain in the…

As seasoned A/B testers know, your experiment just doesn’t look right in the editor. To see the real picture, you have to use preview mode or URL-s.

When I work with VWO or Optimizely, I always have the editor open in one window and preview in another one. As I make changes in the editor I simply press “Save” and almost immediately preview my changes in the preview mode.

When it comes to Google Optimize, the preview doesn’t get updated unless you click on “Save” and “Done”, the latter exits the editor! After doing so you will need to re-activate the preview mode and then load the editor again to make some changes. And that’s how it goes, over and over.

This is why setting up a more advanced experiment takes so much longer in Google Optimize when compared to some other testing tools.

The good

If the “bads” didn’t scare you away, let’s take a look at some of the reasons why I really like Google Optimize.

Works perfectly with other Google tools

As I already mentioned, if you are already using Google Analytics and Google Tag Manager, installing and using Google Optimize goes super smoothly.

While you can integrate Google Analytics with other testing tools, there will almost always be some issues with that. Most common is that the numbers just don’t match up. Google Optimize’s data is based on Google Analytics, so that won’t be an issue.

Very nice visual editor

Whether you are a beginner in A/B testing, don’t write any code or just most of the time run simpler A/B tests, you will really like the visual editor that comes with Optimize.

It is super easy to use, a simple copy or an image change will take a second. Really, it’s that simple.

Growing community

As with other free tools, especially those from Google, the community tends to be huge and rather active. That means you can expect to see all sorts of tutorials and other cool stuff related to Google Optimize, online & for free.

While paid tools might (sometimes) have a good support system, I almost always prefer an active community on StackOverflow where nearly all of the issues have already been solved.

Right now, the community seems to be rather small but I am sure it will grow pretty fast.

Does everything it’s paid competitors do

Doesn’t matter if you come from Optimizely, VWO or some other similar testing tool, you will find all the features you loved about them in Google Optimize. Maybe in a bit different format, but most of them are there.

Of course, there are some slight differences, like missing image upload option etc. but for most of them, you can easily find a workaround.

IT’S FREE

Best things in life are free, so is Optimize.

Just sign up with your Google account, add the snippet on your website, link your Analytics property and start testing. It’s that simple.

Other testing tools are very expensive, especially if you have a lot of traffic and your optimization process is not very effective (yet).

Conclusion

While it has its minuses, Google Optimize is almost as good as any other A/B testing tool and you will definitely like using it.

I really hope that some of the issues I listed in this article are going to be fixed in the nearest future. It’s a rather new tool after all.

Three test limit will most probably stay to keep you considering upgrading to Optimize 360 but if testing is just something you do besides other stuff, three tests can easily be enough for you.

Are you new to A/B testing?

Go ahead, install Google Optimize on your website today and get to know what optimizing your website really means.

Are you already running a few tests every month?

No matter what tool you are using, give Optimize a try. It’s free, you don’t have anything to lose.

Are you an hard-core A/B tester?

Well, if you know for sure that three test limit is just not enough for you, you should probably wait before you cancel the tool you are currently happy with. But still, consider giving Optimize a try. Maybe you have another website where you run fewer tests or maybe you will like the platform so much that will consider the paid version.

Whatever your background is, my recommendation is to try new things (and tools). That’s the whole point of testing and optimization, right?

Click here to get started with Google Optimize.

Have you already tried Google Optimize? Let us know what you think about it in the comments below.

If you liked the article, please subscribe to our newsletter to get posted on what’s new in digital analytics!

The post Using Google Optimize For 3 Months appeared first on Reflective Data.