The main limitations of Google Analytics are related to sampling and data collection limits. Most affected are companies that can’t afford the premium 360 version of Google Analytics (~150k/year) but still have a good amount of traffic visiting their websites. In general, Google Analytics properties with >1M sessions/month or >10M hits/month are being affected by some heavy sampling and data collection limits.

In this article, we’re going to cover the different types of limitations present in the free version of Google Analytics and provide solutions/workarounds to all of them. Oh, and the solution, in most cases, does not include buying the 360 version.

Types of limitations in Google Analytics

Here’s a quick overview of the different types of limitations that come with the free version of Google Analytics.

Sampling in reports

Sampling on the reporting level means that, even if Google Analytics collected 100% of the hits, the reports you are seeing in the user interface are not based on 100% of those hits/sessions. In general, sampling starts if the period you are looking at contains more than 500k sessions in total. It can start a lot sooner, though, when more advanced custom segments are in use.

Data collection limits

From the official documentation.

If a property sends more hits per month to Analytics than allowed by the Analytics Terms of Service, there is no assurance that the excess hits will be processed. If the property’s hit volume exceeds this limit, a warning may be displayed in the user interface and you may be prevented from accessing reports.

Data collection limits that apply to all free Google Analytics accounts.

• up to 10M hits total per month
• 200,000 hits per user per day
• 500 hits per session

Data processing latency

Processing latency in Google Analytics is up to 24-48 hours. Standard accounts that send more than 200,000 sessions per day to Analytics will result in the reports being refreshed only once a day. This can delay updates to reports and metrics for up to two days.

This makes Google Analytics quite useless for things like monitoring real-time performance, detecting usability issues, automated reports on an hourly basis, real-time product recommendations, and other use cases that require fresh data (i.e. machine learning, marketing automation).

Custom dimensions and metrics

There are 20 indices available for different custom dimensions and 20 indices for custom metrics in each free Google Analytics property. 360 accounts have 200 indices available for custom dimensions and 200 for custom metrics.

While 20 is plenty for a small business with a simple website, this number can become quite limiting if you need custom dimensions for things like A/B testing data (one for each live experiment), e-commerce variables, user-specific information, and more.

Aggregated metrics and no access to raw data

Unfortunately, in the free version of Google Analytics, there is no way to access the raw hit-level data. This means that if the way it calculates certain metrics or aggregates hits into sessions doesn’t suit your business requirements then you’re out of luck – without access to raw data, you can’t define your own calculations or aggregations.

Also, in case you would like to analyze the user journey of a specific visitor, there is no way to easily query and analyze all hits from one visitor in a timely order.

Google Analytics 360 does have a BigQuery export but, by default, the metrics and sessions are still defined and pre-aggregated just like in the reports.

Hit payload limited to 8000 bytes

Depending on your analytics setup, some of your hits may include a lot data – ecommerce product impressions, several custom dimensions etc. The Google Analytics hit payload size is limited to 8000 bytes, this means that all hits greater than this will be ignored by Google Analytics’ data processing engine and your dataset ends up being incomplete. What’s worse, Google Analytics doesn’t alert you when this is happening!

This affects both the free and 360 versions of Google Analytics.

Workarounds to sampling and other limitations

Now, let’s take a look at the same list of limitations and provide a solution/workaround to each of them.

Sampling in reports

Option 1 – Divide your queries into smaller chunks and join data using another tool (Excel, Pandas, R). This can be done manually in the UI, using the Reporting API, or by using a library for your favorite programming language (i.e. gago for Go) that has an “anti sampler” feature built-in.

Option 2 – Collect raw hit-level Google Analytics data in your data warehouse (i.e. BigQuery). This process is known as parallel tracking and functions by duplicating all hits that are sent to your Google Analytics property into your data warehouse. This allows you to write ad-hoc queries in standard SQL without any sampling.

Data collection limits

Option 1 – To get past the 10M hits/mo, 200,000 hits per user per day and 500 hits per session limits, the same Google Analytics parallel tracking solution can be used. This means that even if Google Analytics starts to skip some hits, they’re still available in your data warehouse and ready to be queried.

Option 2 – Set up multiple Google Analytics properties. This can be a little tricky but it is possible to divide your Google Analytics integration into multiple properties. Depending on how much traffic your site gets, you might divide all hits from the first week of the month into one property, hits from the second week into another property etc. Alternatively, you could divide hits based on the Client ID into as many groups/properties as needed. Then, you can use the Reporting API to pull data from all properties and join them using Excel, Pandas or some other tool.

Data processing latency

If you need to monitor the site’s performance in real-time or need data for product recommendations or other machine-learning efforts, the 24-48 hour delay is probably way too long. Your only option to get access to near-real-time Google Analytics data is to use the parallel tracking technology. With parallel tracking, processed hit data will be available in ~5 seconds after it was collected from your site. This allows you to build alert systems, monitor performance in real-time or feed your machine learning algorithms with fresh data at all times.

Custom dimensions and metrics

As mentioned earlier, Google Analytics free version limits you to 20 indices of custom dimensions and metrics. If you hit the limit and need to add more, your first action should be to check whether there’s a dimension or metric that you no longer need. If that’s not the case, your only solution is to either upgrade to Google Analytics 360 (to get 200 metrics and dimensions) or to use parallel tracking and get access to unlimited custom metrics and dimensions. When using parallel tracking, dimensions and metrics with indices 20+ will not show up in Google Analytics reports but are available in your data warehouse.

Aggregated metrics and no access to raw data

If you’re not happy with how Google Analytics aggregates data into sessions or calculates certain metrics, you need access to the raw hit-level data that allows you to define your own rules for almost any kind of aggregation or calculation. There are two options you can sue to get access to raw hit-level Google Analytics data.

Option 1 – You guessed it! Parallel tracking sends the raw hit-level Google Analytics data into your data warehouse (with no sampling) and you can use standard SQL to query your data and tools like Excel, R or Pandas to make all sorts of calculations and aggregations. BigQuery has a native connection with most BI and data visualisation tools so reporting is easy and flexible.

Option 2 – If your Google Analytics property is not affected by the data collection limits (it gets <10M hits/mo), you could configure your custom dimensions to collect data like Client ID, Session ID, Hit Timestamp and Hit Type. With all these dimensions available in your dataset, you can use the Reporting API to pull the raw hit-level data into your local machine or data warehouse. If your property receives more than 10M hits a month, this option is only available if you split your data between multiple properties.

Hit payload limited to 8000 bytes

If your enhanced ecommerce transactions include many products or you have lots of custom dimensions or for some other reason some of your hits exceed the 8000-byte limit then you are losing data. Google Analytics will, without any warnings, simply skip those hits. You have three options for solving this issue.

Option 1 – Send less data by removing some data points. Simo Ahava has written a great blog post on automatically reducing the Google Analytics payload length by removing some of the unnecessary data points and creating an order for removing other data points until the required payload size is reached.

Option 2 – Use the data import feature in Google Analytics to import extra information after the data has been sent to Google Analytics. This way you can send only the most important information (Client ID, transaction ID etc.) with the main hit and import other details (product information, custom dimensions etc.) later using the import feature. Check out this post by Dan Wilkerson for a detailed guide for doing this with Google Tag Manager.

Option 3 – With parallel tracking your hits are limited to 16 000 bytes (double from Google Analytics) and this can be raised even more on request.

Key takeaways

Now that you have a pretty good overview of the limitations that come with the free version of Google Analytics and some that come with the 360 (premium) version, it’s a good time to assess your own Google Analytics implementation(s). Does any of the limitations affect your data and its accuracy? May some of the limitations become an issue in the future?

Knowing the limitations of a tool is extremely important. You may be investing a lot of time and money into a setup that may not be sufficient anymore as your business grows.

With all its limitations, Google Analytics is still an extremely powerful tool that has an awesome community. If sampling or some of the other limitations are an issue for you, I believe you can fix them with some of the solutions suggested in this blog post.

Most of the sampling and data collection limitations of Google Analytics are solvable with parallel tracking, a system that sends raw hit-level Google Analytics data into BigQuery (or any other data warehouse) in real-time.

***

Should you have any other issues with Google Analytics or have an alternative solution to any of the problems mentioned in this post, please share your thoughts in the comments below.

The post How to Avoid Google Analytics Sampling and Data Limits? appeared first on Reflective Data.

Cloud databases (i.e. BigQuery) and various data management tools have made it possible for marketers to build data pipelines without a big investment in hardware, software, and custom development.

In this article, we will walk through the steps of gathering data from Google Analytics, Google Ads, Google Search Console, Facebook, CRM and several other sources into Google BigQuery data warehouse and making it the single source of truth for all marketing data.

Takeaways

• Learn why most businesses need a marketing data warehouse
• Learn how to plan the best marketing data warehouse for your business
• Learn the basics of working with marketing data stored in BigQuery

Why (almost) every business needs a marketing data warehouse?

In the past few years, the number of marketing tools an average marketing person uses has grown rapidly. Doesn’t matter whether it’s a SaaS, lead gen or an e-commerce business, besides one or two analytics platforms there usually are a few ads platforms in play, a CRM, CMS, several social media platforms, an email automation system and probably a few more tools and platforms.

All of those tools and platforms are there to make our work as marketers, business owners or data analysts easier and more effective. In reality, though, you will end up having a bunch of silos – systems that don’t really communicate with each other and rarely agree on any of the important KPIs or metrics.

Data silos are the source of confusion and disagreement between teams within a business, leading to a situation where, at the end of the day, no-one knows which tool or numbers they can trust.

The solution? A marketing data warehouse.

Below are a few reasons that should explain quite vividly that your business needs a marketing data warehouse.

– Single source of truth: the main benefit you get from having all of your marketing data in one central data warehouse is that every team in your business will look at the same metrics, calculated the same way. This benefit alone will save you and your colleagues from lots of misunderstanding and possible conflicts between teams and functions.

– Access to raw data: most marketing tools show you aggregated data and calculated metrics. While it’s okay for a more basic user, it is definitely limiting your ability to do your own calculations. It is also the main reason why data from different tools never match – they simply measure and calculate the core metrics like users, sessions, conversion rate, revenue etc. in a different way. When working with raw data, you define your own aggregation and calculation rules – this should be agreed on company level and leads to everyone looking at the same numbers.

– Visualization and BI tools: sadly, not all marketing tools have a connector with most major data visualization and BI tools (Tableau, Looker, Google Data Studio etc.). Working with CSV or Excel files isn’t automatic or very scalable. Data warehouses like BigQuery have a native integration with nearly all better-known BI and data visualization tools.

– More accurate attribution: tools like Google/Adobe Analytics are great when you’re just getting started with marketing channel attribution. What most tools lack, though, are access to all user touch-points (online and offline) and the ability to use custom attribution models (FBA, ML-based, Markov etc.).

– Machine learning: with tools like BigQuery ML, Auto ML and others being widely available, getting started with machine learning has never been easier or more affordable. The only part that’s missing, though, is access to high-quality raw hit-level data that is required for training your models. With all your marketing data in your data warehouse, you can create models from product recommendations to smarter remarketing campaigns.

– Manual data joining doesn’t scale: exporting data from various sources and joining them in Excel/Sheets can give you interesting insights you wouldn’t have seen in any single tool but let’s be honest, this system isn’t very scalable. Working with a data warehouse, you can pre-join commonly combined data into views or pull everything together in a dashboard.

– A complete overview of the user journey: no one tool knows everything about your users thus looking at them doesn’t show you the complete user journey. Combining data from all tools and sources, both online and offline gives you a much more detailed picture of the entire user journey. Furthermore, this gives you much more accurate long term metrics like users’ lifetime value.

***

I could go on for long writing about the benefits that explain having a proper marketing data warehouse. What about you? Let us know of the reasons your business needs (or already has) a marketing data warehouse in the comments below.

How to plan and get the best marketing data warehouse for your business?

Having a marketing data warehouse is becoming the new norm. Setting one up, though, is not a trivial task.

There are some tools that offer sort of a plug-and-play solution but just like every business is different, so are their requirements for a marketing data warehouse. There is no good solution that would meet the requirements of all (or even most) businesses.

Mapping data sources

Before dealing with anything else, start by mapping all the data sources your business has. This may include, but isn’t limited to, Google Analytics, CRM, CMS, offline data, Ads platforms, email automation tools, SMS and push notification platforms etc.

After knowing your sources, try to group all of them into a document and sort them in a logical order. For example, (1) analytics platforms, (2) social media platforms, (3) email etc.

Next, think more deeply about each of the listed data sources. Take a look at their documentation and data structure. In your document, take note of what kind of data and in what format you’d need in your marketing data warehouse. Start by thinking of how your team is using this data today, what kind of dashboards you’ve built etc.

Furthermore, take note of how fresh each dataset should be in the data warehouse. Do you need it in real-time, hourly, or maybe a daily update is fine? Probably you’d want your click-stream data to be as fresh as possible while having your ad cost data update once a day may be fine.

As an extra, I recommend you check if each data source has a public API or if they allow automated data export. If that’s not the case, it’s highly likely you can’t easily export data from that tool and I’d look for an alternative for that tool. Every decent marketing tool should have a public API for exporting data these days.

Mapping use cases

Based on the list of data sources you just listed, it is now time to think about all the great things you could do with a marketing data warehouse that includes all the data.

Once again, I recommend you start a document and structure it logically based on some category. For example, (1) user behavior analysis, (2) attribution, (3) alerts etc.

Of course, you don’t have to think about everything beforehand but it helps you later on deciding the right schema, load frequency (batch load vs stream), pre-calculated views, metrics etc.

Start by thinking of all the ways you are currently using the data available within each marketing tool your company has. Next, think of ways you’ve joined the data in the past using tools like Excel or Google Sheets. Then, I recommend, you go completely creative. Think of things you were never able to get from your data but would really like to. For example, analyzing user journey across all touchpoints, online and offline.

Build or buy?

The truth is, that building a data warehouse has never been easier nor cheaper. Every cloud platform has its own offering and several solutions have been built on top of major cloud platforms by other companies.

To be honest, setting up a data warehouse is actually the easy part. What makes it difficult though, is getting data in the data warehouse – automatically, reliably, fast and free of duplicate records. This part is known as a data pipeline and this is the system that feeds your data warehouse and, ultimately, your business with trustworthy data.

Now, where do you get a solid data pipeline? You have three options.

1. Build: if your company has enough resources (money, time, developers, data scientists, data engineers) then building everything in-house might be a good idea. There are some useful building blocks available in most cloud platforms (i.e. Cloud Dataflow, Pub/Sub, Cloud Functions in GCP). What I would recommend, though, is to make sure you have at least one person in the team that has led a data pipeline build before or at least someone you could consult with. Otherwise, there are things that will probably go wrong in your first attempt.

2. Ready-made solutions: depending on the level of customization you require, some of the ready-made data pipeline solutions might fit most of your needs. Some tools to check out are Stitch, Fivetran, Funnel and Segment. All of them do a great job at what they’re built for – offering fixed or semi-flexible data pipeline to feed your data warehouse with data. Unfortunately, though, each of them has their shortcomings. For example, none of them is able to send raw hit-level Google Analytics data into your data warehouse and process it into sessions based on your rules. Oh, and setting such a tool up can still be a considerable amount of work.

3. Data Pipeline as a Service: to gain full control of your marketing data, you need a system that’s tailored to your business’ needs. A way to get there is to hire a company that has all the necessary infrastructure (connectors, integrations, batch loaders, monitoring systems etc.) and is willing to take a personal approach with every client. This means getting to know your business, interviewing your team and building a solid data pipeline using the building blocks they already have – and code everything they don’t.

***

Start by mapping your data sources and use cases as described above. Then see if any of the ready-made solutions could fit your needs. If you need a more customized/advanced solution, think about your internal resources. If you need help with deciding which way to go or want to learn more about data pipeline as a service, talk to experts at RD.

The basics of working with marketing data stored in BigQuery

As BigQuery is one of the most popular data warehousing solutions for marketing data, let’s use it in our examples here as well.

Having all of your marketing data available in BigQuery is great but quite useless if don’t use it or if you only use it for things that are easily doable within Google Analytics or some other tool. E.g. for checking how many sessions your website received, the bounce rate etc.

You have to use SQL

Coming from a tool that lets you build reports by dragging and dropping various metrics, dimensions and filters in the editor, getting started with BigQuery can look like a bit of a learning curve.

When going to BigQuery, this is literally all your welcomed with.

No fancy data explorer features or a drag and drop report builder. Just a blank white text input area for your SQL queries.

Yes, there are tools like Looker and Google Data Studio that connect with BigQuery and automatically understand your schema. Using these tools with little or no SQL skills, though, you’re likely better off using the reporting features in your data sources (Google Analytics, CRM, Ads) instead.

The power of BigQuery becomes apparent only through (quite advanced) SQL queries. This is what allows you to join various sources, calculate your own metrics, define your own attribution models and so much more.

As a marketer, you really have two options: (1) learn SQL (here’s a super good getting started guide from Mode) or (2) work closely with a developer/data scientist/analyst that knows SQL really well.

No matter which route you take, know that SQL is no rocket science but there is no way around it when getting into more advanced data analysis.

Now, let’s take a look at a few use cases for a marketing data pipeline a.k.a your single source of truth.

Analyzing the journey of a single user

In this example, we are using raw hit-level Google Analytics data sent to BigQuery using RD Marketing Data Pipeline.

What’s good about BigQuery is that, unlike Google Analytics, you can use it to work with personally identifiable information (PII) – for example, our dataset includes users’ email addresses. This makes it easy to connect incoming leads with their activity on the site.

Let’s start by looking at how many sessions our lead has had on our website in the selected time frame. For this example, we’re connecting all sessions using email address, IP address and the client ID.

Seems like this user has visited the site a whopping 151 times (the number of sessions)!

Next, you might be interested in where all those sessions initiated. You could do so with a query like this.

While this gives you decent overview of how they found your site, you should also look into the landing pages (especially for organic, to understand what they’re looking for) and all the pages they’ve visited prior to submitting the contact form.

At Reflective Data, we do a similar analysis with all leads before we contact them – just to understand how much they already know and what they might be looking for.

Custom attribution models

Most tools out there default to one specific attribution model. For example, Google Analytics uses the last non-direct click model with a 6-month look-back window. Some tools let you modify the look-back window and/or compare different models but usually, the options are quite limited.

An attribution model is a really important part of marketing analysis. Different models can yield completely different results. Picking or building an attribution model should be a well-reasoned decision that’s based on the actual behavior of your customers, buying cycle, types of campaigns etc.

In the following example, we are using the same dataset as in the previous example. The default attribution model used there is “last interaction” but we’d like to compare it against the more commonly used “last non-direct model”.

First, let’s see the numbers using the “last interaction” attribution model.

And now let’s see the numbers using the “last non-direct click” attribution model.

The query here is a bit bulkier but it’s actually quite simple and logical when you take a closer look.

While the difference isn’t massive for this dataset, we can see that there’s a lot less direct/none traffic using the “last non-direct click” model – which makes total sense, because all direct clicks that came after a non-direct click were attributed to that other source.

***

We aren’t going in too much detail with all the attribution models you could build using the raw data in BigQuery but let me say that lately, many of our clients have requested us to build them all sorts of custom attribution models like linear and funnel-based or Markov, and even some machine-learning-based models.

Various data sources

While Google Analytics raw hit-level data (from GA 360 or using the parallel tracker) tends to be the most commonly used dataset in BigQuery or any other marketing data warehouse, you can (and should) include data from all marketing tools and platforms. This allows you to join data from various sources and spot insights and connections that weren’t possible before.

For example, one might capture a daily snapshot of their Twitter account in BigQuery and run a quick query to see the data for a current month.

The data above would fit perfectly in a social media marketing dashboard.

In another example, let’s query a table containing Mailchimp data to see how many new subscribers were acquired per day.

As you’ve probably realized, you can (and should) send data from all marketing tools into one unified data warehouse. As BigQuery and most other solutions are available at a rather reasonable pricing point, I would recommend collecting data from all sources, even ones you don’t plan on using in the nearest future because it’s always good to have access to historical data for a more advanced ad-hoc analysis.

Moving forward

If your company already has a marketing data warehouse that’s built on a modern cloud infrastructure and contains data from all the marketing tools that your business is using – good job! You’re working at a good, forward-thinking company. Just make sure this data is actually put in use as well.

If that’s not the case and your company doesn’t have a proper marketing data warehouse in place, start by thinking of all the marketing (and other relevant) tools that are producing data. You’ll probably realize that what you have is a bunch of silos, limiting the efficiency of your teams and holding your business back from being a truly data-driven one.

Whether you’re planning on building the data pipeline that feeds your data warehouse with high-quality data yourself or to buy it in as a service, it never hurts to consult with the experts that have been building data management systems for years.

If you have any questions, feel free to shoot them in the comments below.

The post Making BigQuery a Single Source of Truth for Marketing Data Analysis appeared first on Reflective Data.

Working with Google Analytics data in BigQuery has mostly been a privilege of those having a 360 version of Google Analytics. Its hefty price tag, though, has made that list quite short.

With Google Analytics Parallel Tracking Service from Reflective Data, having access to Google Analytics data in BigQuery has become a lot more affordable. (you can get a quote from the previous link)

Now, regardless of whether you have Google Analytics 360, a Parallel Tracking system from Reflective Data or some other integration in place, the structure of the data in BigQuery is quite similar.

In this article, I’m giving you a tour of the features in BigQuery and some of the ways how you can leverage them when working with your Google Analytics data.

Take a look at this article if you want to learn more about sending your Google Analytics data into BigQuery first.

In case you don’t have access to any BigQuery dataset containing Google Analytics data, you can check out the Google Analytics sample dataset for BigQuery. The sample dataset provides an obfuscated Google Analytics 360 dataset that can be accessed via BigQuery. It’s a great way to look at business data and experiment and learn the benefits of analyzing Google Analytics 360 data in BigQuery.

In this blog post, though, we are going to work with the demo dataset generated by Reflective Data’s Parallel Tracking System. As mentioned before, the structure of the data is pretty much the same.

Creating and running your first query

With Reflective Data’s Parallel Tracking System, you will have three main tables in your BigQuery dataset. Let’s take a quick look at each one of them with a simple query.

Table: raw_hits

This is the most up to date table and contains raw data coming in from the website, app or via the measurement protocol. Data is available withing ~5 seconds after it was sent.

raw_hits table contains only four columns.

• ua – the user agent string of the client
• ip – the ip address of the client
• q – the actual Google Analytics hit payload
• timestamp – when data hit the processing engine (~50ms after it was sent)

For example, we could query the number of hits for every hour in a given day.

Table: processed_hits

This is a processed version of the raw_hits table. Data in this table becomes available ~30 seconds after the hit was sent. This is still really-really fast and considered near-real-time data.

When it comes to columns, this table is much richer compared to the raw_hits table. It has all the time-related columns (date, hour, minute, timestamp etc.) converted into the correct timezone. Some of the data is in a special format known as a record in BigQuery. This includes columns like traffic_source, page, event, device, transaction etc. In fact, this table has a column for every parameter you can send with a Google Analytics hit. Added are some data points normally not available in Google Analytics (ip address, user agent, client ID, user ID etc.)

List of paramaters/columns as seen in Measurement Protocol Reference.

As an example, we are querying top 10 visitors (by client ID) that created the most hits in a given day.

Isn’t that easy? and cool, too!

I certainly like working with my Google Analytics data using SQL queries. Okay, let’s move on.

Table: processed_sessions

This table is, for sure, the richest of the three. It has all the columns from processed_rows but also includes quite a few new ones. These are the columns that require more processing (ip address to geolocation etc.) and/or the session to be ended (total hits in a session etc.). For these reasons, this table is usually updated a few times a day and a final version is generated at the beginning of the next day.

If you aren’t completely sure how a web session is defined in Google Analytics, take a look at this article.

A good thing about raw data is that whether you like the fact that Google Analytics is mostly session-based or not, you have the freedom to calculate your metrics however you wish. You could also mix and match different techniques.

For example, we might query top country/browser combinations by the number of sessions and pageviews in a given month. Let’s see how that would look like with our demo dataset.

Creating and running more complex queries

The best thing about raw hit-level data is that you’re not limited by things like sampling or pre-defined metric/dimension combinations. Both of which are common in Google Analytics, both the UI and the Reporting API.

What’s more, with BigQuery you can load in data from almost anywhere and mix it with your analytics data in any way you like. This may include data from your CRM, CMS, Ads platform or other analytics tools.

Google Analytics’ data model is structured so that session-based dimensions (like source/medium) don’t play well when combined with user-level or page-level dimensions and metrics. And there’s a limit to the number of dimensions we can see side-by-side: two dimensions is usually our limit in the interface, five in custom reports, and the API allows for seven dimensions. With BigQuery, there are no such limitations.

For example – ecommerce customers may have trouble pulling exact stats of the number of users from Social Media that saw a product page and then subsequently purchased the same product. BigQuery users can handle that with a single query.

By Alex Moore, source

One of the queries that almost always ends up being inaccurate (due to sampling) in Google Analytics is when you want to analyze all hits triggered by a single user. With BigQuery, this query is fast and simple, and there is no sampling. Never.

First, let’s go ahead and query some totals.

Now, let’s take a look at all of the hits in chronological order. This will give us a detailed overview of their user journey.

I believe you can see how analyzing a user journey like this can be very useful. For example, when a new lead comes in, before contacting them you could check what content they’ve already interacted with.

***

For someone that has worked mostly with the Google Analytics UI and not with the raw data before, there are some things that can be a bit confusing in the beginning. For example, not all of your common metrics are available as a column in BigQuery. Even the really basic ones, like bounce rate. That means you have to define them in your query.

Let’s take a look at the bounce rate for the top 10 landing pages in our sample dataset.

Calculating your own metrics like this can be a bit frustrating in the beginning but you actually only have to write most queries once and can later save them as views or saved queries in BigQuery. A benefit of defining your own metrics is that you are not limited by pre-defined metrics in Google Analytics. As you can see, in the query above we defined bounce as a session that has less than 2 pageviews but we could also include a rule that there can’t be any interaction hits in the session. We could also play with session duration or other data available in BigQuery.

A true value of having your analytics data in BigQuery, though, is that you can easily join it with other data that you’ve also sent to BigQuery.

For example, you could use Google Ads Data Transfer tool for BigQuery to automatically send all of your Google Ads data into a BigQuery dataset. This is especially useful if you’re using Google Ads auto-tagging solution because that means the only connection between your Google Analytics and Ads data is the gclid ID which is unique for every click.

With a simple query, you can join your processed_sessions table with ClickStats table.

After joining two datasets, you can combine any data points that are available in either dataset. The most important one, perhaps, is the cost data from the Ads dataset. This would allow you to calculate the amount of money you spent on acquiring each user.

Analyzing BigQuery data in a BI tool

Not everyone can write complex SQL queries to access their analytics data in BigQuery. Luckily, BigQuery has a native connector with almost all of the major BI and data visualization platforms.

My default recommendation is Data Studio. It’s free, full of powerful features and since it’s also from Google, you get a smart caching layer between Data Studio and BigQuery which can significantly lower your BigQuery costs.

Connecting BigQuery and Data Studio is really simple. All you have to do is create a new source in Data Studio and select BigQuery. It will automatically show you the list of Google Cloud projects you have access to and from there you can easily navigate to your datasets and tables.

Once you found the correct table, you can either choose the fields you want to include in your report (if not sure, choose all) or write your own query.

In our example, we connected two tables (processed_hits and processed_sessions) and selected all fields available in these tables.

Now, let’s create a quick time-series chart to see the number of hits of each hit type per day in our selected time period. Settings for a chart like this are really simple.

And this is how the chart itself looks like. Nothing special but allows you to have a quick overview of your traffic and spot any anomalies in tracking.

Another useful chart I recommend setting up is a time-series comparison between data in Google Analytics and BigQuery. This means you have to blend two data sources.

As you can see, the numbers are pretty close.

For more advanced dashboards you might want to write a custom query on a Data Studio data source level. This way you can pre-join different tables and/or datasets.

***

While being more difficult than using the Google Analytics UI, the freedom you get from having access to raw hit-level data in BigQuery is definitely worth it. As mentioned at the beginning of this article, you don’t necessarily have to spend a fortune on Google Analytics 360 because with Reflective Data’s Google Analytics Parallel Tracking System you can send your analytics data into BigQuery for a fraction of the money.

Should you have any questions about working with Google Analytics data in BigQuery, post them in the comments below and someone from our team will get back to you.

If you want to learn more about the Parallel Tracking System, click here.

The post How to Query and Analyze Google Analytics Data with BigQuery appeared first on Reflective Data.

Two of the most common shortcomings of Google Analytics that most of the advanced users experience, though, are the lack of hit-level granularity and data sampling. In this article, we are taking a look at some of the ways you can overcome these shortcomings without spending a fortune on Google Analytics 360.

PS! If your company already has 360, these techniques can give you an even more robust and complete dataset.

What is sampling and who’s affected

At first, let’s take a look at how Google Analytics describes sampling in their official documentation.

In data analysis, sampling is the practice of analyzing a subset of all data in order to uncover meaningful information in the larger data set. For example, if you wanted to estimate the number of trees in a 100-acre area where the distribution of trees was fairly uniform, you could count the number of trees in 1 acre and multiply by 100, or count the trees in a half acre and multiply by 200 to get an accurate representation of the entire 100 acres.

So, it simply means that some of the reports you see in Google Analytics (or any other tool that pulls data from it via the Reporting API) may not represent 100% of the relevant hits.

This is especially true when you build more advanced custom reports with detailed custom segments. Google Analytics samples complex ad-hoc queries even below 500k sessions. Unfortunately, this is exactly when you normally care about the data accuracy the most.

You can check if your report is affected from sampling by hovering over the shield icon at the top left of your report.

Normally, the reporting-level sampling starts when your selected date range has more than 500k sessions in total. Be vary, though, because it sometimes happens with less traffic as well.

In some circumstances, you may see fewer than 500k sessions sampled. This can result from the complexity of your Analytics implementation, the use of view filters, query complexity for segmentation, or some combination of those factors. Although we make a best effort to sample up to 500k sessions, it’s normal to sometimes see slightly fewer than 500k sessions returned for an ad-hoc query.

And that is not the only kind of sampling that can haunt you in the free version of Google Analytics. The second type of sampling takes place when data is being collected and the limits you need to know are as follows.

• 500 hits per session [1]
• 200,000 hits per user per day [1]
• 10 million hits per month per account [2]

source 1, source 2

It is important to understand that hits aren’t users, sessions or page views – hits are all data sent to Google Analytics including events, timing and data coming from the Measurement Protocol.

One way to check if you’re getting close to those limits is to go to Admin –> Property Settings in Google Analytics.

It is worth mentioning, though, that Google won’t automatically ignore all hits past the 10M mark. It will notify you in the UI and will likely contact you via email and suggest considering the GA 360 version. In the terms, though, it says “there is no assurance that the excess hits will be processed” and also that the warning message is not guaranteed to appear.

You should take action if…

• your site receives >10M hits a month
• you see the yellow shield next to your reports regularly
• your visitors generate more than 500 hits per session

How to avoid sampling in Google Analytics

If you hit any of the sampling limits but need reliable data in your work, you need to find a solution rather sooner than later.

Let’s take a look at the options you have.

Google Analytics 360 – This is the solution Google itself recommends. And no wonder why, it costs around $150k a year. GA 360 is a great tool and we recommend it to all companies with huge traffic and money to spend. Keep in mind, though, that if you only need the more generous data limits or BigQuery access, there are cheaper solutions (described in this article).

Collect less data – Well, who would want less data, right? We can’t recommend to skip tracking of some important user action like page view or file download but there may some automatic events that you don’t really care about. This could be some timing event that your systems sends every 10 seconds, a scroll depth event every 5% or something similar. Take a look at your events and you may find something. Just don’t remove something useful!

Unsample your Google Analytics data – If you don’t hit the data collection limits and the only worry is the sampling happening on reporting-level, you may be interested in solutions that let you unsample your existing data. How it works is that your query (run via the API) is divided into many sub-queries that are small enough that no sampling is applied. You can do so by writing your own small program in your favorite language or use one that others have built. For example this one written in GO.

There are also some paid tools available but we haven’t used them. Some of them run the API requests periodically to build an unsampled database based on your Google Analytics data. Keep in mind, though, that this doesn’t save you from data collection limits (i.e. 10M sessions/month) or some data aggregation that is inevitable in GA.

Use a parallel tracker – This is the most reliable solution against all data collection, processing and reporting limits you come across in Google Analytics. How it functions is that it duplicates all hits going from your site to Google Analytics, processes them separately and stores everything in your favorite data warehouse – BigQuery, for example.

With a parallel tracking setup, you are always free from any sampling and data collection limits. Furthermore, data processing incidents are rare but do occasionally happen in Google Analytics – having your own raw dataset lets you reprocess your data whenever needed. As a bonus, since you own the data, you may include PII, mix it with any other data or delete the records you don’t want. More on this solution later in the article.

What is hit-level data and why do I need it

Most data you see in Google Analytics is aggregated, and without custom configuration, you can’t get much of the raw hit-level (also known as event-level) data that you may need in more detailed analysis.

Hit-level data means that you can access the underlying hits that were sent to Google Analytics, allowing you to do your own aggregation as you wish, based on any criteria or dimension.

A good example of using hit-level data is analyzing the journey of a single user. On what page did they land on, what was the traffic source, which pages did they visit, how many interactions before converting etc. Having access to hit-level data lets you analyze the journey of each visitor in near-real-time. Furthermore, this data is perfect for machine learning algorithms that could, for example, detect and analyze users that are most likely to convert – you could then target them with ads and other campaigns.

Here’s the simplest query in BigQuery that shows you each hit from a single user in chronological order.

With hit-level data, one can access every single hit that was collected from the site along with all data-points that each hit included.

Unfortunately, there is no way you can get the true raw hit-level data out of Google Analytics. Not even using the API. Using custom dimension for the Client ID, Hit Type, Timestamp etc. can get you closer but it’s still far from perfect.

How to get access to raw, unsampled hit-level data

As mentioned before, you can’t get the raw hits out from Google Analytics. The premium version of Google Analytics (360) and its BigQuery export feature will get you closer (for 150k a year) but even that is not ideal.

The only way to gain access to the real underlying hits, with zero sampling and aggregation, is to leverage technology known as parallel tracking.

Parallel tracking means that all hits sent to Google Analytics are duplicated and sent to another endpoint. Depending on the solution, the data may be stored in Amazon S3, Google BigQuery or some other data warehouse.

Tools like Snowplow offer parallel tracking solution and accessing raw data in your data warehouse without any further processing (by default). This means there are no sessions, channel attribution or other really useful features that you do get in Google Analytics.

With Reflective Data’s Parallel Tracking (RDPT) solution, not only will you get all the raw hits but also a data processing engine that works very similarly to the one in Google Analytics itself. This means you will get sessions, attribution and features like referral exclusion out of the box. More advanced users can build (or request) their own rules for defining sessions, attribution and other features.

In order to break data silos, RDPT can integrate with any tool that has an API. Including your CRM and ad platforms.

RDPT’s default data warehouse is BigQuery but storing data elsewhere (Amazon S3 etc.) is also possible. BigQuery’s native integration to Google Data Studio makes it easy and cost-effective to build all sorts of reports and dashboards. Integrations with most other BI and data visualization tools are widely available.

How much does it cost

Google Analytics 360 is a really good tool for enterprises that want to gain more detailed access to their marketing data. Hefty price tag, limited access to raw data and occasional sampling should make you think twice before upgrading, though.

For companies that aren’t fully sold on Google Analytics 360 or companies that already have 360 but need access to even more complete dataset, the Reflective Data’s Parallel Tracker (RDPT) may be the perfect solution.

Pricing for RDPT depends on the amount of traffic, the complexity of the setup and the number of integrations. Compared to 150k, though, it will always be a bargain. The initial setup usually costs somewhere between $1k and $5k, and the monthly plans start at around $350. This includes a generous quota for BigQuery usage.

So, whatever your current analytics stack looks like, you should consider adding RDPT for the most robust, unsampled raw hit-level digital analytics data you can get. Learn more here.

The post Unsampled Hit-Level Google Analytics Data Without 360 appeared first on Reflective Data.

In this article, we are covering everything you need to know about the problem of duplicate transactions, a root cause of skewed data in many Google Analytics instances.

Detecting Duplicate Transactions

Duplicate transactions in Google Analytics means simply that a single transaction was counted more than once. This can dramatically skew your data because along with the transaction count, this also inflates the revenue, quantity and other metrics directly related to transactions.

To check if your Google Analytics E-commerce setup is affected by duplicate transactions, let’s start by creating a custom report (Customisation –> Custom Reports –> New Custom Report) with the following settings.

Metrics: Transactions

Dimensions: Transaction ID

Hit “Save” and you should see your Duplicate Transactions custom report right away.

Firstly, choose a time frame. I’d recommend starting with 30, 60 or 90 days depending on how many transactions your site generates and how often your analytics setup changes.

Secondly, sort your table by most Transactions first.

If every transaction ID shows only one transaction then congratulations, in the selected time frame there were no duplicate transactions.

Every number greater than one is a sign of duplicate transaction. Now, to get an overview of the magnitude of the problem, compare the number of transactions against the number of rows in the table. The latter is the actual count of unique transactions.

In the above screenshot, there are some duplicate transactions but considering the total amount of transactions, the problem is minor.

If your duplicate transaction ratio is >10% you have a serious problem with your e-commerce numbers and before fixing it, you should take your numbers with a grain of salt.

PS! We should be thankful that Google has kept Transaction ID available as a dimension in almost all reports instead of hiding them as they do with the Client ID, User ID, Session ID and other really useful dimensions (that you should track using custom dimensions).

Fixing Duplicate Transactions

By far the most common reason duplicate transactions are happening is that a transaction hit is sent to Google Analytics every time the “thank you” page is loaded. There are people who bookmark this page and go back to check the order id for tracking or whatnot. Also, some companies send the “thank-you” URL in the order confirmation email.

To fix this, there are two possible solutions.

Option 1

Make the event trigger based on something coming from the back end, something that only triggers when the actual transaction is happening. For example, after the user has entered their credit card information and pressed “Complete payment” and your system has validated the transaction. I.e. just before redirecting the user onto the “thank you” page.

Option 2

Make your “thank you” pages to load only once. If you keep your transaction hits on the “thank you” page, make sure a user can see and visit the page only once. Every subsequent attempt to access this page should redirect them to some other page. This could be an order overview page or something similar (should still be order-related and contain order details).

…

Now, to take it one step further, you can make your e-commerce tracking even more reliable by sending transaction hits straight from the back end using Measurement Protocol. This way you can make sure that no transaction goes untracked or that someone is sending fake transactions towards your Google Analytics instance.

Another option is to use localStorage to keep track of transaction IDs that have already been sent to Google Analytics. This method is described in detail by Simo Ahava in this blog post.

Final thoughts

Google Analytics is an excellent tool for keeping track of your e-commerce business. What you need to keep in mind, though, is that skewed numbers can do more harm than having no numbers at all.

This is why you should audit (or have someone else do it for you) your Google Analytics and Enhanced Ecommerce setups regularly and compare the numbers in GA against the numbers in your back end and accounting.

If you see that the numbers don’t add up, see if you have any duplicate transactions by creating a simple custom report seen in the beginning of this blog post.

Have any questions or thoughts about detecting and fixing duplicate transactions in Google Analytics? Post them in the comments below.

***

To make sure you have access to reliable data even after problems like duplicate transactions have skewed your data, we recommend sending all of your Google Analytics data into a data warehouse like BigQuery. This makes sure you can filter out duplicated data and reprocess your reports to be 100% accurate. One solution to achieve this is to use a Google Analytics Parallel Tracker from Reflective Data.

The post How to Detect and Fix Duplicate Transactions in Google Analytics appeared first on Reflective Data.

Python is a programming language with virtually limitless functionalities and one of the best languages for working with data. Jupyter Notebooks, on the other hand, is the most popular tool for running and sharing both your Python code and data analysis.

Putting Python and Notebooks together with Google Analytics, the most popular and a really powerful tool for tracking websites, gives you almost like a superpower for doing your analysis.

This is exactly what this post is about. Pulling in and analysing your Google Analytics data using Python and Notebooks.

Oh, and by the way, this blog post itself is a Jupyter Notebook created in Google’s Colab, a version of Jupyter Notebooks that makes it super easy to collaboratively work on and share your Notebooks.

We wrote more on how we managed to turn a Notebook into a WordPress blog post in this article.

Without furder ado, let’s start by loading in some dependencies we are going to need for superfsadfsadfsdfsafasdfsadfpulling data from Google Analytics.

import pandas as pd # Pandas is an open source library providing high-performance, easy-to-use data structures and data analysis tools for Python
import json # JSON encoder and decoder for Python
import requests # Library for sending HTTP requests

There are many ways you can pull data from Google Analytics but one of the simplest is by using the API Query URI that you can grab from the Step 1 – Go to Query Explorer

If you are not familiar with the Query Explorer, this is an official tool from Google that lets

you play with the Core Reporting API by building queries to get data from your Google Analytics views (profiles). You can use these queries in any of the client libraries to build your own tools. In this case, the tool is a Notebook.

Step 2 – Authenticate

Choose the Google Account that has access to the Google Analytics View from which you want to get data.

Step 3 – Build the query

If you know how Google Analytics works, building a query is rather straightforward.

First, choose the right Account, Property and View you want to access.

Then choose the date range. You can use the calendar picker or write dynamic ranges like from 90daysAgo to yesterday.

Next, pick the metrics and dimensions. For example, you might pick metrics ga:sessions and ga:users and a dimension ga:date

.

Finally, you can apply sorting, filtering and segmenting rules as you wish. I’d recommend using them after you feel comfortable with the basics.

Perfect. Now test your query by pressing the “Run Query” button. If everything went as expected, you should see a table with results appear in a few seconds.

Step 4 – Get the API Query URI

Now, copy the API Query URI – we are going to use it for making requests straight from the Notebook.

Other options to pull data from Google Analytics using Python include:

• Using the Google Analytics Reporting API library for Python in your Notebook – based on Google API Client for Python
• Building a custom endpoint for handling your queries using the abovementioned API
• Exporting data from Google Analytics UI into CSV or XLSX and importing using pd.read_csv()

Making the request

Now that we have the API Query URI, we can make our first request using Python in the Notebook.

In this example, we are loading the number of Sessions and Users per day for the last 30 days.

from getpass import getpass # required only for hiding your variables

api_query_uri = getpass('Enter API Query URI here') # get the URI
r = requests.get(api_query_uri) # make the request
data= r.json() # read data from a JSON format
df = pd.DataFrame(data['rows']) # turn data into a Pandas data frame
df = df.rename(columns={0: 'Date', 1: 'Sessions', 2: 'Users'}) # giving the columns some proper titles
df['Sessions'] = df['Sessions'].astype(int) # formatting sessions as ints
df['Users'] = df['Users'].astype(int) # formatting users as ints
df['Date'] = pd.to_datetime(df['Date'])

df.head() # printing the first five rows

Enter API Query URI here··········

	Date	Sessions	Users
0	2019-09-25	64076	83335
1	2019-09-26	74569	103784
2	2019-09-27	59752	77642
3	2019-09-28	57743	76690
4	2019-09-29	63712	82980

Great, so our data is now loaded and formatted for our needs.

Let’s go ahead and try visualizing it. For timeseries data, a line chart should work well.

df.plot.line(x='Date', y=['Sessions', 'Users'], ylim=[0,None], figsize=[10, 6])

<matplotlib.axes._subplots.AxesSubplot at 0x7f3cb89692b0>

That was easy, right?

Now, let’s take a look at a bit more advanced visualization by drawing a population pyramid based on data we are now pulling from Google Analytics.

This is how the query looks like:

dimensions: ga:userAgeBracket, ga:userGender
metrics: ga:sessions

api_query_uri = getpass('Enter API Query URI here') # get the URI
r = requests.get(api_query_uri) # make the request
data= r.json() # read data from a JSON format
df = pd.DataFrame(data['rows']) # turn data into a Pandas data frame
df = df.rename(columns={0: 'Age', 1: 'Gender', 2: 'Sessions'}) # giving the columns some proper titles
df['Sessions'] = df['Sessions'].astype(int) # formatting sessions as ints
df.head()

Enter API Query URI here··········

	Age	Gender	Sessions
0	18-24	female	56244
1	18-24	male	9362
2	25-34	female	228866
3	25-34	male	33987
4	35-44	female	109795

import matplotlib.pyplot as plt # Matplotlib is a popular library for drawing plots in Python
import seaborn as sns # Seaborn is a Python data visualization library based on matplotlib

# Making copy of the dataframe and modifying session values to be negative for females (see chart below to see why)
df['Sessions'] = df.apply(lambda row: row['Sessions'] * -1 if row['Gender'] == 'female' else row['Sessions'], axis=1)

# Draw Plot
plt.figure(figsize=(8,5), dpi= 80)
group_col = 'Gender'
order_of_bars = df.Age.unique()[::-1]
colors = [plt.cm.Spectral(i/float(len(df[group_col].unique())-1)) for i in range(len(df[group_col].unique()))]

for c, group in zip(colors, df[group_col].unique()):
    sns.barplot(x='Sessions', y='Age', data=df.loc[df[group_col]==group, :], order=order_of_bars, color=c, label=group)

# Decorations    
plt.xlabel("$Sessions$")
plt.ylabel("Age")
plt.yticks(fontsize=12)
plt.title("Population pyramid of the website visitors", fontsize=16)
plt.legend()
plt.show()

Beautiful, now we know that majority of our audience is younger females.

More custom visualizations

With the wide variety of Python packages available for data visualization, the number of different charts you can use is virtually limitless.

In the next example, let’s try out joy plot. If you don’t know what a joy plot is then you are about to learn it now.

First, let’s install the required Python package.

!pip install joypy # install joypy
import joypy # JoyPy is a one-function Python package based on matplotlib + pandas with a single purpose: drawing joyplots.

Requirement already satisfied: joypy in /usr/local/lib/python3.6/dist-packages (0.2.1)

Now, let’s make the request to load required data. For this example we are using the number of transactions per age bracket and hour of the day for the past 90 days.

api_query_uri = getpass('Enter API Query URI here') # get the URI
r = requests.get(api_query_uri) # make the request
data= r.json() # read data from a JSON format
df = pd.DataFrame(data['rows']) # turn data into a Pandas data frame
df = df.rename(columns={0: 'Hour', 1: 'Age', 2: 'Transactions'}) # giving the columns some proper titles
df['Hour'] = df['Hour'].astype(int) # formatting hours as ints
df['Transactions'] = df['Transactions'].astype(int) # formatting transactions as ints
df.head()

Enter API Query URI here··········

	Hour	Age	Transactions
0	0	18-24	47
1	0	25-34	195
2	0	35-44	32
3	0	45-54	26
4	0	55-64	26

To make our dataframe more suitable for the joy plot, we need to pivot it.

pivot = pd.pivot_table(df, values='Transactions', index=['Hour'],
    columns=['Age'])

pivot = pivot.fillna(0)

pivot.head()

Age	18-24	25-34	35-44	45-54	55-64	65+
Hour
0	47	195	32	26	26	0
1	37	68	37	16	42	0
2	21	16	5	11	5	0
3	0	16	11	0	5	0
4	5	5	5	0	0	0

Now, let’s see how our joy plot looks like by running the following code.

x_range = list(range(24))
fig, axes = joypy.joyplot(pivot, kind="values", x_range=x_range, figsize=(10,8))
axes[-1].set_xticks(x_range);

Looks quite good already. but since most of our customers were in the age bracket 25-34 and we want to visualize the difference in their behavior rather than absolute numbers, let’s normalize the data.

df_norm = (pivot - pivot.min()) / (pivot.max() - pivot.min())
df_norm.head()

Age	18-24	25-34	35-44	45-54	55-64	65+
Hour
0	0.356061	0.313531	0.096429	0.130	0.206349	0.0
1	0.280303	0.103960	0.114286	0.080	0.333333	0.0
2	0.159091	0.018152	0.000000	0.055	0.039683	0.0
3	0.000000	0.018152	0.021429	0.000	0.039683	0.0
4	0.037879	0.000000	0.000000	0.000	0.000000	0.0

Now that the data is normalized, let’s see how a joy plot would look like.

fig, axes = joypy.joyplot(df_norm, kind="values", x_range=x_range, ylim='own', overlap=1.5, figsize=(10,8))
axes[-1].set_xticks(x_range);

Much better in my opinion.

So, what does this joy plot tell us?

• Most transactions are made between 8 AM and 10 PM, almost no one buys between 2 AM and 6 AM.
• Older (65+) people don’t buy after 11 PM while the youngest bracket (18-24) continues buying until 3 AM.

While I agree that this plot didn’t teach us something extremely interesting or new, it is a good example of what’s possible with a few lines of code and Jupyter Notebook (or any other similar tool).

Working with raw hit-level Google Analytics data

While connecting Jupyter Notebooks directly to your Google Analytics view is enough for many use cases, having access to raw hit-level Google Analytics data opens completely new opportunities for more advanced analysis.

To gain access to raw hit-level Google Analytics, one can use a tool like Marketing Data Pipeline to send data into BigQuery. Python has a library for connecting to BigQuery which makes it easy to explore and visualize based on raw data and metrics calculated based on your own criteria.

Besides other use cases, having access to raw hit-level data enables you to get a complete overview of the user journey, calculate metrics like LTV and use your own models for marketing channel attribution (Markov, ML-based, FPA etc.).

Learn more about the benefits of having a marketing data warehouse.

Conclusion

Combining powerful tools like Google Analytics, Python and Jupyter Notebooks gives you the flexibility and options you probably didn’t even dream about. The first time I got to play around with these three together made me feel like a little child, like I had discovered something really-really cool (and it is).

I hope this was enough to get you excited and that you will now go and try doing it on your own. And as always, your comments, questions and suggestions are welcome in the comments below.

The post Working with Google Analytics Data Using Python and Jupyter Notebooks appeared first on Reflective Data.

Custom Events provide a perfect solution for adding context and tracking more specific user actions. In this article, we are giving you a good amount of ideas for custom events you should implement on your own and/or your clients’ websites.

This post is divided into sections by different event categories so you could skip the ones that are not relevant for you.

Clicks

For almost all websites, the most common user interaction is a click. By default, Google Analytics isn’t very good at tracking them. Let’s see how we could improve this situation.

Navigation clicks

Tracking usage of the main menu, sidebar and other navigation elements is critical. This is how your visitors explore your business and move in the funnel.

Detecting navigation interactions is easy with Google Tag Manager and sending data to Google Analytics works as with any other event.

Since there may be several navigation elements containing links to the same page, it is important to also pass the information about which navigation was used. For example, you might a have link to your blog both in the header and in the footer.

Recommended event parameters:

category: click
action: top-navigation, sidebar etc.
label: link text
non-interaction: false

CTA clicks

Tracking clicks on all of your CTAs (call to actions) is extremely important if you have more than one CTA on a single page pointing to the same location (i.e. long-form sales pages).

Don’t forget that buttons like “Get a Quote”, “Add to Cart” and “Checkout” are also CTAs.

Setup is the same as with generic click tracking using GTM.

Recommended event parameters:

category: click
action: CTA
label: text (and index the same CTA appears multiple times)
non-interaction: false

Outbound link clicks

Also known as tracking website exits, this custom event tells you exactly to which external pages people navigate from your site.

Again, setting this up is quite easy with GTM.

Recommended event parameters:

category: click
action: outbound link
label: link URL - link text
non-interaction: false

File download clicks

Doesn’t matter if you provide whitepapers, ebooks, PDFs, or other types of downloadable files, you should be tracking how often they get downloaded.

There are some really good tutorials online but we recommend setting it up using Google Tag Manager. A simple “just links” click trigger with a regex that matches desired filetypes should work just fine.

Regex Example:

\.(pdf|docx)$

Recommended event parameters:

category: click
action: download
label: filename / link text
non-interaction: false

Rage clicks

Visitors rage clicking on certain elements on your website is a good indicator of a UX error. For example, people may click on a blue text that is not a link or on an image that has no click functionality.

We have a separate blog post about tracking rage clicks using Google Tag Manager and Google Analytics.

Recommended event parameters:

category: click
action: rage click
label: element selector
non-interaction: false

Button clicks

Your website is likely to have all sorts of clickable buttons. There might be some other ways for you to get an overview of the usage of the features related to those buttons but knowing how many clicks each button received definitely helps.

Some ideas for which buttons should be tracked:

• Add to wishlist/favorites
• Add/remove item from cart
• Update cart
• Continue shopping
• Log in/out
• Register
• etc.

Recommended event parameters:

category: click
action: button
label: button text/description
non-interaction: false

Form Interactions

When it comes to tracking forms, I would like to recommend a few good articles by Simo Ahava that cover everything you need for getting started.

• Track Form Engagement With Google Tag Manager
• Track Form Abandonment With Google Tag Manager
• Form Field Timing With Google Tag Manager

Since forms are the most important part of many websites (i.e checkout, lead form etc.) you should really nail tracking them correctly.

A/B Testing

Experiment and Variation

Most A/B testing tools provide some sort of integration with Google Analytics. Most of them, though, are using custom dimensions. A problem with custom dimension based integration is that you need to make sure that the dimensions aren’t overlapping between experiments because otherwise, you might end up overwriting the dimension value for some experiments.

What we recommend having, besides the default integration, is a secondary event-based integration. That is, sending a custom event for every running experiment in a following format:

category: ab testing
action: experiment ID
label: variation number
non-interaction: true

PS! One event per experiment-variation combination.

This gives you raw information about each experiment and variation that the visitor saw. You can easily use this information in your custom segments or reports.

Every testing has a bit different solution for exposing experiment data to the front-end but here are a few links that should be helpful.

• VWO – instead of custom dimension, send this data using custom event action and label.
• Google Optimize – this allows you to get experiment data to the datalayer, now send it using a custom event.
• Optimizely X – access data about live experiments using Javascript.

Google Optimize Anti-Flicker Snippet timeout

If you have any experience with JavaScript-based A/B testing and/or personalization tools you know that flicker can be a real headache.

In case your tool of choice is Google Optimize, you should be using their official anti-flicker snippet to minimize the flicker effect.

We wrote a full separate article on Tracking Optimize Anti-Flicker Snippet Timeout in Google Analytics.

Recommended event parameters:

category: optimize
action: snippet timeout
label: true/false
non-interaction: true

Scroll depth

Whether you are working with a blog, some landing pages or any other page that is longer than a fold, you want to know how far down do the visitors scroll.

It used to required some custom Javascript but now, tracking scroll depth is a built-in feature of Google Tag Manager.

Recommended event parameters:

category: scroll
action: scroll depth threshold
non-interaction: false

Social interactions

While Google Analytics has a dedicated feature for tracking social interactions, having this data in custom events gives you more flexibility in reporting.

There is a really thorough blog post about tracking social interactions using Google Tag Manager – although, if you want something simpler, tracking clicks on your social icons is a good starting point.

Recommended event parameters:

category: social
action: share/like/follow
label: social platform
non-interaction: false

Video events

Doesn’t matter if it’s a sales video, a video that explains how your product works or a tutorial – you do care about how many people start the video and how far through will they make.

Google Tag Manager provides a really easy solution for tracking YouTube videos but tracking others such as Vimeo is completely doable as well.

Recommended event parameters:

category: videos
action: start/pause/percentage
label: video id
non-interaction: false

Errors

No matter how good your site is, there will always be errors. Since errors are directly related to bad user experience, tracking them is absolutely crucial.

The first step here should be sending your Javascript errors to Google Analytics. Next, consider sending other errors like popup alerts and system errors as well.

Recommended event parameters:

category: error
action: error message
label: feature or other relevant information
non-interaction: true

reCAPTCHA bot score

Google Analytics does have a simple bot-filtering system in place but it comes nowhere near catching all the bots that visit your site. Even tools like reCAPTCHA by Google can’t usually say for sure whether a visitor is a bot or a human but they can give you a score (from 0.1 to 1). Simo Ahava has put together a great tutorial for tracking the reCAPTCHA both score using Google Analytics custom dimensions.

Recommended event parameters:

category: reCAPTCHA
action: reCAPTCHA answer
label: reCAPTCHA score
non-interaction: true

Don’t forget to test whatever new custom event you add to your setup.

Should you have any questions related to custom events in Google Analytics, shoot them in the comments below.

PS! If you need help with implementing custom events, you can hire us to help you out!

Photo by Annie Spratt on Unsplash

The post Data You Should Be Tracking Using Google Analytics Custom Events appeared first on Reflective Data.

Detecting and fixing such usability issues is crucial, and luckily the detection part is doable using Google Tag Manager and Google Analytics.

Rage click is a situation where, in a short period of time, a website user clicks multiple times on a specific element. I.e. in two seconds, a visitor clicks three times on an image.

Without further ado, let’s see how tracking rage clicks works in Google Tag Manager.

Step 1 – Create a new Tag that detects rage clicks

This tag is responsible for detecting rage clicks and pushing a custom event rage_click into the data layer.

There are a few variables that you can play with to make it best fit your use case:

no_of_clicks – after how many clicks do we consider it a rage click. Default is 3.

time – in what time period should the clicks happen. Default is 2 seconds.

radius – in what radius should the clicks happen. Default is 100px.

The following variables are pushed into the data layer when a rage click is detected:

event – rage_click

rc_element – selector of the element on which the clicks happened

rc_count – the number of clicks

rc_max_distance – maximum distance between the clicks

rc_time_diff – period from first to last click

PS! You don’t have to send all of them to GA, pick the ones you think make the most sense.

Full code snippet. – Requires jQuery to be loaded on the page.

if ( typeof(jQuery) === 'function' ) {
    jQuery(document).ready(function($) {
        jQuery.fn.extend({
            getPath: function() {
                var path, node = this;
                while (node.length) {
                    var realNode = node[0],
                        name = realNode.localName;
                    if (!name) break;
                    name = name.toLowerCase();
                    var parent = node.parent();
                    var sameTagSiblings = parent.children(name);
                    if (sameTagSiblings.length > 1) {
                        var allSiblings = parent.children();
                        var index = allSiblings.index(realNode) + 1;
                        if (index > 1) {
                            name += ':nth-child(' + index + ')';
                        }
                    }
                    path = name + (path ? '>' + path : '');
                    node = parent;
                }
                return path;
            }
        });
        // Number of rage clicks
        var no_of_clicks = 3;
        //Time interval - 3 for 3 secs, 4 for secs and likewise
        var time = 2;
        var click_events = [];
        //internal variables
        var possible_click = 3;
        var radius = 100;
        function detectXClicks(count, interval) {
            var last = click_events.length - 1;
            var time_diff = (click_events[last].time.getTime() - click_events[last - count + 1].time.getTime()) / 1000;
            //returns false if it event period is longer than 5 sec
            if (time_diff > interval) return null;
            //check click distance
            var max_distance = 0;
            for (i = last - count + 1; i < last; i++) {
                for (j = i + 1; j <= last; j++) {
                    var distance = Math.round(Math.sqrt(Math.pow(click_events[i].event.clientX - click_events[j].event.clientX, 2) + Math.pow(click_events[i].event.clientY - click_events[j].event.clientY, 2)));
                    if (distance > max_distance) max_distance = distance;
                    if (distance > radius) return null;
                }
            }
            var result = {
                count: count,
                max_distance: max_distance,
                time_diff: time_diff
            }
            return result;
        }
        function removeUsedClickPoints(count) {
            click_events.splice(click_events.length - count, count);
        }
        $("body").click(function(event) {
            click_events.push({
                event: event,
                time: new Date()
            });
            //remain only required number of click events and remove left of them.
            if (click_events.length > possible_click) {
                click_events.splice(0, click_events.length - possible_click);
            }
            //detect 3 click in 5 sec
            if (click_events.length >= 3) {
                var result = detectXClicks(no_of_clicks, time);
                if (result != null) {
                    var path = $(event.target).getPath();
                    //console.log('Rage Click: ' + JSON.stringify(result));
                    dataLayer.push({
                        'event': 'rage_click',
                        'rc_element': path,
                        'rc_count': result.count,
                        'rc_max_distance': result.max_distance,
                        'rc_time_diff': result.time_diff
                    });
                    removeUsedClickPoints(3);
                }
            }
        });
    });
}

Step 2 – Create a new Trigger

The snippet from the previous step is pushing the rage_click event into the data layer every time a rage click is detected. Now we need to create a trigger that listens to that event.

Step 3 – Send an event to Google Analytics

Awesome, now let’s create a new tag that is responsible for sending the custom event to Google Analytics.

You can play around with the event parameters, especially with the event label to best match your needs but the recommended values are as seen in the screenshot above.

Before publishing your changes, don’t forget to enter the preview mode in Google Tag Manager and test the setup yourself. Go ahead and do a few rage clicks and see if an event is getting sent to Google Analytics. Also, check the event parameters and see if they match your expectations.

Step 4 – Using this data

Okay, so you have configured everything correctly, tested it and it works fine. Now, how do you put this data into use?

First, go into Google Analytics and find your rage click events. If you did as described in this post, you find them by clicking on Overview under Behavior –> Events, then find the “Rage Click” event category.

Click on the “Rage Click” event category and choose Event Action to see the elements on which the rage clicks happened.

Since elements with the same selectors might be on different pages, it might be helpful to apply a secondary dimension “Page”.

Now, all you need to do is to pick one element, copy its selector and go to the page seen in the secondary dimension.

Next, open the Javascript console of your browser (press F12) and paste your selector in the following format:

jQuery('YOUR>SELECTOR>HERE');

Hover over the line that was returned in the console and it should highlight the element in the DOM.

If you prefer element class over the selector, you can change the event action in GTM from the rc_element to GTM’s built-in Click Class variable.

Great! That’s it, should you need any help with tracking rage clicks using Google Tag Manager and Google Analytics, post your questions in the comments below.

Should you need further help with your analytics setup, take a look at our services.

Cover photo by Emmanuel on Unsplash

The post Tracking Rage Clicks Using Google Tag Manager and Google Analytics appeared first on Reflective Data.

In case your tool of choice is Google Optimize, you should be using their official anti-flicker snippet to minimize the flicker effect.

<!-- Anti-flicker snippet (recommended) -->
<style>.async-hide { opacity: 0 !important} </style>
<script>(function(a,s,y,n,c,h,i,d,e){s.className+=' '+y;h.start=1*new Date; h.end=i=function(){s.className=s.className.replace(RegExp(' ?'+y),'')}; (a[n]=a[n]||[]).hide=h;setTimeout(function(){i();h.end=null},c);h.timeout=c; })(window,document.documentElement,'async-hide','dataLayer',4000, {'OPT_CONTAINER_ID':true});</script>

Looking at the anti-flicker snippet (previously known as page-hiding snippet), you see that the default timeout is set to 4000ms. That means that, potentially, some of your visitors may see a blank white page for 4 seconds.

Since four seconds is a long time, many companies reduce it. A common timeout I’ve seen is 2 seconds. Changing the maximum timeout to a shorter period, although, means that there can be a good amount of people to whom the snippet occasionally times out.

When the anti-flicker snippet times out, no experiments will be loaded on that page load (people will see the control for all experiments and personalizations) which can lead to a user experience where a single user is seeing different variations in single session.

What you probably want to know is the optimal timeout period for your website. To figure that out, you’d need to know how many timeouts occur with your current setup. Let’s take a look how to set that up using Google Tag Manager and Google Analytics.

Before moving forward, you should read about how the anti-flicker snippet works.

Catching the timeout

To make this a little bit easier for us, Google has added a variable in the data layer that indicates whether the snipped has timed out or not.

You can access this variable like so:

window.dataLayer.hide["GTM-xxxxxx"] where GTM-xxxxxx stands for your Optimize container ID.

To make sure it’s not too easy, though, they don’t push this variable into the data layer as you normally would. This means you can’t access it using GTM Data Layer variable nor the Custom Javascript variable.

What you need is a Custom Javascript function variable in GTM to check the value.

Here’s the function you can use

function () { 
    if ( window.dataLayer.hide ) { 
        return window.dataLayer.hide["GTM-xxxxxxx"]; 
    } 
}

And here is the same function implemented in Google Tag Manager.

PS! In the beginning, the value is always true, it turns to false if the timeout period is over and it managed to load Optimize before this happened (the snippet did not time out).

Our recommendation is to check the value on GTM’s default DOM Ready event.

Sending data to Google Analytics

This part is quite straightforward for anyone that has used GTM to send custom events to Google Analytics so let’s get right to it.

1. Create the DOM Ready trigger

In case you haven’t configured this default trigger in GTM, go ahead do it now.

2. Create a new Tag for sending the event to Google Analytics

Create a new Tag with a tag type of Google Analytics: Universal Analytics and choose track type of Event.

These are the recommended event parameters but you can use whatever works for you.

Category: optimize

Action: timeout

Label: true/false (using the Custom Javascript function variable you created before)

Non-Interaction Hit: True (because it’s not related to a user action)

That’s it. This is all you need to start tracking Optimize anti-flicker snippet timeouts using Google Tag Manager and Google Analytics.

How to use this data?

Your goal should be to find the optimal timeout duration for your website and audience. The percentage of users who experience the timeout should be kept under 10% and you might consider excluding those with timeouts from your post-analysis process using a custom segment in Google Analytics.

Any questions or ideas? As always, feel free to post them in the comments below.

The post Tracking Optimize Anti-Flicker Snippet Timeout in Google Analytics appeared first on Reflective Data.

Custom Alerts – an alerting system in Google Analytics that makes sure you get notified whenever something unusual happens with your website’s metrics.

Why use Google Analytics Custom Alerts?

Have you ever noticed a sudden drop or spike in your website traffic or some of its key metrics? Wouldn’t it be great if next time something like this happens you’d get an email informing you about what’s happened?

This is exactly where Custom Alerts can help you. You tell Google Analytics what segments and metrics it should track and the system will automatically let you know if your set threshold has been exceeded. You can configure both absolute and relative thersholds and make them equal to any value that makes the most sense in your case.

Having a custom-built business dashboard is a must and it’s great if you already check them quite often but nothing is more effective than an automated alert. You might be busy with other stuff or out of the office for a few days and not check the dashboards but your alerts will do the work for you.

Our recommendation is to keep it simple, have alerts for the most important metrics and segments. You should still have and check your dashboards for detecting long term changes and nothing can really replace the good old analyzing of the reports and working with the raw data. But, having a Custom Alert to back you up is really important, too.

How to create a new Custom Alert in Google Analytics?

Before you start setting up your Custom Alerts, here’s what you should keep in mind.

• Custom Alerts are set and work on a view level
• You can have your alerts delivered via email (and SMS in the US)

1. Planning your alerts

The first step should be planning your Custom Alerts. Think of your most important metrics, traffic sources and segments. Try to map the things you usually check first in your dashboards and reports. Write down 2 – 10 combinations that seem most important and create alerts for them.

2. Head to Custom Alerts section

In Google Analytics, go to the admin panel and look at the first column from the right (View). In the Personal Tools & Assets section, you’ll find Custom Alerts.

3. Configuring the Custom Alert

Alert Name – choose something that describes your alert in detail. A good example is

“all traffic -> Revenue -> % decreases by more than -> 10% -> previous week”

Apply to – choose your main reporting view

Period – Choose either Day (compared to previous day / the same day in the previous week / the same day in the precious year), Week (compared to the previous week) or Month (compared to previous month / the same month in the previous year).

When choosing the period, think about your traffic and how fast things usually change. If you have just a few conversions each day, maybe a 50% difference between days is usual/random.

Delivery method – make sure your alerts get delivered via email. you can add more than one email address. In the US, alerts can be delivered via SMS, too.

Alert Conditions – this is the most important part which defines how your Custom Alerts work.

First, choose the segment of visitors that the alert should track. A few common ones are desktop/mobile, organic, direct etc.

Next, choose the metric in focus. You can choose from whatever you like but some common ones are sessions, bounce rate, goal conversion rate, e-commerce conversion rate etc.

Finally, choose the condition. This determines how big and in which direction the change has to be in order to trigger the alert. This really depends on your traffic. For some, a 10% change is big while others see 50% changes daily.

Awesome! Now hit “Save Alert” and you’re all set. Time for your next Custom Alert.

Which Custom Alerts to choose?

Now that you know how to set up a new Custom Alert in Google Analyitcs, let us give you a few ideas about which alerts you should have.

Site usage

Sessions

all traffic -> Sessions -> % decreases by more than -> 10% -> same day in the previous week

all traffic -> Sessions -> % increases by more than -> 10% -> same day in the previous week

Bounce rate

all traffic -> Bounce Rate -> % decreases by more than -> 10% -> same day in the previous week

all traffic -> Bounce Rate -> % increases by more than -> 10% -> same day in the previous week

Goals

Goal X Conversion rate

all traffic -> Goal 1 – Conversion Rate -> % decreases by more than -> 20% -> previous week

all traffic -> Goal 1 – Conversion Rate -> % increases by more than -> 20% -> previous week

E-commerce

Revenue

all traffic -> Revenue -> % decreases by more than -> 10% -> previous week

all traffic -> Revenue -> % increases by more than -> 10% -> previous week

Transactions

all traffic -> Transactions -> % decreases by more than -> 10% -> previous week

all traffic -> Transactions -> % increases by more than -> 10% -> previous week

E-commerce Conversion Rate

all traffic -> E-commerce Conversion Rate -> % decreases by more than -> 10% -> previous week

all traffic -> E-commerce Conversion Rate -> % increases by more than -> 10% -> previous week

Acquisition

Organic Traffic

Medium -> matches exactly -> organic -> Sessions -> % increases by more than -> 10% -> same day in the previous week

Medium -> matches exactly -> organic -> Sessions -> % decreases by more than -> 10% -> same day in the previous week

Direct Traffic

Source -> matches exactly -> (direct) -> Sessions -> % increases by more than -> 10% -> same day in the previous week

Source -> matches exactly -> (direct) -> Sessions -> % decreases by more than -> 10% -> same day in the previous week

Referral Traffic

Medium -> matches exactly -> referral -> Sessions -> % increases by more than -> 10% -> same day in the previous week

Medium -> matches exactly -> referral -> Sessions -> % decreases by more than -> 10% -> same day in the previous week

Congratsulations! You should now have a decent overview of how Custom Alerts work in Google Analytics and at least a few ideas for setting up alerts for your own website.

Should you have any recommendations or questions about Custom Alerts, shoot them in the comments below.

The post Creating and Using Custom Alerts in Google Analytics appeared first on Reflective Data.