In this article, we’re going to explain some of the approaches you should be taking to make sure you get the most out of your valuable data without losing any of the valuable insights. Oh, and without having to upgrade to GA360.

Common problems with GA4 for high-traffic websites and apps

1. Quota Limits and Sampling

• Issue: GA4 has quotas for API calls, data collection, and BigQuery exports. For high-traffic sites, these limits can be exceeded, leading to incomplete or delayed data.
• Symptoms:
  • Delayed or partial data in reports.
  • Data sampling in the interface or when querying large datasets.
  • Only partial data is exported to BigQuery every day

2. Latency in Data Processing

• Issue: High traffic volumes can lead to delays in GA4 processing, resulting in longer wait times for data to appear in reports.
• Symptoms:
  • Real-time reports lagging or missing data.
  • Delays in standard report updates.

3. Data Thresholding

• Issue: GA4 applies data thresholding for reports that include user identifiers or demographic data, especially when dealing with large volumes of traffic.
• Symptoms:
  • Reports showing incomplete or aggregated data.
  • Warning messages about thresholding.

4. Event Limitations

• Issue: GA4 has a limit of 500 distinct event names per property. High-traffic sites often generate a large variety of events, which can exceed this limit.
• Symptoms:
  • Events not being logged or missing in reports.

5. Data Retention Settings

• Issue: GA4 defaults to a data retention period of 2 or 14 months for detailed user data, which can be insufficient for long-term analysis.
• Symptoms:
  • Historical data is no longer accessible in GA4 after the retention period.

6. Overwhelming Volume of Custom Dimensions/Parameters

• Issue: GA4 allows up to 50 custom dimensions per property. High-traffic apps often push this limit, causing issues with tracking extra custom data.
• Symptoms:
  • Missing or dropped dimensions in reports.

7. Cross-Platform and Cross-Domain Tracking

• Issue: High-traffic businesses with complex setups (e.g., mobile app + website) may face difficulties implementing seamless cross-platform tracking.
• Symptoms:
  • Duplicate or fragmented user sessions across platforms.

8. Debugging and Testing Challenges

• Issue: High traffic can make it difficult to test changes without impacting production data.
• Symptoms:
  • Errors in tracking configurations affecting a large user base.

9. Increased Cost for BigQuery Analysis

• Issue: Exporting large volumes of data to BigQuery can result in significant costs, especially when running frequent or complex queries.
• Symptoms:
  • Unexpectedly high cloud bills for BigQuery.

10. User Privacy and Compliance Challenges

• Issue: High-traffic businesses are more likely to face scrutiny regarding GDPR, CCPA, and other privacy laws.
• Symptoms:
  • Compliance risks due to insufficient data anonymization or consent management.

If any of the issues mentioned above sounds familiar, continue reading as there are solutions to all of them.

How to tackle common issues with GA4 for high-traffic websites and apps

As companies start hitting the limits in the free version of GA4, oftentimes they consider upgrading to GA360. True, it will solve some of the issues like offering a higher number of daily exported events to BigQuery, more custom event parameters, dimensions etc. but eventually, you may end up hitting those, too. Besides, upgrading to 360 doesn’t solve some of the issues around GDPR, and data sampling can still be an issue. Not to mention the cost…

The solution we are discussing in this article is known as Parallel Tracking.

In short, here’s how Parallel Tracking for GA4 works.

1. Tracking Code Adjustment

A minor update to the Google Analytics tracking code is necessary to enable streaming all hits to Reflective Data’s endpoint. This approach, called Parallel Tracking, is compatible with all types of GA4 implementations, including GTM, server-side setups, gtag.js, and third-party applications.

2. Event Processing

The Data Processing Engine captures and processes all events in the same manner as Google Analytics. Designed for nearly unlimited scalability, it avoids the data processing restrictions found in GA4. All operations are hosted on Google Cloud, with the flexibility to choose your preferred region.

3. Data Storage in Your Warehouse

By default, data is stored in Google BigQuery, though we support other data warehouses such as AWS, Azure, and Snowflake. All data is fully processed and ready for reporting within seconds.

Reflective Data does not store any events or other data on its servers at any time.

How does Parallel Tracking solve common limitations in GA4

1. Quota Limits and Sampling

• Issue: GA4 has quotas for API calls, data collection, and BigQuery exports.
• Solution:
  • Parallel Tracking for GA4 has no limits on the BigQuery exports. We know sites that are exporting well over 10M events per day without losing any events.

2. Latency in Data Processing

• Issue: High traffic volumes can lead to delays in GA4 processing, resulting in longer wait times for data to appear in reports.
• Solution:
  • With Parallel Tracking, you can have GA4 events processed and stored in your data warehouse within seconds.

3. Data Thresholding

• Issue: GA4 applies data thresholding for reports that include user identifiers or demographic data, especially when dealing with large volumes of traffic.
• Solution:
  • Parallel Tracking for GA4 allows you to work with 100% of the data. No limits on data cardinality or event counts.

4. Event Limitations

• Issue: GA4 has a limit of 500 distinct event names per property. High-traffic sites often generate a large variety of events, which can exceed this limit.
• Solution:
  • With Parallel Tracking, there are no limits to distinct event names. Create, trigger and send as many unique events as necessary for the use case.

5. Data Retention Settings

• Issue: GA4 defaults to a data retention period of 2 or 14 months for detailed user data, which can be insufficient for long-term analysis.
• Solution:
  • Using Parallel Tracking to send GA4 data into a data warehouse of your choice (BigQuery, S3, Snowflake, Redshift etc.) ensures you have full control and ownership of your data.

6. Overwhelming Volume of Custom Dimensions/Parameters

• Issue: GA4 allows up to 50 custom dimensions per property. High-traffic apps often push this limit, causing issues with tracking extra custom data.
• Solution:
  • Parallel Tracking doesn’t set any limits on the number of custom dimensions, metrics or event parameters that you are allowed to collect.

7. Cross-Platform and Cross-Domain Tracking

• Issue: High-traffic businesses with complex setups (e.g., mobile app + website) may face difficulties implementing seamless cross-platform tracking.
• Solution:
  • With Parallel Tracking for GA4 comes the ultimate flexibility around distinguishing and joining data from various sources. This includes web domains (including subdomains), mobile apps, web apps, server applications and more.

8. Debugging and Testing Challenges

• Issue: High traffic can make it difficult to test changes without impacting production data.
• Solution:
  • Parallel Tracking comes with a robust environment for staging and testing all changes before pushing them into production. Never lose valuable data because of a broken tracking system again.

9. Increased Cost for BigQuery Analysis

• Issue: Exporting large volumes of data to BigQuery can result in significant costs, especially when running frequent or complex queries.
• Solution:
  • Parallel Tracking gives you full control over your data pipeline. This means you can adjust the settings to collect only what you need to avoid overpaying for the data warehouse vendor.

10. User Privacy and Compliance Challenges

• Issue: High-traffic businesses are more likely to face scrutiny regarding GDPR, CCPA, and other privacy laws.
• Solution:
  • With Parallel Tracking you can choose in which region your data is processed and stored. For GDPR compliance, for example, several sites keep all their data within the EU (this includes collection, processing and long-term storage).

As you can see, Parallel Tracking is the best companion to your GA4 implementation on a high-traffic website. Not only is it multiple times more affordable than its alternatives (including GA360) but it’s the only platform that provides a solution to all of the common issues.

If you want to learn more about GA4 Parallel Tracking, please schedule a free consultation session with one of Reflective Data’s account managers or data engineers (depending on your needs).

The post Optimal GA4 Setup For High-Traffic Websites and Apps in 2025 appeared first on Reflective Data.

Recently, several European countries, including Austria, France, Italy, and Denmark have declared the use of Google Analytics illegal. That is, even if you asked for the consent of your website visitors.

The debate over Google Analytics (including GA4) being illegal in certain countries or not continues but considering the fines can reach millions, it is certainly better to be cautious.

Here at Reflective Data, we work with a lot of businesses that rely heavily on Google Analytics data to make marketing and product-related decisions. Simply removing GA4 from the site for all visitors coming from a country that has declared it illegal isn’t a choice they want to make. Paying a huge fine is even less so.

Therefore, we’ve worked hard on a workaround that would yield a dataset and user experience that closely resembles what you get with GA4 itself. And we succeeded!

Here’s how we keep GA4 GDPR compliant

In short, for traffic coming from the EU, we skip the the step where GA4 tracker sends data to GA4 servers. Instead, we process this data within the EU (just like GA4 does on their servers) and send it straight into a BigQuery dataset located within the EU.

In the background, this solution uses our Parallel Tracking product. The main use case, besides being able to make GA4 GDPR compliant, is to circumvent the 1 million events BigQuery export limit seen in the free version of GA4. You can read more about this use case in another article. Hundreds of businesses across the world are using Parallel Tracking to track billions of events every month.

Getting started is really simple. Below is a quick step-by-step guide for making your GA4 implementation fully GDPR compliant.

1. Get a free consultation with Reflective Data’s expert

Get in touch with Reflective Data to schedule a free consultation session with one of our data experts. This is the best way to determine if Parallel Tracking is the right solution for you.

2. Implement Parallel Tracking

The exact setup depends on how GA4 is implemented on your site but in most cases, the setup involves adding one new tag in Google Tag Manager or a small tracking code modification to your gtag.js snippet.

Reflective Data offers free implementation support with all plans. Most clients have the system up and running in less than one week.

No need to change your existing GA4 tags, all events will continue working as expected. No double tagging is required either, Parallel Tracking uses your existing GA4 tags.

3. See data flowing into your BigQuery dataset

As mentioned above, Parallel Tracking provides a solution that skips sending data from the EU into the US. Instead, it will process this data within the EU and stores it in your BigQuery dataset located in the EU. This ensures full GDPR compliance.

***

More and more countries are introducing and starting to enforce their data privacy regulations, including GDPR in the EU. This means that anyone doing business or having website visitors from those countries needs to comply with the local rules to avoid fines.

To make being GDPR compliant as painless as possible, Reflective Data has built a solution that lets you continue using your existing GA4 setup while not sending data from the EU into the US.

Learn more and schedule a free consultation session.

Feel free to ask any questions in the comments below.

The post How to Make Your Google Analytics 4 [GA4] Implementation GDPR Compliant? appeared first on Reflective Data.

While Google Analytics 4 (GA4) comes with the native BigQuery export feature available in the free version and some of the other quotas aren’t as tight anymore (compared to UA), GA4 still has a fair share of limitations we need to account for.

The limit I’m covering in this post is set on the BigQuery export. More specifically, the GA4 to BigQuery native export feature has a limit of 1 million hits per day.

From the docs.

Standard properties have a daily BigQuery Export limit of 1 million events. If your property consistently exceeds the export limit, the daily BigQuery export will be paused and previous days’ exports will not be reprocessed.

Considering the BigQuery export will be a core element for most power users, this can become a serious problem for any business with a good amount of traffic. Again, this limit is not on users or sessions but on hits. A single visitor can generate hundreds or even thousands of hits, depending on your business type and GA4 configuration.

Ways to overcome the 1 million hits limit in GA4

In short, you’ll have three paths to choose from.

• Collect less data
• Upgrade to GA360
• Use Parallel Tracking

Below is a more detailed overview of each of these solutions to help you find the best one for your use case.

Reduce the number of hits collected with GA4

There are two parts to this one. First one being that you simply take a second look at the list of events you’re currently collecting and consider skipping some of them. Could be scroll events or other automatically triggered events like product impressions.

The second part is that you can actually filter in/out the hits that are sent from GA4 to BigQuery. So, perhaps you could send only the most critical data to BigQuery.

Now, neither of those solutions is really a good one because no one should be forced to give up on some of the useful data and insights because of some arbitrary hit limit. So, if you’re serious about your data game, check out the other two options.

Upgrade to the GA360 version of GA4 and get more quota

Understanding that GA360 doesn’t fit every budget and a simple hit limit shouldn’t be enough to convince you into spending this kind of money but if you’re considering GA360, then know that it also comes with a much more generous BigQuery export quota.

The official documentation doesn’t say the exact number but they claim the daily limit to be “billions of events”. That’s probably enough for most use cases.

Use Parallel Tracking for a limitless BigQuery export

If a compromise of collecting less data isn’t want you want to make and GA360 is too expensive then Parallel Tracking is your best option.

Depending on your GA4 setup, the implementation is very simple. Either a small update to your tracking code or GTM tag and unlimited data pipeline is up and running.

As a bonus, you can toggle on an option that processes your data into sessions (just like in Universal Analytics – together with all the metrics and dimensions from UA) in a separate BigQuery table. How awesome is that?

Learn more about Parallel Tracking for GA4.

Wrapping up

If you’re serious about analytics, BigQuery export is a must-have. Unfortunately (but expected) the free version of GA4 comes with an export limit of 1 million hits.

In case you’re already exceeding this or are getting close, now is the time to look for ways to overcome this limit. My recommendation is that you take a closer look at the Parallel Tracking, get in touch with the team and see if that’s the solution you should implement.

As always, feel free to post your thoughts and questions in the comments below.

The post How to Avoid Google Analytics 4 [GA4] BigQuery Export Quota Limit of 1 Million Hits appeared first on Reflective Data.

While Google Analytics is such a great tool and I’ve been using it for years myself, some of our clients have run into problems and limitations that made them search for other options. In this article, I’m going to discuss some of the reasons why you might need an alternative (or addition) to Google Analytics and what most of our clients have found to be the best option for them (and why).

  This article will prepare you for July 1, 2023 when Google will stop processing Universal Analytics data.

Before I get any deeper into the topic, this solution may not be ideal for every business but if the reason you may be considering another analytics tool is in the list below, it’s likely the one you should consider.

• Forced transfer to GA4
• Sampling
• Data collection limits (10M hits/mo,  20 dimensions & 20 goals)
• Sending data to the US
• Forced data aggregation
• No access to raw hit-level data
• API limitations

So, if you found the reason you’re looking for an alternative to Google Analytics in the l, the solution we’re going to discuss now is probably a good fit for you.

Parallel Tracking as an alternative to Google Analytics

Before I give you the reasoning why Parallel Tracking is a very good alternative to Google Analytics, let me give you a quick overview of how it works.

Parallel Tracking is a data pipeline that’s built based on Google Analytics Universal Analytics (UA) and follows the exact same data structure and processing logic. It can do so while letting you choose whether to send any data to Google Analytics itself or not. Parallel Tracking will send both raw and processed data straight into your data warehouse. BigQuery being the default storage solution.

What makes it stand out from most other analytics tools is that it uses literally 100% the same tracking setup as Google Analytics UA. This means you can use analytics.js, GTM, Segment, Measurement Protocol and almost all other methods for implementing it while using the syntax you’re already familiar with.

In the following sections, we’re going to take a closer look at the core areas of the Parallel Tracking solution. In each one, I’m going to compare it to Google Analytics and explain why it’s probably the best alternative out there.

Implementation

If you’re using Google Analytics today, getting Parallel Tracking implemented will normally take 2-8 hours. All plans come with full implementation support included. Parallel Tracking works out of the box with most Google Analytics implementations, including the following.

• analytics.js (Google Analytics snippet in your source code)
• Google Tag Manager
• Measurement Protocol (Google Analytics server-side)
• Segment.com
• Shopify
• Littledata
• WooCommerce and most other apps and plugins

  PS! All of those integrations will work after July 1, 2023 when Google will stop processing Universal Analytics data.

In case it’s a fresh setup, the integration time will depend a lot on the complexity of your site. As it’s using the exact same tracking syntax as Google Analytics, the setup time will be the same as it would take to install Google Analytics from scratch.

Considering this, Parallel Tracking is by far the quickest, least painful and most effective tool to replace Google Analytics with.

Reporting

While Google Analytics Universal Analytics has a familiar and rather intuitive user interface for quick reporting, most analytics platforms don’t. This includes the new Google Analytics 4 which will be the only version after July 1, 2023 when Universal Analytics reaches its end of life.

As Parallel Tracking will send both raw and processed data into your own data warehouse (i.e. BigQuery), you can choose the reporting layer of your choice. For most users, Google Data Studio (free) should be enough. More advanced users are successfully connecting it with tools like Looker, Tableau and Power BI.

By the time Universal Analytics sunsets, there will be a set of getting-started reports, templates and potentially even a full user interface to ease your reporting with Parallel Tracking.

This means that Parallel Tracking is on par with most other analytics tools when it comes to reporting and offers you virtually limitless flexibility by connecting natively with most BI and dataviz platforms. Plus, Parallel Tracking is probably the only one that gives you sessions and related metrics like bounce rate out-of-the-box!

Integrations

Google Analytics is known for its wide range of integrations. Starting from Google Ads and ending with most of the CMSes and A/B testing platforms on the market.

Since Parallel Tracking uses the same tracking system as Google Analytics, it’ll natively support most of those integrations! This includes Google Ads (even cost data).

The fact that you’ll have full control over your data in BigQuery means you can integrate with almost any data source out there. Ads platforms, CRMs, A/B testing, e-commerce, custom backends – you name it. As a bonus, since Parallel Tracking is just one of the hundreds of data connectors Reflective Data offers, you can get it all from one place.

Considering the above, Parallel Tracking is on par or even exceeds Google Analytics in terms of integrations and easily beats almost any other analytics tool on the market.

Cost

Google Analytics is free for most users but the free plan comes with a set of serious limitations (discussed further in the next section). The premium, GA360 would cost you at least $150k a year which makes it one of the most expensive offerings in that segment.

Parallel Tracking’s pricing is based on usage – the number of hits collected per month to be more specific. The number of integrations and a method of implementation can be pricing factors, too. So, I recommend you get in touch for the exact quote customized for your business and use case. Based on my experience, most clients will be happy with plans in the range between $375 to $950 per month.

Comparing the pricing with other analytics platforms, Parallel Tracking should be quite affordable for most businesses and it’s a lot cheaper than GA360. Even if you’re collecting hundreds of millions of hits each month.

Limitations

Google Analytics, especially the free version, is known for its wide range of limitations.

Here’s a table comparing Google Analytics against Parallel Tracking in terms of quotas and other limitations.

Reliability

Google Analytics tends to be very reliable. I can recall some bugs and losses in data but it is very rare.

Parallel Tracking is built based on the same tracking system as Google Analytics and runs on Google Cloud – making it one of the closest competitors in terms of reliability and scalability. Fortune 500 companies have put their trust in Parallel Tracking to collect and process hundreds of millions of hits per day.

Should something go wrong in data processing (as has happened with Google Analytics a few times), Parallel Tracking can promptly re-process the raw data, minimizing the risk of losing any data for good.

Comparing this with other tools, some of which are self-hosted and most of them using custom-built tracking syntax, makes Parallel Tracking stand out for sure.

Support

While Google Analytics doesn’t offer support themselves, it does have a huge community making it easy to find answers to almost any question you might have.

Since Parallel Tracking is based on the same tracking syntax, data is processed and stored just like in Google Analytics, you can still benefit from that massive community – most answers will be applicable for both.

Besides community, Parallel Tracking comes with technical support available with all plans. You don’t have to worry about the implementation, custom configuration and maintenance. Just contact support and you’re almost guaranteed to get a solution to whatever you were struggling with.

While there are other tools with technical support available, many of them don’t offer it with their lower-end plans. Certainly, no other tool benefits from Google Analytics’ massive community like Parallel Tracking, besides Google Analytics itself.

Privacy and EU laws

Since GDPR and other data protection and privacy laws were introduced, Google Analytics and several other analytics platforms have struggled to meet the requirements. Some EU countries have gone as far as saying using Google Analytics to track their citizens online is illegal.

With Parallel Tracking, you are in control of almost everything that could be related to privacy and data protection. This includes saying where data is processed and stored (within EU, in the US, on-premise).

This flexibility goes both ways, meaning that if you’ve asked your visitors’ permission, you can use Parallel Tracking to collect PII data like an email address or phone number, too. Something that’s not an option with Google Analytics.

Unlike almost any other analytics platform (besides open-source and self-hosted ones), Parallel Tracking does not store any data on its servers. After processing data, it is stored in your data warehouse. This means data is never shared with or sold to third parties.

Is it really the best replacement for Google Analytics?

After reading this article, I hope you have a pretty good understanding of how Parallel Tracking works and why it’s likely to be one of the best alternatives to Google Analytics. Both UA and GA4.

One of the only downsides this solution has is that, compared to the free version of Google Analytics, Parallel Tracking will still cost some money. If you’re using this kind of data for making business decisions, that shouldn’t be an issue as it’s still more affordable than most of its competitors. Including GA360.

Click here for a quote and consultation on Parallel Tracking.

Further reading

We’ve written on similar topics before. Have a look at these articles to learn more about Parallel Tracking and why it’s a good alternative to Google Analytics.

• Product overview
• Hit-Level Unsampled Google Analytics to BigQuery Without 360
• Query Google Analytics Data Using SQL
• How to Avoid Google Analytics Sampling and Data Limits?
• How to Query and Analyze Google Analytics Data with BigQuery
• Case Study: Storing Google Analytics Data Within The European Union or Locally
• Measure Long-Term Metrics Like Customer Lifetime Value (LTV) Using Google Analytics
• Case Study: Sending Data From Shopify and Google Analytics to BigQuery for More Advanced Analysis

I’d be happy to answer any questions you might have about Parallel Tracking in the comments below.

The post Best Google Analytics Alternative for 2023 appeared first on Reflective Data.

In this article, we are going to explore some of the ways we can leverage Google Analytics to track churn, LTV and other really useful metrics.

Depending on the software you’re using, there may be some off-the-shelf solutions that you can install. For example, if you’re on Shopify then you can use something like Littledata to send a more accurate LTV value into a custom dimension in Google Analytics. More often than not, though, there is no good solution available or you just need more control over the setup.

One common misconception is that such long-term retention metrics are relevant for a few specific business types only. Yes, metrics like churn are vital for SaaS and subscription products but any company that gets return business should have their long-term KPIs in place. And I don’t mean simply tracking them but actually analyzing them and optimizing the business with those metrics in mind.

Acquiring a new customer is anywhere from five to 25 times more expensive than retaining an existing one. It makes sense: you don’t have to spend time and resources going out and finding a new client — you just have to keep the one you have happy.

Harvard Business Review

So, if you’ve been focusing on getting new customers and metrics like revenue or transactions, this article is just for you!

How to measure retention metrics like LTV and churn

The long-term retention metrics most relevant to you depend on the type of business you’re working with but the most common ones are customer lifetime value (LTV) and churn. Below is a list of other popular retention KPIs. Think about the ones that would be relevant for your business.

Common Customer Retention Metrics

1. Customer Churn
2. Revenue Churn
3. Existing Customer Growth Rate
4. Repeat Purchase Ratio
5. Product Return Rate
6. Days Sales Outstanding
7. Net Promoter Score
8. Time Between Purchases
9. Loyal Customer Rate
10. Customer Lifetime Value

Source

Almost all retention metrics require a proper User ID implementation. This means you’d have to identify the user over time and even if they’re using multiple devices or browsers. Luckily, in most cases, actions like completing a purchase or signing up for a subscription do involve some kind of authentication.

While it’s possible to track retention metrics with Google Analytics alone, in most cases you’d get much better (more accurate) results when combining it with some other technology. Let’s explore two of the more popular options.

Sending retention data into Google Analytics

This solution involves sending retention data into a custom dimension or a custom metric in Google Analytics.

The exact workflow depends on the software (CRM, CMS, database etc.) your site uses but the general process would look something like this.

1. Create a custom dimension in Google Analytics (should be user-scoped)
   
2. For logged-in/identified users, pull/calculate the values for the relevant retention metrics from a database or other system (CRM, CMS etc.)
   Something like this if your order data is stored in BigQuery.
   
3. Make the retention metrics available in the data layer
   
4. Use Google Tag Manager to send your retention metrics to Google Analytics, using the custom dimension or metrics slots/indices according to how you configured them in step #1

Now, having this data available in Google Analytics, you do whatever you want with it. Here are a few examples.

Using LTV in a Google Analytics custom report

LTV in the Google Analytics user explorer report

Notice the difference between LTV that Google Analytics is reporting by default ($439) and the value we see in our custom dimension ($2,016). This is because Google Analytics can’t keep a track of the user as accurately as your backend system or an e-commerce platform you’re using. The same goes with other retention metrics, getting accurate metrics requires some custom work.

The list of possible use cases for this kind of data is unlimited. For example, think about creating custom segments in Google Analytics for customers that are in the top 10% in terms of LTV and see what differentiates them from the rest of the visitors. Besides making more/larger purchases, of course. Things like their traffic source, what pages they landed on, what A/B test variants they saw etc. can be quite insightful.

Talking about ways you can analyze your data and the insights it will give you. Let’s take measuring retention metrics to a whole new level by sending data into a data warehouse.

Storing data in a data warehouse

If you’re just getting started with retention metrics and you still mostly optimize for generic metrics like leads, total transactions and total revenue, then you’ll still be better off with having them in Google Analytics. Compared to not having them at all, that is. If you’re serious about analyzing and optimizing for retention and customer lifetime value then you need a data warehouse.

Here’s a quick step-by-step guide that will lead you in the right direction.

1. Send all Google Analytics data into a data warehouse (i.e. BigQuery). Tools using the Reporting API (most of them) can get you started but for true unsampled hit-level data you need something like Parallel Tracking.
2. Send, pull, push data from other relevant sources into your data warehouse. This should include your database, CRM, CMS, marketing tools, ads platforms, customer support, live chat and every other tool that has data about your customers and their interactions with your brand. Self-service tools like Stitch will get you started but we’d recommend more flexible managed solutions.
3. Solution for accessing data stored in your data warehouse. You’d need something (could be separate tools) that can handle ad-hoc queries, dashboarding, automated reports, and building data models. Tools like Google Data Studio will get you started. Something like Looker or Tableau would be better. Our recommendation is to go with a managed service that will put together the best set of tools for you and configure the rest as well.

If having retention metrics in Google Analytics enabled you to do all sorts of useful new reports and analysis then your options with the setup above are truly limitless.

Having a proper data warehouse will be your competitive advantage. Not only does it give you an option to get a very good overview of the current status of your business and your customers, but it will also allow for truly optimizing the user experience and user journey. Leading to a better user experience and improved retention metrics. Remember, acquiring a new customer is anywhere from five to 25 times more expensive than retaining an existing one!

One way to describe the usefulness of a data warehouse is by giving you a few sample questions. Questions that would be very difficult to answer without a data warehouse.

• Purchases from which traffic channels are most likely to be refunded at some point in the future? Might lead to revising your marketing budget.
• Which traffic sources have the highest retention/LTV?
• What is the correlation between subscription value ($) and churn rate?
• What is the long-term impact of your campaigns or A/B experiments? Do quick wins lead to higher churn or lower LTV?
• Does data from different sources add up? Maybe Google Analytics is missing some transactions that are in Shopify or perhaps some of them are duplicates?

Here’s an example of the last one on the list above.

As you can see, Google Analytics is missing a good amount of transactions and this requires further investigation. Definitely something you should include in your Google Analytics dashbaord.

This was just a short list of ideas to get you thinking about what is possible with a proper data warehouse. You can trust me when I say that I’ve seen companies become truly data-driven after they’ve implemented a tailor-made data warehouse and started digging for insights they couldn’t before.

Working with automatically recurring events

It is important to keep in mind that some retention metrics can change without the user themselves taking any action. You need to make sure that those cases are being tracked and taken into account. Here are a few examples.

• Recurring orders/payments
• Subscription expirations
• Payment method expirations
• Orders being changed/cancelled (i.e. due to missing item)

If your data warehouse was configured properly, you should have this data already available. Just make sure to include it in your analysis and reports.

In case you don’t have a data warehouse and you’re trying to solve this with Google Analytics alone then you need to use the Measurement Protocol. Some of the more common subscription platforms like ReCharge for Shopify have this built-in or solvable with some 3-rd party solutions but quite often custom development is required. In which case, you should think about implementing a data warehouse instead.

Conclusion

If you’re in a business where customers are expected to generate value more than once (repeat purchase, subscription etc.) then you need to start focusing on your retention metrics.

Google Analytics can get you started with the basic metrics and limited accuracy. A much better setup would be Google Analytics combined with Parallel Tracking and if you’re serious about optimizing for those metrics then you need a custom-built data warehouse where all marketing data is pulled together.

Feel free to post your ideas and questions in the comments below. If you’d like to get some consultation and discuss your ideas further, get in touch with us.

The post Measure Long-Term Metrics Like Customer Lifetime Value (LTV) Using Google Analytics appeared first on Reflective Data.

With the help from Reflective Data and using their custom data pipeline for Google Analytics, we were able to avoid sending data to Google Analytics, store everything within the EU and still benefit from the rich and actionable dataset you get with Google Analytics.

Aaron, Chief Data Officer

Everything mentioned in this article applies to both Google Analytics Universal Analytics and GA4, including the built solutions.

Data protection and privacy rules are getting tougher all over the world. This is especially true for the European Union and even more so for some specific industries. Including finance, medical and others that handle sensitive information about their users.

While Google Analytics has been making some improvements in the privacy area and is GDPR compliant, this is not enough for many businesses and industries.

At Reflective Data, we’re often working with companies that are under close monitoring of their regulators. To help them out, we’ve built custom solutions that allow storing Google Analytics data within the European Union or sometimes even completely locally, on the servers of our client.

The challenge

Do we want it or not, the regulations for collecting, processing and storing data are getting stiffer each year. European Union, with rules like GDPR, is a trendsetter in data protection.

While it’s good from a customers’ perspective, following all those rules and regulations can be quite a challenge for businesses that have to follow them. Especially, as new rules are introduced and the penalties for not following them can reach millions.

More often than not, the regulations are being launched faster than companies can adopt. Even if the company themselves are doing everything right, it’s likely that some third-party software they’re using is lacking behind.

At Reflective Data, we have been contacted by numerous companies that, for a reason or another, aren’t able to use Google Analytics but would still require a similar dataset and comparable functionality. Let’s be honest, Google Analytics and its Universal Analytics data model is still one of the best ones available for marketing attribution and website user behavior analysis.

Yes, Google Analytics 4 will be the future of tracking but it’s nowhere near being ready for production for most businesses due to lacking several critical features.

The solution

One of our most popular tools is the Parallel Tracking for Google Analytics. It’s a system that allows sending raw hit-level Google Analytics data into a data warehouse. Most commonly, our clients choose this solution for the following reasons.

• They hit the 10M/hits/mo limit
• Problems with data sampling
• Looking for alternatives for Google Analytics 360
• Need access to raw hit-level data

These are all nice features that our clients love but there’s one more – Parallel Tracking can work without sending data to Google Analytics at all. This is thanks to how our technology was built. Instead of pulling data from Google Analytics using the Reporting API (as many solutions do), we are duplicating all hits that are sent to Google’s data processing endpoint and sending them to our engine as well. This means, with a simple tracking code modification, we can disable hits being sent to Google Analytics.

Our data processing engine works very similarly to what’s happening in Google Analytics. Hits are processed and enriched using the same methodology and sessions are processed just like in Google Analytics.

Ultimately, this means you’ll get the best from both worlds. The rich and actionable dataset of Google Analytics and full ownership of your data. So, whatever regulations apply to how you can store this kind of data, now you have a solution to comply with them.

Everything mentioned in this article applies to both Google Analytics Universal Analytics and GA4, including the built solutions.

Conclusion

There are many reasons why you’d want to use Google Analytics as your analytics platform. Including the rich data model that works for both marketing attribution and user behavior analysis, (almost) unlimited amount of resources, great community, ease of finding people that can work with this model and more.

If, for some reason, you can’t use Google Analytics. Could be because they’re storing data outside of the EU or that you need to store everything locally. Then we have a solution for you!

For details and contact information, check out the product page for Parallel Tracking.

Ending with another quote from a customer here.

Before joining the financial institution that I’m currently working for, I had been a marketing director for various e-commerce companies. This meant that I was really used to tools like Google Analytics. Now, working in finance, I learned that using Google Analytics was not an option due to several regulations and we had to store all data locally on our own servers.

We had some kind of an in-house solution but that was nowhere near the capabilites and functionality I was used to. I was really lucky to learn about Reflective Data and their Parallel Tracking solution! They were really helpful, worked closely with our team to learn about our needs and finally built a solution that exceeded our expectations. 10/10 would recommend to anyone that for some reason isn’t allowed to use Google Analytics.

Michelle, Director of Online Marketing, Frankfurt

It’s feedback like this that makes us love the work we do even more! Get in touch and learn how we can help you, too.

For more case studies, see here.

The post Case Study: Storing Google Analytics Data Within The European Union or Locally appeared first on Reflective Data.

About the client

Our client is a successful e-commerce business, headquartered in the UK.

Their focus is on food supplements, nutritional products, and diet plans.

Depending on a season, their monthly online revenue is 1-1.5 million GBP (1.3 – 2 million USD). This comes from around 14k orders every month.

Due to NDAs, we are not allowed to disclose the name of the client. Everything else in this case study remains unchanged.

The challenge

When the company in the focus of this case study first contacted us, their main concerns were about properly tracking customer lifetime value (LTV) and churn. Being in the business of food supplements where, naturally, a big chunk of revenue is coming through subscriptions (automatically recurring orders), keeping track of these metrics is extremely important.

Tech stack

The main tech tool stack for this client was a fairly standard one for an e-commerce business of this size. Shopify Plus for website and order processing, ReCharge for handling subscriptions, Google Analytics for general reporting, and ShipStation for managing shipments.

***

While Shopify can provide you with reports for both LTV and churn (via ReCharge analytics), it is rather limited when it comes to attributing those numbers to your marketing and other efforts like CRO or A/B testing.

The client pointed out three main areas where they need a more detailed look into LTV and churn-related metrics:

• Ad Spend (mostly Google Ads and Facebook)
• E-mail campaigns
• CRO and A/B testing

The solution

After having several meetings with various teams from the client’s side, we had a quite good understanding of what kind of reports they’re trying to build and what kind of data it requires.

In order to accommodate all of their needs, we had to involve data from the following sources:

• Shopify
• ReCharge
• Google Analytics
• Google Ads
• Facebook Ads
• ShipStation
• Google Optimize

As our solution involved data from more than three sources, it was quite obvious that setting up a data warehouse is required. The client decided to go with our default recommendation for data warehouse software — BigQuery.

Getting data in BigQuery

As LTV and churn had always been such essential metrics for the client, they had already tried solving this in-house. Their existing tool stack consisted of Stitch, Supermetrics, MySQL database and Google Sheets. While this could’ve worked in theory, the reality was a really slow system that constantly failed and produced inaccurate numbers.

Our solution involved using Parallel Tracking technology for Google Analytics and a custom data pipeline for the rest of the sources. This enabled us to modify the ETL for maximum efficiency while meeting all the data requirements.

Building the reports

Since our client was already familiar with the rest of Google’s Marketing Suite and it works really well with BigQuery, using Data Studio for building the reports was an easy choice. Besides being a free (!) tool, by leveraging the BigQuery BI Engine, Data Studio can provide you with the best performance as well!

After understanding the reports that would be used most frequently, we built a set of 2 main dashboards:

• LTV per source/medium/country/product/campaign/age
• Churn per source/medium/country/product/campaign/age

These dashboards provide the client with a good overview of their key metrics and how they correlate with some of the more common dimensions/attributes. This is a good stepping stone for a deeper analysis and digging for insights.

Luckily, the client’s marketing team already had a few people that were comfortable working with SQL, Jupyter Notebooks etc. This meant that by pointing them in the right direction and providing a few guidelines, they had everything needed for digging deeper into data stored in BigQuery.

Besides dashboards and enabling ad-hoc analysis, a few teams also wished for a periodic automated report in their inbox. This was also something we were happy to set up using some custom software on Google Cloud.

Conclusion

While this project started as a rather standard one for us, it turned out to include several custom parts that made it fun for us and enabled maximum effectiveness for the client.

What better way to take it all together than a few words from the team we worked with on the client’s side.

We turned to Reflective Data with a rather concrete goal in mind — to fix our LTV and churn tracking and reporting. Being in the business of  food supplements, a large portion of our revenues come from recurring orders. This makes keeping track of those metrics extremely important for us. Not only was Reflective Data able to set up a proper tracking system, they helped us figure out the dashboards, email reports and helped our analysts on SQL analysis! Couldn’t be happier about this!

Nothing’s better than such warm words from a happy client!

If you, too, need help with your analytics & data setup, get in touch with us for a free consultation session. I’m sure we can find a solution that suits both of us.

The post Case Study: Sending Data From Shopify and Google Analytics to BigQuery for More Advanced Analysis appeared first on Reflective Data.

About the client

Our client, in this case study, is a large global e-commerce website, headquartered in the US.

Their main focus is on men’s clothing and they’re specialising in dress shirts, as well as ties, suits, casualwear, shoes and accessories.

Depending on a season, their monthly online revenue is between 10 and 15 million USD. This comes from around 45k monthly transactions.

Due to NDAs, we are not allowed to disclose the name of the client. Everything else in this case study remains unchanged.

The challenge

Working with skewed data can be worse than having no data at all. This is why we’re always promoting all sorts of analytics audits and making sure all data sources agree with each other. At the very least, you should know why the numbers in different tools don’t match (i.e. analytics doesn’t include offline sales but backend does).

Our client in this case study contacted us with a quite specific problem. They were running a decent CRO program with 4-6 A/B experiments running every month. The problem they had with the program, though, was that the numbers they saw in their testing tool Optimizely, Google Analytics and backend didn’t match. In fact, there was a ~35% discrepancy overall.

While it is almost impossible to reach 100% accuracy with Javascript based tools due to ad-blockers, cookie deletion etc. and some tools have different logic for calculation metrics like conversions, sessions, users — we knew that this discrepancy is way off the limits and immediate action is needed.

This kind of discrepancy was even more concerning as the client was actively running their CRO program. It wasn’t rare for them to see test results flip based on where they got their numbers (Google Analytics vs Optimizely). This put their entire CRO effort under question internally and the management wasn’t far from shutting it down. The results just weren’t trustworthy enough.

The solution

Even though the client described some rather clear symptoms, it is necessary to always start with a proper diagnosis of the problem. This way we can be sure that there indeed is a discrepancy between all these tools – in some cases, the problem can be in how you understand/read the numbers. Another goal for the diagnosis was to understand the size/reach of the problem.

Diagnosis

The very first thing we did to diagnose and measure the discrepancy was to get access to all data sources. That included the testing tool (Optimizely), Google Analytics and the client’s backend numbers. Next, we made sure the metrics are comparable (i.e. all the transactions we see in the backend are indeed supposed to show up in Google Analytics as well). We then put those numbers side-by-side in a spreadsheet and grouped them by date.

After analyzing the daily numbers, we came to the conclusion that there is indeed an average of 35% discrepancy between Optimizely and Google Analytics (less transactions in the testing tool). The discrepancy between Google Analytics and backend, on the other hand, was rather small — just below 3%. This can be considered OK for a Javascript-based analytics tool.

The discrepancy between Google Analytics and Optimizely meant we had to dig deeper on that side. Unfortunately, Optimizely doesn’t give access to the raw underlying transaction ID level data by default. Luckily, though, we were able to get this data from their support team. With Google Analytics, exporting transaction-level data was doable with a custom report and CSV export. For the Google Analytics dataset, we included dimensions that would give us some extra context — payment gateway, products, device category, country etc.

After pulling transaction-ID-level data from Optimizely and Google Analytics in spreadsheets, we wrote a quick function that would mark the transactions in Google Analytics that didn’t show up in Optimizely. Then it was time to start looking for some patterns. Is there something in common for all missing transactions?

This is how one version of this spreadsheet looked like. This was to highlight transactions in the backend that didn’t show up in Google Analytics.

Implementing the fixes

Since the website and payment processing system for this website were rather complex, it turned out there are more than one such patterns. They were all related to the use of various payment gateways, device categories and local sub-sites for specific countries.

In order to come up with a proper solution, we had to dig even deeper into how transaction tracking was set up for both Google Analytics and Optimizely. Luckily, we were able to detect the shortcomings quite fast and fixing them wasn’t rocket science either. After a few rounds of testing and a couple of A/A tests run in Optimizely, we were able to get the discrepancy number from 35% down to 2.5%.

The main reason for this mismatch was that while Google Tag Manager was in use to unify transaction tracking for Google Analytics, tracking snippets for Optimizely were all over the place. The solution involved moving all Optimizely tracking into Google Tag Manager and use the same triggers that were being used for Google Analytics.

Going forward

Since the site has a rather complicated structure with multiple payment gateways and different pages for different countries/markets, we wanted to make sure that after implementing the fixes, the discrepancy would not start sneaking in again.

The solution here was a proper monitoring system that would get data from Google Analytics, Optimizely and the backend. Since the client was already on Google Cloud, we recommended using BigQuery as their data warehouse technology. Besides being really fast and cost-effective, BigQuery has some of the best integrations which makes implementing and using it as a marketing data warehouse much easier.

Getting data into BigQuery

For Google Analytics data, we used a technology called Parallel Tracking. This allowed us to send raw hit-level unsampled Google Analytics into BigQuery. In real-time and for much cheaper than using Google Analytics 360.

To get data from Optimizely into BigQuery, we built a custom connector using Optimizely API. This solution allowed us to pull only the information that was necessary and make sure it’s always up to date.

Since the client had a custom backend infrastructure, we built a custom connector between that and BigQuery as well.

All connectors are now running on Google Cloud.

Reporting and alerts

In order for the client to be able to check the numbers from all three sources in one place, we built a dashboard using Google Data Studio which has a native connector with BigQuery.

To detect any discrepancies as soon as possible, we also created an automated reporting system with some custom code on Google Cloud Platform. This will send an email to relevant parties every time a certain threshold is passed. So far, it has already detected a few issues the client was able to fix before larger damage was done.

Conclusion

Overall, we were quite happy with the results of this project but I believe it’s better to end with a quote from the client.

We hadn’t worked with Reflective Data before but they were recommended by our friends in the industry. We needed a quick but reliable solution to our problem. Our management was questioning the results of our CRO program and we weren’t far from getting shut down.

Team at Reflective Data was able to immediately understand our problem and jumped right into solving it. I think all together it took just around 2 weeks for them to diagnose and together with out IT team fix the problems behind this rather large discrepancy.

After great success with our first project together, we’ve continued working with them on various data-related projects around our data warehouse, reporting and dashboards.

Director of e-commerce

We are so glad about these words from our client and it’s moments like this that give meaning to our work.  By the way, their CRO program has only grown since we were able to fix the mismatch. So, in a way, we were part of saving the jobs of an entire team.

Should you have similar problems with discrepancy or other data related requirements, please contact us and I’m quite sure we can work together and find a solution.

The post Case Study: Solving The Discrepancy Between A/B Testing Tool, Google Analytics and Backend Data For a Large E-Commerce Business appeared first on Reflective Data.

In this article, originally posted on medium.com, we’re going to walk through the reasons you might want to access raw Google Analytics data in BigQuery and a few solutions that will get you there.

***

Google Analytics is, without a doubt, the most popular tool in the digital analytics segment. Besides being the best tool for tracking e-commerce, content and lead-gen websites, with some modification you can use Google Analytics for tracking SaaS and other web apps as well.

While being a really powerful and flexible tool, Google Analytics, too, has a few shortcomings. Perhaps the biggest problem every advanced user faces is the fact that there is no way to access raw hit-level data. The second major issue with Google Analytics is related to sampling and data collection limits.

Now, let’s take a look at some of the ways you can overcome these shortcomings without spending a ton on Google Analytics 360 — a premium version of Google Analytics that, to some extent, also eliminates these issues.

PS! If you already have GA 360, these techniques will give you an even more robust and flexible tracking solution.

Table of Contents

– Sampling in Google Analytics
– Ways to avoid sampling in Google Analytics
– Why do you even need the raw data
– How to get access to unsampled hit-level raw data
– What about the price

***

Read the full post on medium.com

The post [Medium] Hit-Level Unsampled Google Analytics to BigQuery Without 360 appeared first on Reflective Data.

Google Analytics, on the other hand, is the most popular tool in the digital analytics world. It is used to keep track of marketing efforts, user behavior, traffic sources and more. The most common way to access data from Google Analytics is the web user interface. Alternative options include the Reporting API or external tools like Looker or Data Studio.

What’s interesting, though, is that there is no way to use the most popular query language to query data from the most popular analytics platform. That’s right, you cannot query Google Analytics data using SQL.

This is no big deal for the more basic users just checking the built-in reports in the Google Analytics UI. It is, for sure, a limitation for power users working with tools like Python and use data in custom models or even feed it into machine learning algorithms and recommendation engines.

In this article, we’re going to cover the solutions that will enable you to query your Google Analytics data using SQL.

Step 1 – Getting data into a database/data warehouse

As mentioned above, there is, unfortunately, no way to query data directly from Google Analytics using SQL. This means that the first step is to get data into some sort of a relational database or a data warehouse that support SQL queries.

There are three options for sending Google Analytics data into some external data storage.

1. Google Analytics 360 to BigQuery export

If your company has the 360 (premium) version of Google Analytics then you can use its native BigQuery export feature. Here are the details for setting this up. Keep in mind, though, that this solution will not work with the standard (free) version of Google Analytics.

2. Google Analytics Parallel Tracking

Google Analytics Parallel Tracking is a third party service that sends all of the raw hits into a data warehouse of your choice (i.e. BigQuery). These hits are then processed into sessions to provide a dataset similar to Google Analytics 360 export.

While this solution is not free, it will cost you only a small fraction of the cost of Google Analytics 360.

Getting started with Google Analytics Parallel Tracking.

3. Export data using the API

This is the most technical solution of the three. It leverages the Google Analytics Reporting API to pull data from Google Analytics and into your database or data warehouse.

One way to get started with this solution is to use the gago library in Go language to communicate with the Reporting API. Then you need the code for writing this data into your database.

The limitations with data export solution are that you can’t query all of the metrics and dimensions at the same time, you’re still affected by the data collection and sampling limits of Google Analytics and a modification in your tracking system is required (custom dimensions for hit timestamp, hit type, sessions ID, client ID and more).

Step 2 – Query Google Analytics data using SQL

This depends a bit on the setup you used to send data into a database/data warehouse and the type of data warehouse being used. Since the most common option is Parallel Tracking and Google BigQuery, we are going to use these in our examples as well.

The simplest way you can run your first SQL query against your Google Analytics data stored in BigQuery is to go into the BigQuery user interface and choose the right dataset containing your Google Analytics data.

With parallel tracking, you will have your Google Analytics data stored in three separate tables.

• raw_hits
• processed_hits
• processed_sessions

Depending on the type of query you want to run, choose the right table. Keep in mind that some of the information (bounce rate, geo-location etc.) isn’t available before hits are processed into sessions.

For example, we could write a simple SQL query like this and see the top 10 countries by the number of Google Analytics sessions.

Now, to make this query a bit more interesting, we might add in metrics like users and bounce rate. All doable with SQL, of course.

Notice how you can define your own rules for things like “bounce” and “bounce rate”.

Pretty neat, right?

Using the BigQuery UI is good for testing your setup and quick ad-hoc queries. In most cases, though, you’ll probably use some sort of a BI tool. One of the popular options these days is Google Data Studio. So, let’s use this in our examples a well.

Google Data Studio is free and probably the easiest to get started with. That being said, it is still packed with useful features and makes creating interactive dashboards a fun and enjoyable process.

Connecting BigQuery with Google Data Studio

Google Data Studio, like many other BI tools, has a native integration for BigQuery as a data source.

While you can automatically pull all data in a certain table, in most cases it’s a better idea to use a custom SQL query.

Data Studio has this feature built-in as well.

In our example, we’re going to analyze the results of an A/B test run using VWO. We’re using VWO’s custom-event-based integration to send experiment data into Google Analytics.

It’s always a good idea to try your query in the BigQuery’s query editor before implementing it in Data Studio.

Once we’re happy with the query and the results it returned, it’s time to put the query in Data Studio.

In Data Studio, we can start building all sorts of cool visualizations. For example this one for visualizing the funnels for Control and Variant 1 of our A/B test.

Data Studio is really flexible when it comes to building custom reports, interactive dashboards and quick visualizations. If you can’t find the right chart from the default chart types, check out the community visualizations or go ahead and build one yourself.

Conclusion

Working with your Google Analytics data using SQL opens up a whole new world of analysis opportunities for your digital analytics data. You no longer have to use the metrics and concepts defined and pre-calculated by Google. Feel free to come up with your own rules for things like calculating bounce rate or defining a session.

To get started, you first need to transfer your Google Analytics data into a data warehouse that supports SQL queries. For example, Google BigQuery. There are a few ways you can send Google Analytics data into BigQuery, our recommended method is Parallel Tracking. This will provide you with the most complete dataset, delivered in near-real-time and for a fraction of the cost of GA 360.

Once you have data in your database or data warehouse, you can start writing ad-hoc SQL queries right away. To take your productivity to a next level, we recommend running your queries and reports through a BI platform (i.e. Data Studio).

If you have any questions about analyzing your Google Analytics data using SQL or sending data into a data warehouse, feel free to ask them in the comments below.

The post How to Query Google Analytics Data Using SQL appeared first on Reflective Data.