⚠️ If you need help with planning or building a data pipeline or a data warehouse then get in touch for a free consultation session with one of our top data engineers!
These days, data is at the core of almost all successful companies. The system that feeds those companies with data is known as a data pipeline.
This blog post is for anyone eager to learn more about data pipelines, planning to build a new pipeline or upgrade their current setup.
An end-to-end analytics data pipeline is a secure and reliable mechanism that is responsible for feeding your business with valuable data that can be used for reporting, analysis, machine learning or any other activity that requires accurate data about your business.
Every business is different and so are the analytics data pipelines that best suits their needs. Our model of an enterprise-grade, fully customizable data pipelines are divided into six logical steps. Let’s take a closer look at each one of them.
Events → Enrichment → Process → Manage & Monitor → Storage → Report & Visualize
Events
Events are the basis of most data pipelines, they trigger other actions and take the largest part (storage) of your data lake/warehouse.
There is a wide variety of events that may act as a source for your data pipeline. Here are the six main categories of sources that generate an ongoing flow of events.
- Website users
- Click stream
- Google Analytics parallel tracking
- Form analytics
- Server events
- Orders
- Payments
- Mobile Apps
- Real-time
- Batch
- Ads
- Google Ads
- Facebook Ads
- Twitter Ads
- Others
- Feedback tool
- Surveys
- On-site polls
- 3rd parties
- Testing tools
- Personalization tools
- Email providers
- Call tracking
Events in data pipelines are usually invoked by webhooks that put all events into the processing queue where they are validated, enriched and batched.
For example, here’s a webhook for Google Analytics parallel tracking.
https://analytics-6785943.appspot.com/collect?v=1&_v=j79&a=1359602631&t=pageview&_s=1&dl=https%3A%2F%2Freflectivedata.com%2F&ul=en-us&de=UTF-8&dt=Homepage&sd=24-bit&sr=1920x1080&vp=1336x977&je=0&_u=SCCACEAjR~&jid=&gjid=&cid=1177657878.1573135112&tid=UA-3696947-1&_gid=775207114.1575821022&z=1779653917
The above webhook is quite similar to Google Analytics hit payload and is sent at the same time. It is then processed, enriched and streamed into the data warehouse.
All events are generally stored in three locations:
- Log files – raw hits
- Data Lake – Processed data (sometimes also enriched)
- BigQuery – Processed & enriched data
This setup ensures us that we can re-process everything should there be something wrong with the enrichment process or should the data somehow get lost from the BigQuery instance.
Enrichment
To give more context to the raw event-level data, our systems are pulling data from various sources, including:
- CRM & CMS
- Google Analytics
- 3rd party APIs
For sources that don’t allow real-time access, we are using periodic batch load processes.
Data from various sources is either combined with the hit level data or sent to the data warehouse separately for query-level joining or processing after collection.
Process
Processing makes sure your data is secure, reliable, accurate and free from duplicates. To securely process your data, we are using the following tools from Google Cloud:
Similar tools are also available in Amazon Web Services. There are several open-source solutions (often used by providers like Google and Amazon) but we prefer and recommend using managed services that can scale automatically. This allows us (and you) to focus on the data pipeline itself instead of maintaining the infrastructure.
We build the data processing queues to handle late-arriving data, duplicates and storage system outages in a way that best fits every use case. For example, people using your app on a 10-hour flight can produce a lot of events that are arriving and getting processed hours after they actually happened.
Manage & Monitor
To make sure everything is working as expected and to alert us and our clients if something is not, we are setting up a set of tools built in-house and from Google Cloud. Such as Stackdriver Monitoring.
Our goal is to provide our clients with the most accurate data with the least amount of latency.
In the pipeline management tools, you can see the pipeline’s structure, sources, processors, storage options etc. You would also see how data is moving between different elements and if there have been any errors or other incidents.
⚠️ If you need help with planning or building a data pipeline or a data warehouse then get in touch for a free consultation session with one of our top data engineers!
Storage
We’ve built the data pipeline so that it can send and store data in almost any database, data lake or data warehouse available. Here’s our standard recommendation.
- Google BigQuery
- Digital Analytics Platforms
- Google/Adobe Analytics
- Long-term storage
We make sure that your data stays safe and meets all the security requirements while giving you the full ownership and control over your data.
As mentioned earlier, we store raw hits as well to ensure that we can re-process everything should there be something wrong with the enrichment process or should the data somehow get lost from the BigQuery (or other reporting) instance.
Report & Visualize
Collecting data is pointless if it’s never being used to benefit your business. We think about the value our clients’ data has to provide in the earliest phase of planning and building the data pipeline. Our team works closely with our clients in order to figure out the KPIs, reports and dashboards that are needed for their companies growth.
Tools we are using for reporting, analysis and visualization include:
- Google Data Studio
- Jupyter Notebooks
- Redash
- Other BI tools
We build all analytics data pipelines and storage systems so that they could be connected with almost any BI tool available on the market. Wherever possible, we make data available for reporting in near real-time.
It’s no news that data is running most of the successful businesses these days and more and more companies as implementing modern data pipelines.
When thinking of a modern data pipeline, keep these keywords on mind:
- Serverless
- Scalable
- Real-time access
- Security
- Flexibility
Our team is happy to answer any analytics data pipeline related questions in the comments below.
Need help with planning or upgrading your data pipeline? Contact us and get a free consultation with one of our top data engineers.