An Overview of Data Integration and Analytics
The following blog post is an excerpt from the book, The Essential Guide to Data Integration: How to Thrive in an Age of Infinite Data. The rest of the book is available to you for free here.
Analytics long predates modern data collection. During the Crimean War, Florence Nightingale used coxcomb diagrams to identify and reduce the causes of hospital mortality.
Source: “Notes on Matters Affecting the Health, Efficiency and Hospital Administration of the British Army,” by Florence Nightingale. London: Harrison & Sons, 1858.
In the years since, the growth of modern computing and the internet has enabled the collection and analysis of data at vastly larger scales than was possible using pen, paper and tabulating machines.
Today, even organizations of modest means have access to vast amounts of data. A typical commercial entity, nonprofit or governmental organization might produce terabytes of data every year from a wide range of operational systems, applications and devices. This wealth of data creates unprecedented opportunities for modern organizations.
Goals of Analytics
Analytics can be used to improve customer acquisition, retention and loyalty; identify new product opportunities; and enhance existing opportunities.
Broadly speaking, you can use analytics in the following ways:
- Ad-hoc reporting. Key stakeholders and decision-makers will sometimes need very specific questions answered on a one-time or occasional basis.
- Business intelligence. Often used interchangeably with “analytics,” business intelligence (BI) refers to the use of visualizations and data models to identify opportunities and guide business decisions and strategies. This usually comes in the form of regular, consistent reports and up-to-date dashboards.
- Data as a product. Data your organization collects or produces can be made available to third parties in the form of embedded dashboards, data streams, recommendations, and other data products.
- Artificial intelligence/machine learning. The pinnacle of analytics is building products and systems that use predictive modeling to automate important decisions and processes.
Knowledge is power, and it is always advantageous to know more than the competition.
A Major Obstacle to Analytics: Data Integration
A central data repository of record offers your organization the following benefits:
- You gain a big-picture view of your organization’s operations and see how the parts work together, instead of viewing siloed, isolated representations.
- You can match records and track the same entities (customer, partner, etc.) across different stages of their life cycles.
- You can perform analytics in an environment separated from operational systems, preventing your queries from interfering with your operations.
- You exercise granular control over access and permissions, ensuring that your team gets the information they need to perform their jobs without compromising sensitive systems.
Creating this central data repository can be a Herculean task. Every data source requires separate procedures and tools to ingest, clean and model its data.
This challenge has been amplified by the recent proliferation of cloud-based applications and services.
SaaS Data: A Growing Challenge and Opportunity
In the cloud era, SaaS applications have become one of the predominant sources of business data. SaaS apps span a huge range of operations and industries: marketing, payment processing, customer relationship management, ecommerce, engineering project management and many more. SaaS applications commonly record actions by users, offering organizations highly granular pictures of their operations from which they can deduce patterns and causal relationships. Generally, the more facets of your business you can quantify and analyze, the more competitive you are.
A typical company now uses more than 100 apps. At that scale, manual data integration is virtually impossible.
Heavy time commitments divert analysts, data scientists and engineers from other activities.
Luckily, cloud technology offers a solution to this challenge. Modern data pipeline tools, data warehouses and business intelligence platforms are cloud-based applications in their own right, and they have proliferated alongside cloud technology. They effectively eliminate the need to manually develop customized, in-house tools and solutions for data integration and analytics.
Read about approaches to data integration here.
The excerpt above is from The Essential Guide to Data Integration: How to Thrive in an Age of Infinite Data. The book covers topics such as how data integration fuels analytics, the evolution from ETL to ELT to automated data integration, the benefits of automated data integration, and tips on how to evaluate data integration providers. Get your free copy of the guide today: