09 Aug 2019 | Data Strategy

Treat Pipeline Automation Deficiency Syndrome Before It’s Too Late

Charles Wang
Charles Wang
Treat Pipeline Automation Deficiency Syndrome Before It’s Too Late
Don’t miss these warning signs for pipeline automation deficiency! Your organization’s health depends on it.

TC is a three-year-old, 30-person startup presenting to the emergency room unresponsive, with debilitating CSV file incontinence and data impaction resulting from months of business intelligence backlogs.

TC’s relative youth should have provided some protection against such a condition, but its small size and occupation, which involved generating high volumes of personalized recommendations for customers, facilitated the rapid progression of the disease. In the months preceding TC’s initial illness, TC began using a number of data sources dealing with customer relationship management, e-commerce, advertising, and event-tracking. TC had also hired two data scientists to develop a collaborative filtering engine to recommend products to its customers.

The symptoms of TC’s illness first manifested during the winter holiday season, when TC provided dozens of recommendations each to hundreds of thousands of accounts, triggering an acute inflammation in the business intelligence department as analysts worked overtime to ensure the timely and secure integration of data. TC attempted to self-medicate by exhorting its analysts to manually assemble ETL pipelines using raw data files, Python scripts, and ad-hoc orchestration. The inflammation spread to other departments as the data pipeline problem worsened, forcing engineers to get involved. In desperation, analysts sometimes ran queries directly on operational databases, delaying customer transactions in the process. TC suffered malaise and severe cognitive deficits for an entire quarter as team members burned out.

The symptoms appeared to subside briefly until the first day of the following summer, when TC’s investors found TC soaked in data file effluent and unresponsive, bringing us to the emergency room where we are now.

Symptoms of TC’s illness, called Pipeline Automation Deficiency Syndrome (PADS), include:

  1. Subdural effusion of CSV, XLSX, and other types of data files
  2. Gradual leakage of data files from effusions
  3. Data impaction as analysts wrangle data instead of analyzing it
  4. Slowed movement as operational databases handle transactions intended for data warehouses
  5. Impaired coordination as reports are delayed
  6. Disorganized and fragmented thinking as patient’s sensory input is spread across multiple, siloed data sources and dashboards
  7. Paranoid delusions resulting from unanticipated schema changes
  8. General loss of executive function and decision-making ability
  9. Wasting syndrome and acute malnutrition resulting from heightened caloric demands posed by business intelligence, engineering, and IT
  10. Chronic fatigue as the patient is only able to GET REST from REST APIs
  11. Deteriorating hygiene as the only SOAP the sufferer is able to access is from the Salesforce documentation
  12. Insomnia resulting from compulsive late-night use of Python and data engineering tutorials
  13. General symptoms of Stack Overflow Use Disorder
  14. Hypopigmentation of the skin as the patient stays indoors and limits light exposure to blue light from monitors

As a “smart” e-commerce company, TC’s symptoms were acute and highly seasonal in nature, exacerbated by periodic spikes in consumer spending. The admitting physician noted that TC’s poor judgment, exhibited by its refusal of help following the initial episode, was consistent with the general cognitive deterioration associated with PADS.

The root of TC’s illness was an inability to integrate large volumes of data. Many organizations share this deficiency, but for organizations like TC, survival often depends on ingesting and analyzing large volumes of data, so the option to simply ignore data is hazardous.

To treat TC’s ailments, his physicians prescribed the modern data stack. With the use of highly-optimized data connectors and a cloud data warehouse, the modern data stack centralizes data, quickly dissolving and clearing data impaction and file effusions, as well as resolving the competing demands posed by business intelligence, engineering, and IT. This can immediately relieve and reverse the deterioration associated with PADS, restoring normal function to the patient.

As an automated, fully-managed service, it requires little intervention by the patient beyond the decision to accept the treatment. Since there are no known negative side effects of the modern data stack, current guidelines recommend it as a preventative measure, as well.

With the modern data stack treatment, TC was able to make a full recovery.

Although this story has a happy ending, your organization may be susceptible to PADS. Risk factors for PADS include:

  1. Heavy data consumption
  2. Integrating data from multiple applications, tools, and other sources
  3. Purchasing and using additional applications, tools, and other sources
  4. Rapid company growth
  5. Small, understaffed business intelligence or data science teams
  6. Expanding volume and complexity of data
  7. Directly querying operational databases to extract insights
  8. Data integration on-premise
  9. Data warehousing on-premise
  10. Attempts to build artificial intelligence, predictive analytics, and data-driven products
  11. Self-medication using custom scripts to perform extract, transform, and load
  12. Previous episodes of PADS

If any of these risk factors apply to you, you should proactively consider the services of an automated, modern data stack from Fivetran. Request a personalized demo or begin your free trial today.

Are You A Data Expert?

Start a free trial today.

Discover the smartest solution for data-driven results.
We have detecting that you are using an adblocking plugin in your browser. We don't show ads, but we rely on advertising services, so it might restrict you from completing important functions or seeing important content. Please make sure you whitelist our website in your adblocking plugin.
Fivetran uses cookies to enhance your user experience and improve the quality of our website. Unless you disable cookies, you consent to the placement and use of cookies as described in our Privacy Policy by continuing to use this website.
Adblock Detection