The Promise of Big Data is Broken for Consumer Product Companies – Here’s Why


The promise of big data in the consumer products industry has filled a thousand lakes.  In “Why the promise of big data hasn’t delivered yet," Rosemary Barnett wrote “The basic premise of the industry’s offering is this: Hidden in that huge mass of enterprise data are latent patterns.  If only you could interpret your data properly, like an explorer deciphering an ancient scroll, you would be able to unearth these precious business secrets. Specialist analytic software tools are needed to crack the code. The big, diverse, disparate, messy data go into these tools, and actionable insights come out.”

Big Data is Broken

I challenge you to search the web for one piece of evidence of a hidden intelligence that business took action on and had a quantifiable result. You might find what I found, nothing to validate the big data-insights-action-value chain as a concept to the value of promises being made. I did not find a case study to validate the premise.”1

Why Big Data Failed in Consumer Products

Gartner dropped Big Data from their Hype Cycle of Emergent Technologies back in 2015.  What is the reason for big data’s failure to launch when it comes to consumer products?  

Big Data fails in Consumer Products when there is not a data platform for customer (retailer) data.  The Demand Signal Repository (DSR), invented by John Beckett from Retail Velocity in 1994, is a more complicated build than previously thought.   Beckett explains the challenges to retailers in his groundbreaking white paper, “The Tower of Babel."

According to Lora Cecere of Supply Chain Insights, an analyst who follows the space:

Synchronizing demand data requires a Demand Signal Repository (DSR).  We have not interviewed any company successful in building their own.  The gap for most companies is master data synchronization.”2

This may or may not be, “Companies that experiment with downstream data in any form, for any usage, report that the ROI is in days or weeks.  In short, the savings are so great that no company wants to go on the record and share their results.  They consider it their secret weapon."(3) continues Cecere.  “Most successful DSR’s have an ROI measured in weeks. The landscape is scattered with failures with some of (the) larger companies failing multiple times.”2

We Can Fulfill the Promise of Big Data

It does not make sense for a consumer products company to invest in big data without a way to bring downstream demand data into a clean and harmonized single version of the truth.  The DSR can stream clean retailer data that flows into a lake; this data will be used to create valuable, lasting insights. Ultimately, this information will form ideas and innovations that drive revenue; it can be combined and blended with social media, weather, census, and other 3rd party data.  Without clean and harmonized demand data, it would be difficult to get a positive ROI.

Each silo within an organization uses different forms of demand data.” writes Lora Cecere.  “The supply chain team uses order and shipment data.  The sales account teams use POS data for trade promotion planning, performance reporting, assortment planning/category management and retail execution.  In contrast, the marketing department relies on syndicated data.  Each form of data moves at a different cadence, level of granularity and accuracy.”3 

The lack of accuracy of the syndicated data, the latency of supply chain data and the inability to clean and harmonize sales data become barriers to success to a Big Data initiative.  

The initial promise of big data and machine learning was to move our industry away from managing the business based on what happened in the past, towards managing business using insights based on what events are likely to occur in the future.  There is one step that is missing in this approach, though -- managing a business in real time.  To do this, we need a Demand Signal Repository that connects to our customers (retailers), cleans and harmonizes the data and ultimately creates one version of the truth.  We need a repository that everyone in the organization can use to access near-real-time sales data by market in an easy to use BI tool.

The transformation of the industry will come from big data; it will happen when consumer product companies can collect clean and harmonized data from all their customers. That data will come in a variety of forms, streaming in near-real time, into one repository, with one version of the truth by SKU, by store, by district, by market by region, and by state across all their retail customers.  

A Retailer Data Platform (My new name for a DSR) is a foundation that can move your enterprise from managing your business based on what happened in the past, to driving your business in near-real time.  The transformation is complete when you add unstructured data and incorporate machine learning to mine insights and innovations to effectively plan the future. With a clean, harmonized data stream in hand, we can fulfill the promise of Big Data for Consumer Product Companies.


  1. Rosemary Barnett, “Why the promise of big data hasn’t delivered yet” Crunch Network, January 2017.
  2. Lora Cecere, The Power of Downstream Data: Gaining an understanding of the value proposition for Channel Data, Supply Chain Insights, July 2017
  3. Lora Cecere, “Integrated Demand Management: When will we start using downstream data?” Supply Chain Insights, November 2012.
partner1 partner1 partner1 partner1