Big Data: Understanding How Data Powers Big Business

Chapter 2

Big Data History Lesson

Chapter 1 hinted at how retail point-of-sale (POS) scanner data caused a big data revolution that transformed the Consumer Package Goods (CPG) and retail industries in the late 1980s and 1990s. Let's spend a bit more time on that event, as there are some valuable lessons to be learned that apply to today's big data revolution.

Consumer Package Goods and Retail Industry Pre-1988

In the 1970s and early 1980s, CPG manufacturers such as Procter & Gamble, Unilever, Colgate-Palmolive, Kraft, and General Mills, to name but a few, and large grocery, drug, and mass merchandise retailers made their marketing decisions based on bi-monthly Nielson store audit data. That is, Nielsen would send people into a sample of stores (in only about 12 cities across the United States) to conduct physical audits—to count how much product was on the shelf, the price of the product, how much linear footage the product had across the front of the shelf, product sales within that store, and other data. Nielsen would aggregate this information by product category in order to calculate market share by volume and revenue, share of shelf space, etc. The results of the audits were then delivered every two months to the retailers and CPG manufacturers, usually in booklet format. CPG manufacturers could also request the data in tape format, but the data volumes were easily in the megabyte range.

So a company like Procter & Gamble would use this data for their Crest brand toothpaste, ...