Fast Data: Smart and at Scale

Foreword

We are witnessing tremendous growth of the scale and rate at which data is generated. In earlier days, data was primarily generated as a result of a real-world human action—the purchase of a product, a click on a website, or the pressing of a button. As computers become increasingly independent of humans, they have started to generate data at the rate at which the CPU can process it—a furious pace that far exceeds human limitations. Computers now initiate trades of stocks, bid in ad auctions, and send network messages completely independent of human involvement.

This has led to a reinvigoration of the data-management community, where a flurry of innovative research papers and commercial solutions have emerged to address the challenges born from the rapid increase in data generation. Much of this work focuses on the problem of collecting the data and analyzing it in a period of time after it has been generated. However, an increasingly important alternative to this line of work involves building systems that process and analyze data immediately after it is generated, feeding decision-making software (and human decision makers) with actionable information at low latency. These “fast data” systems usually incorporate recent research in the areas of low-latency data stream management systems and high-throughput main-memory database systems.

As we become increasingly intolerant of latency from the systems that people interact with, the importance and prominence of fast data ...