Mastering Azure Analytics, 1st Edition

Chapter 3. Storing Ingested Data in Azure

In this chapter, we explore where to land the transferred data and how to choose among the storage options. These options fall into two broad categories: file-oriented storage and queue-oriented storage. The particular category selected impacts the type (and latency) of processing performed at later stages in the pipeline. We intentionally omit other data stores (such as NoSQL or document stores) as the initial landing place for ingested data, as the file and queue options are the simplest and least likely to impose changes on the ingested data before processing can begin.

In terms of our analytics pipeline we are going to examine the storage items outlined in Figure 3-1.

File-Oriented Storage

The more things change, the more they stay the same. This is also true of the innovations in approaches for storing big data used in analytics scenarios—the notion of a filesystem that contains a tree of directories, which in turn can contain files of different formats and encodings, has persisted in storing data at cloud scale. In this section, we examine three such “filesystems” prevalent in Azure: Blob Storage, Azure Data Lake Store, and the Hadoop File System (HDFS).

Figure 3-1. Transient and persistent storage services are the focus of this chapter.

Blob Storage

Azure Blob Storage provides highly available, high-scale object storage and allows ...