In this chapter, we explore where to land the transferred data and how to choose among the storage options. These options fall into two broad categories: file-oriented storage and queue-oriented storage. The particular category selected impacts the type (and latency) of processing performed at later stages in the pipeline. We intentionally omit other data stores (such as NoSQL or document stores) as the initial landing place for ingested data, as the file and queue options are the simplest and least likely to impose changes on the ingested data before processing can begin.
In terms of our analytics pipeline we are going to examine the storage items outlined in Figure 3-1.
File-Oriented Storage
The more things change, the more they stay the same. This is also true of the innovations in approaches for storing big data used in analytics scenarios—the notion of a filesystem that contains a tree of directories, which in turn can contain files of different formats and encodings, has persisted in storing data at cloud scale. In this section, we examine three such “filesystems” prevalent in Azure: Blob Storage, Azure Data Lake Store, and the Hadoop File System (HDFS).
Figure 3-1. Transient and persistent storage services are the focus of this chapter.
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month, and much more.
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.
Julian F.
Head of Cybersecurity
I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.
Addison B.
Field Engineer
I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.
Amir M.
Data Platform Tech Lead
I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.
