Big Data

Chapter 4. Data storage on the batch layer

This chapter covers

• Storage requirements for the master dataset
• Distributed filesystems
• Improving efficiency with vertical partitioning

In the last two chapters you learned about a data model for the master dataset and how you can translate that data model into a graph schema. You saw the importance of making data immutable and eternal. The next step is to learn how to physically store that data in the batch layer. Figure 4.1 recaps where we are in the Lambda Architecture.

Figure 4.1. The batch layer must structure large, continually growing datasets in a manner that supports low maintenance as well as efficient creation of the batch views.

Like the last two chapters, this chapter is dedicated ...