Operationalizing the Data Lake

Chapter 4. Setting the Foundation for Your Data Lake

In Chapter 3, we examined the maturation stages that you will go through as you begin to actualize the value of your data. Now, it’s time to look more closely at what it takes to build the data lake.

Setting Up the Storage for the Data Lake

One of your first considerations in building your data lake will be storage. There are three basic types of data storage: immutable raw storage, optimized storage, and scratch databases. The type of data and how you use it will determine which data goes where.

Immutable Raw Storage Bucket

Data kept in immutable storage cannot, and should not, be changed after it has been written. In an immutable raw storage area in your data lake, you store data that hasn’t been scrubbed. You might never have even looked at it. But it should have sufficient self-descriptive language, or metadata, around it—such as table names and column names—so that you can determine where the data came from. You might store it in a text format such as JavaScript Object Notation (JSON) or comma-separated values (CSV), or perhaps even Apache Avro. Most people choose to store it in either JSON or CSV files.

Immutable raw storage fills many data storage needs. Three of the most important are:

Disaster recovery

If anything ever happens to the original data stores, you have an exact replica.

Forensic analysis

Immutable raw storage records can be used to trace problems, such as when bugs were introduced into a program. ...