Understanding Log Analytics at Scale, 2nd Edition

Chapter 5. The Role of Object Stores for Log Data

Object storage and file storage take different approaches to organizing data, as illustrated in Figure 5-1.

Figure 5-1. Data structures used by file and object storage

Object storage uses a flat address space and comprehensive metadata to store chunks of data referred to as “blocks” in place of the hierarchical folder structure used in file storage, making it far more scalable. Thus, data retrieval remains fast, even as data stores swell from larger historical datasets to feed increasingly complex analytics. In addition, object storage is well suited to managing unstructured data by means of custom metadata to describe an object’s contents. This characteristic makes data self-describing, which provides flexibility for implementations of advanced analytics. Scaling analytics pipelines in the cloud can quickly become unpredictable in both cost and performance, so having an infrastructure which facilitates portability of data is vital for successful analytic deployments.

Object stores are designed to scale to hundreds of petabytes in size without degraded performance. They also protect data integrity while maximizing usable storage space. This technique divides data into shards, each of which is a subset of the table’s full set of rows. Shards are distributed across multiple database instances to spread load and increase parallelism ...