Data Lake for Enterprises

Speed layer - near real time data processing

This layer is expected to perform near-real-time processing on the data received from the ingestion layer. Since the processing is expected to be in near real time, such data processing will need to be quick, fast, and efficient, with support and design for high-concurrency scenarios and an eventually consistent outcome. A lot of factors play a role in making this layer fast, which will be discussed in detail later in this book. Broadly, the specifications for such a layer can be summarized as follows:

• Must support fast operation on very specific data streams ingested.
• Must be able to produce a data model relevant to near-real-time processing needs. All long-running processes must be delegated ...