Data Lake for Enterprises

Exploring the Data Ingestion Layer

The data ingestion layer is responsible for consuming messages from the messaging layer and performing the required transformation to ingest them in the lambda layer such that the transformed output conforms to the expected storage or processing formats. This layer must also make sure that the messages are consumed in a consistent way, such that no message is lost and every message is processed at least once.

This layer is expected to have multiple consumers/threads for parallel consumption of messages. Every such consumer in this layer must be stateless and must have fast streaming capability. These streams must be drawn from the messaging layer and the generated output must also be streamed into the lambda ...