Data Lake for Enterprises

Batch layer - batch processing of ingested data

The batch processing layer of a Lambda Architecture is expected to process the ingested data in batches so as to ensure optimum utilization of system resources; at the same time, long-running operations may be applied to the data to ensure high quality of data output, which is also known as modeled data. The conversion of raw data to modeled data is the primary responsibility of this layer, wherein the modeled data is the data model that can be served by the serving layers of the Lambda Architecture. The primary specifications for this layer can be defined as follows:

• The batch layer must be able to apply data cleaning, data processing, and data modeling algorithms on the raw data ingested ...