Data Lake for Enterprises

Batch layer

Batch processing has been one of the most traditional ways of processing large amounts of data and are usually long-running processes. With the advent of many recent big data technologies, these Batch processes have become much more efficient and performant; this has greatly helped in reducing processing times.

The batch process is usually aware of the expected data to be consumed and the expected output. Historically, these processes were monolithic in nature and would process the entire dataset in a single run and some level of multi-threading, with specific mechanisms for handling failure scenarios and operational procedures to maintain such processes in production.

Hadoop, as a big data technology, provided all of the required ...