Data Lake for Enterprises

Messaging Layer - guaranteed data delivery

The messaging layer would form the Message Oriented Middleware (MOM) for the data lake architecture, and hence would be the primary layer for decoupling the various layers with each other, but with guaranteed delivery of messages.

In order to ensure that the delivery of messages is guaranteed, the messages will need to be persistent. This persistence of messages is usually done on a storage drive. The storage drive used for persisting these messages should be fit for the purpose based on number and size of the messages to be stored. Fundamentally, since the nature of message oriented middleware is to queue up the messages, for both writes and reads, this fits well into the characteristics of  serial ...