Architectural Patterns

Data sources and ingestion layer

Enterprise big data systems face a variety of data sources with non-relevant information (noise) alongside relevant (signal) data. Noise ratio is very high compared to signals, and so filtering the noise from the pertinent information, handling high volumes, and the velocity of data is significant. This is the responsibility of the ingestion layer. The common challenges in the ingestion layers are as follows:

• Multiple data source load and prioritization
• Ingested data indexing and tagging
• Data validation and cleansing
• Data transformation and compression

The preceding diagram depicts the building blocks of ...