Mastering Hadoop 3

Data lookups

The first question to be considered is why we need external data surveys in the pipeline for stream processing. The answer is that, sometimes, operations such as enhancement, validation of data, or filtering of data are necessary on the basis of some often changing external system data. However, these data surveys pose certain challenges in the streaming design context. These data searches could lead to increased end-to-end latency because calls to external systems are frequently made. Not all external reference data can be stored, because such external datasets are too large to fit in the memory. They change too often, which makes it difficult to refresh your memory. If these external systems are down, they will become a streaming ...