Practical Real-time Data Processing and Analytics

Data collection

This is the beginning of the journey of all data processing. Be it batch or real-time, the foremost challenge is getting the data from its source to the systems for processing. We can look at the processing unit as a black box and a data source, and at consumers as publishers and subscribers. This is captured in the following diagram:

The key aspects that come under the criteria of data collection tools, in the general context of big data and real-time specifically, are as follows:

• Performance and low latency
• Scalability
• Ability to handle structured and unstructured data

Apart from this, any data collection tool should be ...