Solutions Architect's Handbook

Data Engineering and Machine Learning

In the era of the internet and digitization, data is being generated everywhere with high velocity and in high quantities. Getting insight from these huge amounts of data at a fast pace is challenging. We need to innovate continuously to ingest, store, and process this data to derive business outcomes.

With the convergence of many cloud, mobile, and social technologies, advancements in many fields such as genomics and life sciences are growing at an ever-increasing rate. Tremendous value is found in mining this data for more insight. A fundamental difference between streaming systems and batch systems is the requirement to process unlimited data. Modern stream processing systems need to produce continual ...