Azure Cookbook

Chapter 6. Big Data

In Chapter 5, you learned about Azure Event Hubs, which allow ingesting big data streams from various sources into Azure. Network access logs, application logs, and online banking audit data are a few examples of datasets that you can send to Azure using Azure Event Hubs. Azure IoT Hub is another service, which specializes in ingesting telemetry data from connected IoT devices. For instance, you can configure tens, hundreds, or even thousands of connected sensors to send temperature, humidity, magnetic, pressure, or other data points to your applications. So far you have successfully ingested the raw data on a large scale. So what is next?

Raw data on its own does not bring much value. Think of it as the raw materials you need to come up with a great product. You need to process your big dataset to mine trends, patterns, and insights from it. Then you will have actionable information that is useful to your business. Microsoft Azure enables you to process large datasets using well-known products and technologies including Hadoop, Spark, and AI/ML models. For example, these technologies can be used to:

• Run analysis on historical weather data to predict natural disasters in the future

• Monitor live temperature telemetry sent by IoT sensors to catch and act on anomalies

• Monitor live access to an online banking application to detect and remediate fraud

• Calculate data aggregations for business reporting and decision making

This chapter introduces key Azure ...