Stream Analytics with Microsoft Azure

Stream Analytics with Microsoft Azure

by Anindita Basak, Krishna Venkataraman, Ryan Murphy, Manpreet Singh
Released December 2017
Publisher(s): Packt Publishing
ISBN: 9781788395908

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Start your free trial

Book description

Develop and manage effective real-time streaming solutions by leveraging the power of Microsoft Azure

About This Book

  • Analyze your data from various sources using Microsoft Azure Stream Analytics
  • Develop, manage and automate your stream analytics solution with Microsoft Azure
  • A practical guide to real-time event processing and performing analytics on the cloud

Who This Book Is For

If you are looking for a resource that teaches you how to process continuous streams of data in real-time, this book is what you need. A basic understanding of the concepts in analytics is all you need to get started with this book

What You Will Learn

  • Perform real-time event processing with Azure Stream Analysis
  • Incorporate the features of Big Data Lambda architecture pattern in real-time data processing
  • Design a streaming pipeline for storage and batch analysis
  • Implement data transformation and computation activities over stream of events
  • Automate your streaming pipeline using Powershell and the .NET SDK
  • Integrate your streaming pipeline with popular Machine Learning and Predictive Analytics modelling algorithms
  • Monitor and troubleshoot your Azure Streaming jobs effectively

In Detail

Microsoft Azure is a very popular cloud computing service used by many organizations around the world. Its latest analytics offering, Stream Analytics, allows you to process and get actionable insights from different kinds of data in real-time.

This book is your guide to understanding the basics of how Azure Stream Analytics works, and building your own analytics solution using its capabilities. You will start with understanding what Stream Analytics is, and why it is a popular choice for getting real-time insights from data. Then, you will be introduced to Azure Stream Analytics, and see how you can use the tools and functions in Azure to develop your own Streaming Analytics. Over the course of the book, you will be given comparative analytic guidance on using Azure Streaming with other Microsoft Data Platform resources such as Big Data Lambda Architecture integration for real time data analysis and differences of scenarios for architecture designing with Azure HDInsight Hadoop clusters with Storm or Stream Analytics. The book also shows you how you can manage, monitor, and scale your solution for optimal performance.

By the end of this book, you will be well-versed in using Azure Stream Analytics to develop an efficient analytics solution that can work with any type of data.

Style and approach

A comprehensive guidance on developing real-time event processing with Azure Stream Analysis

Table of contents

  1. Preface
    1. What this book covers
    2. What you need for this book
    3. Who this book is for
    4. Conventions
    5. Reader feedback
    6. Customer support
      1. Downloading the example code
      2. Downloading the color images of this book
      3. Errata
      4. Piracy
      5. Questions
  2. Introducing Stream Processing and Real-Time Insights
    1. Understanding stream processing
    2. Understanding queues, Pub/Sub, and events
      1. Queues
      2. Publish and Subscribe model
        1. Real-world implementations of the Publish/Subscribe model
      3. Azure implementation of queues and Publish/Subscribe models
    3. What is an event?
      1. Event streaming
      2. Event correlation
      3. Azure implementation of event processing
      4. Architectural components of Event Hubs
      5. Simple event processing 
      6. Event stream processing
      7. Complex event processing 
    4. Summary
  3. Introducing Azure Stream Analytics and Key Advantages
    1. Services offered by Microsoft
    2. Introduction to Azure Stream Analytics
      1. Configuration of Azure Stream Analytics
      2. Key advantages of Azure Stream Analytics
        1. Security
        2. Programmer productivity
        3. Declarative SQL constructs
        4. Built-in temporal semantics
        5. Lowest total cost of ownership
        6. Mission-critical and enterprise-less scalability and availability
        7. Global compliance
    3. Microsoft Cortana Intelligence suite integration
      1. Azure IoT integration
    4. Summary
  4. Designing Real-Time Streaming Pipelines
    1. Differencing stream processing and batch processing
      1. Logical flow of processing
        1. Out of order and late arrival of data
        2. Session grouping and windowing challenges 
        3. Message consistency  
        4. Fault tolerance, recovery, and storage
        5. Source
        6. Communication and collection
        7. Ingest, queue, and transform
          1. Hot path
          2. Cold path
        8. Data retention
        9. Presentation and action
    2. Canonical Azure architecture
    3. Summary
  5. Developing Real-Time Event Processing with Azure Streaming
    1. Stream Analytics tools for Visual Studio
      1. Prerequisites for the installation of Stream Analytics tools
      2. Development of a Stream Analytics job using Visual Studio
    2. Defining a Stream Analytics query for Vehicle Telemetry job analysis using Stream Analytics tools
      1. Query to define Vehicle Telemetry (Connected Car) engine health status and pollution index over cities
      2. Testing Stream Analytics queries locally or in the cloud
      3. Stream Analytics job configuration parameter settings in Visual Studio
    3. Implementation of an Azure Stream Analytics job using the Azure portal
      1. Provisioning for an Azure Stream Analytics job using the Azure Resource Manager template
        1. Azure ARM Template - Infrastructure as code
        2. Getting started with provisioning Azure Stream Analytics job using the ARM template
        3. Deployment and validation of the Stream Analytics ARM template to Azure Resource Group
      2. Configuration of the Azure Streaming job with different input data sources and output data sinks
        1. Data input types-data stream and reference data
      3. Data Stream inputs
        1. Reference data
        2. Job topology output data sinks of Stream Analytics
    4. Summary
  6. Building Using Stream Analytics Query Language
    1. Built-in functions
      1. Scalar functions
      2. Aggregate and analytic functions
      3. Array functions
      4. Other functions
        1. Data types and formats
        2. Complex types
    2. Query language elements
    3. Windowing
      1. Tumbling windows
      2. Hopping windows
      3. Sliding windows
    4. Time management and event delivery guarantees
    5. Summary
  7. How to achieve Seamless Scalability with Automation
    1. Understanding parts of a Stream Analytics job definition (input, output, reference data, and job)
      1. Deployment of Azure Stream Analytics using ARM template 
        1. Configuring input
        2. Configuring output
          1. Building the sample test code
    2. How to scale queries using Streaming units and partitions
      1. Application and Arrival Time
        1. Partitions
          1. Input source
          2.  Output source
          3. Embarrassingly parallel jobs and Not embarrassingly parallel jobs
    3. Sample use case 
    4. Configuring SU using Azure portal
    5. Out of order and late-arriving events
    6. Summary
  8. Integration of Microsoft Business Intelligence and Big Data
    1. What is Big Data Lambda Architecture?
      1. Concepts of batch processing and stream processing in data analytics
      2. Specifications for slow/cold path of data - batch data processing
    2. Moving to the streaming-based data solution pattern
    3. Evolution of Kappa Architecture and benefits 
    4. Comparison between Azure Stream Analytics and Azure HDInsight Storm
    5. Designing data processing pipeline of an interactive visual dashboard through Stream Analytics and Power BI
      1. Integrating Power BI as an output job connector for Stream Analytics
    6. Summary
  9. Designing and Managing Stream Analytics Jobs
    1. Reference data streams with Azure Stream Analytics
      1. Configuration of Reference data for Azure Stream Analytics jobs
        1. Integrating a reference data stream as job topology input for an Azure Stream Analytics job
          1. Stream Analytics query configuration for Reference Data join
        2. Refresh schedule of a reference data stream
      2. Configuration of output data sinks for Azure Stream Analytics with Azure Data Lake Store 
        1. Configuring Azure Data Lake Store as an output data sink of Stream Analytics
        2. Configuring Azure Data Lake Store as an output sink of Stream Analytics jobs
      3. Configuring Azure Cosmos DB as an output data sink for Azure Stream Analytics 
        1. Features of Azure Cosmos DB for configuring output sinks of Azure Stream Analytics
        2. Configuring Azure Cosmos DB integrated with Azure Stream Analytics as an output sink
      4. Stream Analytics job output to Azure Function Apps as Serverless Architecture 
        1. Provisioning steps to an Azure Function 
        2. Configuring an Azure function as a serverless architecture model integrated with Stream Analytics job output
    2. Summary 
  10. Optimizing Intelligence in Azure Streaming
    1. Integration of JavaScript user-defined functions using Azure Stream Analytics
      1. Adding JavaScript UDF with a Stream Analytics job
        1. Stream Analytics and JavaScript data type conversions
      2. Integrating intelligent Azure machine learning algorithms with Stream Analytics function
      3. Data pipeline Streaming application building concepts using Azure .NET Management SDK
        1. Implementation steps of Azure Stream Analytics jobs using .NET management SDK
    2. Summary
  11. Understanding Stream Analytics Job Monitoring
    1. Troubleshooting with job metrics
    2. Visual monitoring of job diagram
    3. Logging of diagnostics logs
      1. Enabling diagnostics logs
      2. Exploring the logs sent to the storage account
    4. Configuring job alerts
    5. Viewing resource health information with Azure resource health
    6. Exploring different monitoring experiences
      1. Building a monitoring dashboard
    7. Summary
  12. Use Cases for Real-World Data Streaming Architectures
    1. Solution architecture design and Proof-of-Concept implementation of social media sentiment analytics using Twitter and a sentiment analytics dashboard
      1. Definition of sentiment analytics
        1. Prerequisites required for the implementation of Twitter sentiment analytics PoC
        2. Steps for implementation of Twitter sentiment analytics
      2. Remote monitoring analytics using Azure IoT Suite 
        1. Provisioning of remote device monitoring analytics using Azure IoT Suite
      3. Implementation of a connected factory use case using Azure IoT Suite
        1. Connected factory solution with Azure IoT Suite
      4. Real-world telecom fraud detection analytics using Azure Stream Analytics and Cortana Intelligence Gallery with interactive visuals from Microsoft Power BI
        1. Implementation steps of fraud detection analytics using Azure Stream Analytics
        2. Steps for building the fraud detection analytics solution
    2. Summary

Product information

  • Title: Stream Analytics with Microsoft Azure
  • Author(s): Anindita Basak, Krishna Venkataraman, Ryan Murphy, Manpreet Singh
  • Release date: December 2017
  • Publisher(s): Packt Publishing
  • ISBN: 9781788395908