book

Machine Learning for Streaming Data with Python

by Joos Korstanje

July 2022

Beginner to intermediate

258 pages

4h 57m

English

Packt Publishing

Read now

Unlock full access

ContributorsAbout the authorAbout the reviewer
Who this book is forWhat this book coversTo get the most out of this bookDownload the example code filesDownload the color imagesConventions usedGet in touchShare Your Thoughts
Technical requirementsSetting up a Python environmentA short history of data scienceWorking with streaming dataStreaming data versus batch dataAdvantages of streaming dataExamples of successful implementation of streaming analyticsChallenges of streaming dataHow to get started with streaming dataCommon use cases for streaming dataStreaming versus big dataReal-time data formats and importing an example dataset in PythonSummaryFurther reading
Technical requirementsPython environmentDefining your analytics as a functionUnderstanding microservices architectureCommunicating between services through APIsDemystifying the HTTP protocolThe GET requestThe POST requestJSON format for communication between systemsRESTful APIsBuilding a simple API on AWSAPI Gateway in AWSLambda in AWSData-generating process on a local machineImplementing the exampleMore architectural considerationsOther AWS services and other services in general that have the same functionalityBig data tools for real time streamingCalling a big data environment in real timeSummaryFurther reading
Technical requirementsPython environmentDescriptive statistics on streaming dataWhy are descriptive statistics different on streaming data?Introduction to sampling theoryComparing population and samplePopulation parameters and sample statisticsSampling distributionSample size calculations and confidence levelRolling descriptive statistics from streamingExponential weightTracking convergence as an additional KPIOverview of the main descriptive statisticsThe meanThe medianThe modeStandard deviationVarianceQuartiles and interquartile rangeCorrelationsReal-time visualizationsOpening the dashboardComparing Plotly's Dash and other real-time visualization toolsBuilding basic alerting systemsAlerting systems on extreme valuesAlerting systems on process stability (mean and median)Alerting systems on constant variability (std and variance)Basic alerting systems using statistical process controlSummaryFurther reading
Technical requirementsPython environmentWhat is online machine learning?How is online learning different from regular learning?Advantages of online learningChallenges of online learningTypes of online learningUsing River for online learningTraining an online model with RiverImproving the model evaluationBuilding a multiclass classifier using one-vs-restSummaryFurther reading
Technical requirementsPython environmentDefining anomaly detectionAre outliers a problem?Exploring use cases of anomaly detectionFraud detection in financial institutionsAnomaly detection on your log dataFault detection in manufacturing and production linesHacking detection in computer networks (cyber security)Medical risks in health dataPredictive maintenance and sensor dataComparing anomaly detection and imbalanced classificationThe problem of imbalanced dataThe F1 scoreSMOTE oversamplingAnomaly detection versus classificationAlgorithms for detecting anomalies in RiverThe use of thresholders in River anomaly detectionAnomaly detection algorithm 1 – One-Class SVMAnomaly detection algorithm 2 – Half-Space-TreesGoing further with anomaly detectionSummaryFurther reading
Technical requirementsPython environmentDefining classificationIdentifying use cases of classificationUse case 1 – email spam classificationUse case 2 – face detection in phone cameraUse case 3 – online marketing ad selectionOverview of classification algorithms in RiverClassification algorithm 1 – LogisticRegressionClassification algorithm 2 – PerceptronClassification algorithm 3 – AdaptiveRandomForestClassifierClassification algorithm 4 – ALMAClassifierClassification algorithm 5 – PAClassifierEvaluating benchmark resultsSummaryFurther reading

Technical requirementsPython environmentDefining regressionUse cases of regressionUse case 1 – ForecastingUse case 2 – Predicting the number of faulty products in manufacturingOverview of regression algorithms in RiverRegression algorithm 1 – LinearRegressionRegression algorithm 2 – HoeffdingAdaptiveTreeRegressorRegression algorithm 3 – SGTRegressorRegression algorithm 4 – SRPRegressorSummaryFurther reading
Technical requirementsPython environmentDefining reinforcement learningComparing online and offline reinforcement learningA more detailed overview of feedback loops in reinforcement learningThe main steps of a reinforcement learning modelMaking the decisionsUpdating the decision rulesExploring Q-learningThe goal of Q-learningParameters of the Q-learning algorithmDeep Q-learningUsing reinforcement learning for streaming dataUse cases of reinforcement learningUse case one – trading systemUse case two – social network ranking systemUse case three – a self-driving carUse case four – chatbotsUse case five – learning gamesImplementing reinforcement learning in PythonSummaryFurther reading
Technical requirementsPython environmentDefining driftThree types of driftIntroducing model explicabilityMeasuring driftMeasuring data driftMeasuring concept driftMeasuring drift in PythonA basic intuitive approach to measuring driftMeasuring drift with robust toolsCounteracting driftOffline learning with retraining strategies against driftOnline learning against driftSummaryFurther reading
Technical requirementsPython environmentChallenges of data preparation with streaming dataScaling data for streamingIntroducing scalingAdapting scaling to a streaming contextTransforming features in a streaming contextIntroducing PCAMathematical definition of PCARegular PCA in PythonIncremental PCA for streamingSummaryFurther reading
Technical requirementsPython environmentIntroducing catastrophic forgettingCatastrophic forgetting in online modelsDetecting catastrophic forgettingUsing Python to detect catastrophic forgettingModel explicability versus catastrophic forgettingExplaining models using linear coefficientsExplaining models using dendrogramsExplaining models using variable importanceSummaryFurther reading
Going furtherSummary
Other Books You May EnjoyPackt is searching for authors like youShare Your Thoughts

Content preview from Machine Learning for Streaming Data with Python

Chapter 5: Online Anomaly Detection

Anomaly detection is a good starting point for machine learning on streaming data. As streaming data delivers a continuous stream of data points, use cases of monitoring live solutions are among the first that come to mind.

There are many domains in which monitoring is essential. In IT solutions, there is generally continuous logging of what happens in the systems, and those logs can be analyzed as streaming data.

In the Internet of Things (IoT), sensor data is being collected on sometimes a large number of sensors. This data is then analyzed and used in real time.

Real-time and online anomaly detection can be of great added value in such use cases by finding values that are far from the expected range of ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Start your free trial

Practical Machine Learning for Data Analysis Using Python

Publisher Resources

ISBN: 9781803248363Supplemental Content

Machine Learning for Streaming Data with Python

by Joos Korstanje

Chapter 5: Online Anomaly Detection

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

You might also like

Practical Machine Learning for Data Analysis Using Python

Interpretable Machine Learning with Python - Second Edition

Machine Learning for Time-Series with Python

Training Data for Machine Learning

Publisher Resources

Chapter 5: Online Anomaly Detection

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,and much more.

You might also like

Practical Machine Learning for Data Analysis Using Python

Interpretable Machine Learning with Python - Second Edition

Machine Learning for Time-Series with Python

Training Data for Machine Learning

Publisher Resources

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.