book

Machine Learning with the Elastic Stack - Second Edition

Name: Machine Learning with the Elastic Stack - Second Edition
ISBN: 9781801070034

by Rich Collier, Camilla Montonen, Bahaaldine Azarmi

May 2021

Intermediate to advanced

450 pages

9h 36m

English

Packt Publishing

Read now

Unlock full access

Machine Learning with the Elastic Stack
Second EditionContributorsAbout the authorsAbout the reviewers
Preface
Who this book is forWhat this book coversTo get the most out of this bookDownload the example code filesDownload the color imagesConventions usedGet in touchReviews
Section 1 – Getting Started with Machine Learning with Elastic Stack
Chapter 1: Machine Learning for IT
Overcoming the historical challenges in ITDealing with the plethora of dataThe advent of automated anomaly detectionUnsupervised versus supervised MLUsing unsupervised ML for anomaly detectionDefining unusualLearning what's normalProbability modelsLearning the modelsDe-trendingScoring of unusualnessThe element of timeApplying supervised ML to data frame analyticsThe process of supervised learningSummary
Chapter 2: Enabling and Operationalization
Technical requirementsEnabling Elastic ML featuresEnabling ML on a self-managed clusterEnabling ML in the cloud – Elasticsearch ServiceUnderstanding operationalizationML nodesJobsBucketing data in a time series analysisFeeding data to Elastic MLThe supporting indicesAnomaly detection orchestrationAnomaly detection model snapshotsSummary
Section 2 – Time Series Analysis – Anomaly Detection and Forecasting
Chapter 3: Anomaly Detection
Technical requirementsElastic ML job typesDissecting the detectorThe functionThe fieldThe partition field The by fieldThe over fieldThe "formula"Exploring the count functionsOther counting functionsDetecting changes in metric valuesMetric functionsUnderstanding the advanced detector functionsrareFrequency rareInformation contentGeographicTimeSplitting analysis along categorical featuresSetting the split fieldThe difference between splitting using partition and by_fieldUnderstanding temporal versus population analysisCategorization analysis of unstructured messagesTypes of messages that are good candidates for categorizationThe process used by categorization Analyzing the categoriesCategorization job exampleWhen to avoid using categorizationManaging Elastic ML via the APISummary
Chapter 4: Forecasting
Technical requirementsContrasting forecasting with prophesyingForecasting use casesForecasting theory of operationSingle time series forecastingLooking at forecast resultsMultiple time series forecastingSummary
Chapter 5: Interpreting Results
Technical requirementsViewing the Elastic ML results indexAnomaly scoresBucket-level scoringNormalizationInfluencer-level scoringInfluencersRecord-level scoringResults index schema detailsBucket resultsRecord resultsInfluencer resultsMulti-bucket anomaliesMulti-bucket anomaly exampleMulti-bucket scoringForecast resultsQuerying for forecast resultsResults APIResults API endpointsGetting the overall buckets APIGetting the categories APICustom dashboards and Canvas workpadsDashboard "embeddables"Anomalies as annotations in TSVBCustomizing Canvas workpadsSummary
Chapter 6: Alerting on ML Analysis
Technical requirementsUnderstanding alerting conceptsAnomalies are not necessarily alertsIn real-time alerting, timing mattersBuilding alerts from the ML UIDefining sample anomaly detection jobsCreating alerts against the sample jobsSimulating some real-time anomalous behaviorReceiving and reviewing the alertsCreating an alert with a watchUnderstanding the anatomy of the legacy default ML watch Custom watches can offer some unique functionalitySummary

Chapter 7: AIOps and Root Cause Analysis
Technical requirementsDemystifying the term ''AIOps''Understanding the importance and limitations of KPIsMoving beyond KPIsOrganizing data for better analysisCustom queries for anomaly detection datafeedsData enrichment on ingestLeveraging the contextual informationAnalysis splitsStatistical influencersBringing it all together for RCAOutage backgroundCorrelation and shared influencersSummary
Chapter 8: Anomaly Detection in Other Elastic Stack Apps
Technical requirementsAnomaly detection in Elastic APMEnabling anomaly detection for APMViewing the anomaly detection job results in the APM UICreating ML Jobs via the data recognizerAnomaly detection in the Logs appLog categoriesLog anomaliesAnomaly detection in the Metrics appAnomaly detection in the Uptime appAnomaly detection in the Elastic Security appPrebuilt anomaly detection jobsAnomaly detection jobs as detection alertsSummary
Section 3 – Data Frame Analysis
Chapter 9: Introducing Data Frame Analytics
Technical requirementsLearning how to use transformsWhy are transforms useful?The anatomy of a transformUsing transforms to analyze e-commerce ordersExploring more advanced pivot and aggregation configurationsDiscovering the difference between batch and continuous transformsAnalyzing social media feeds using continuous transformsUsing Painless for advanced transform configurationsIntroducing PainlessWorking with Python and Elasticsearch A brief tour of the Python Elasticsearch clientsSummaryFurther reading
Chapter 10: Outlier Detection
Technical requirementsDiscovering the four techniques used for outlier detectionUnderstanding feature influenceHow does outlier detection differ from anomaly detection? Applying outlier detection in practiceEvaluating outlier detection with the Evaluate API Hyperparameter tuning for outlier detection Summary
Chapter 11: Classification Analysis
Technical requirementsClassification: from data to a trained modelFeature engineeringEvaluating the model Taking your first steps with classificationClassification under the hood: gradient boosted decision treesIntroduction to decision treesGradient boosted decision treesHyperparametersInterpreting resultsSummaryFurther reading
Chapter 12: Regression
Technical requirementsUsing regression analysis to predict house pricesUsing decision trees for regression Summary Further reading
Chapter 13: Inference
Technical requirementsExamining, exporting, and importing your trained models with the Trained Models APIA tour of the Trained Models APIExporting and importing trained models with the Trained Models API and Python Understanding inference processors and ingest pipelines Handling missing or corrupted data in ingest pipelines Using inference processor configuration options to gain more insight into your predictionsImporting external models into Elasticsearch using elandLearning about supported external models in eland Training a scikit-learn DecisionTreeClassifier and importing it into Elasticsearch using elandSummary
Appendix: Anomaly Detection Tips
Technical requirementsUnderstanding influencers in split versus non-split jobsUsing one-sided functions to your advantageIgnoring time periodsIgnoring an upcoming (known) window of timeIgnoring an unexpected window of time, after the factUsing custom rules and filters to your advantageCreating custom rulesBenefiting from custom rules for a "top-down" alerting philosophyAnomaly detection job throughput considerationsAvoiding the over-engineering of a use caseUsing anomaly detection on runtime fieldsSummary
Why subscribe?
Other Books You May EnjoyPackt is searching for authors like youLeave a review - let other readers know what you think

Content preview from Machine Learning with the Elastic Stack - Second Edition

Chapter 6: Alerting on ML Analysis

The previous chapter (Chapter 5, Interpreting Results) explained in depth how anomaly detection and forecasting results are stored in Elasticsearch indices. This gives us the proper background to now create proactive, actionable, and informative alerts on those results.

At the time of writing this book, we find ourselves at an inflection point. For several years, Elastic ML has relied on the alerting capabilities of Watcher (a component of Elasticsearch) as this was the exclusive mechanism to alert on data. However, a new platform of alerting has been designed as part of Kibana (and was deemed GA in v7.11) and this new approach will be the primary mechanism of alerting moving forward.

There are still some ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Publisher Resources

ISBN: 9781801070034

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Machine Learning with the Elastic Stack - Second Edition

by Rich Collier, Camilla Montonen, Bahaaldine Azarmi

Chapter 6: Alerting on ML Analysis

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.