book

Outlier Detection in Python

Name: Outlier Detection in Python
Author: Brett Kennedy
ISBN: 9781633436473

by Brett Kennedy

December 2024

Beginner to intermediate

560 pages

19h 50m

English

Manning Publications

Read now

Unlock full access

Outlier Detection in Python
Copyright
contents
front matter
prefaceacknowledgmentsabout this bookWho should read this bookHow this book is organized: A road mapAbout the codeliveBook discussion forumabout the authorabout the cover illustration
Part 1.
1 Introducing outlier detection
1.1 Why do outlier detection?1.1.1 Financial fraud1.1.2 Credit card fraud1.1.3 Network security1.1.4 Detecting bots on social media1.1.5 Industrial processes1.1.6 Self-driving vehicles1.1.7 Healthcare1.1.8 Astronomy1.1.9 Data quality1.1.10 Evaluating segmentation1.2 Outlier detection’s place in machine learning1.3 Outlier detection in tabular data1.4 Definitions of outliers1.5 Trends in outlier detection1.6 How does this book teach outlier detection?Summary
2 Simple outlier detection
2.1 One-dimensional numeric outliers2.1.1 z-score2.1.2 Interquartile range2.1.3 Median absolute deviation2.1.4 Modified z-score2.1.5 Visualization for numeric outliers2.1.6 Internal and external outliers2.1.7 Scoring outliers2.2 One-dimensional categorical outliers: Rare values2.3 Multidimensional outliers2.3.1 Types of multidimensional outliers2.3.2 Visualization for multidimensional data2.4 Rare combinations of categorical values2.4.1 Rare combinations using their absolute count2.4.2 Combinations that are rare given their marginal probabilities2.5 Rare combinations of numeric values2.6 Noise vs. inliers and outliers2.7 Local and global outliers2.8 Combining the scores of univariate testsSummary
3 Machine learning-based outlier detection
3.1 The curse of dimensionality3.1.1 Data sparsity3.1.2 Data appearing in the margins3.1.3 Distance calculations3.2 Types of algorithms3.2.1 Distance based3.2.2 Density based3.2.3 Cluster based3.2.4 Frequent item set based3.2.5 Model based3.3 Types of detectors3.3.1 Clean vs. contaminated training data3.3.2 Numeric vs. categorical3.3.3 Local vs. global detectors3.3.4 Scores vs. flags3.3.5 The time required for training and predicting3.3.6 The ability to process many features3.3.7 The parameters requiredSummary
4 The outlier detection process
4.1 Outlier detection workflow4.2 Determining the types of outliers we are interested in4.2.1 Statistical outliers4.2.2 Specific outliers4.2.3 Known and unknown outliers4.3 Choosing the type of model to be used4.3.1 Selecting the category of outlier detector4.3.2 Rules-based approaches4.3.3 Classifier-based approaches4.4 Collecting the data4.5 Examining the data4.6 Cleaning the data4.7 Feature selection4.8 Feature engineering4.9 Encoding categorical values4.10 Scaling numeric values4.11 Fitting a set of models and generating predictions4.12 Evaluating the models4.13 Setting up ongoing outlier detection systems4.14 Refitting the models as necessarySummary
Part 2.

5 Outlier detection using scikit-learn
5.1 Introducing scikit-learn5.2 Isolation Forest5.2.1 The KDD Cup dataset5.2.2 Using Isolation Forest on the KDD Cup dataset5.3 LocalOutlierFactor (LOF)5.4 One-class SVM (OCSVM)5.4.1 OneClassSVM class5.4.2 SGDOneClassSVM5.5 Elliptic Envelope5.5.1 Mahalanobis distance5.5.2 Example using the EllipticEnvelope class5.6 Gaussian mixture models5.7 BallTree and KDTreeSummary
6 The PyOD library
6.1 The PyOD common API6.2 Histogram-based Outlier Score (HBOS)6.3 Empirical Cumulative Distribution Function (ECOD)6.4 Copula-based outlier detection (COPOD)6.5 Angle-based outlier detection (ABOD)6.6 Clustering-based local outlier factor (CBLOF)6.7 Local correlation integral (LOCI)6.8 Connectivity-based outlier factor (COF)6.9 Principal component analysis (PCA)6.9.1 Univariate tests on the components6.9.2 PyODKernelPCA6.9.3 The PCA detector6.9.4 KPCA6.10 Subspace outlier detection6.11 FeatureBagging6.12 Cook’s Distance6.13 Using SUOD for faster model training6.14 The PYOD thresholds moduleSummary
7 Additional libraries and algorithms for outlier detection
7.1 Example synthetic test sets7.2 The alibi-detect library7.3 The PyCaret library7.4 Local outlier probability (LoOP)7.5 Local distance-based outlier factor (LDOF)7.6 Extended Isolation Forest (EIF)7.7 Outlier Detection Using In-degree Number (ODIN)7.8 Clustering7.8.1 Mahalanobis distance per cluster7.8.2 Kernel density estimation per cluster7.8.3 Clustering with categorical data7.9 Entropy7.10 Association Rules7.11 Convex Hull7.12 Distance metric learning (DML)7.13 NearestSampleSummary
Part 3.
8 Evaluating detectors and parameters
8.1 The effect of the number of neighbors8.1.1 2D plots of results8.1.2 1D plots of results8.2 Contour plots8.2.1 Examining parameter choices with contour plots8.2.2 Examining detectors with contour plots8.3 Visualizing subspaces in real-world data8.3.1 2D plots of results on real-world data8.3.2 Contour plots on real-world data8.4 Correlation between detectors with full real-world datasets8.5 Modifying real-world data8.5.1 Adding known anomalies8.5.2 Evaluation metrics8.5.3 Evaluating detectors using accuracy metrics8.5.4 Adjusting the training size used8.5.5 Adding extreme values8.6 Testing with classification datasets8.7 Timing experimentsSummary
9 Working with specific data types
9.1 Null values9.2 Special data types9.2.1 Phone numbers9.2.2 Addresses9.2.3 Email addresses9.2.4 ID/Code values9.2.5 Dates9.2.6 High-cardinality categorical columns9.3 Text features9.3.1 Extracting NLP features9.3.2 Topic modeling9.3.3 Clustering text values9.4 Encoding categorical data9.4.1 One-hot encoding9.4.2 Ordinal encoding9.4.3 Count encoding9.5 Scaling numeric values9.6 Binning numeric data9.7 Distance metrics9.7.1 Distance metrics for numeric data9.7.2 Gower’s distance metric for mixed data9.7.3 Distance metrics for categorical data9.7.4 Using categorical distance metrics for mixed dataSummary
10 Handling very large and very small datasets
10.1 Data with many features10.1.1 Dimensionality reduction10.1.2 Feature subspaces10.2 Data with many rows10.2.1 Training and predicting on data samples10.2.2 Testing models for stability10.2.3 Segmenting data10.2.4 Running in parallel vs. in sequence10.2.5 Tools for working with large datasets10.3 Working with very small datasetsSummary
11 Synthetic data for outlier detection
11.1 Creating synthetic data to represent inliers11.1.1 Testing with realistic inlier data11.1.2 Using realistic synthetic inliers for training11.2 Generating new synthetic data11.2.1 Libraries to generate new synthetic data11.2.2 Using patterns between features11.2.3 Using GMMs11.3 Doping11.4 Simulations11.5 Training classifiers to distinguish real from fake dataSummary
12 Collective outliers
12.1 Purchases data12.2 Preparing the data12.3 Testing for duplicates12.4 Testing for gaps12.5 Testing for missing combinations12.6 Creating new tables to capture collective outliers12.6.1 Aggregating by entity12.6.2 Aggregating by two or more entity types12.6.3 Aggregating by time12.6.4 Aggregating by entity and time12.6.5 Merging in additional information12.7 Identifying trends12.8 Unusual distributions12.9 Rolling windows features12.10 Tests for unusual numbers of point anomaliesSummary
13 Explainable outlier detection
13.1 Introducing XAI13.1.1 Interpretability vs. explainability13.1.2 Global vs. local explanations13.2 Post hoc explanations13.2.1 Feature importances13.2.2 Proxy models13.2.3 Plotting13.2.4 Counterfactuals13.2.5 General notes on post hoc explanations13.3 Interpretable outlier detectors13.3.1 Outlier detection on sets of 2D subspaces13.3.2 Bayesian Histogram-based Anomaly Detection13.3.3 CountsOutlierDetector13.3.4 DataConsistencyCheckerSummary
14 Ensembles of outlier detectors
14.1 Overview of ensembling for outlier detection14.2 Accuracy metrics with ensembles14.3 Methods to create ensembles14.3.1 Different model types14.3.2 Different preprocessing14.3.3 Different hyperparameters14.3.4 Sampling the original features14.3.5 Different engineered features14.3.6 Sampling rows14.4 Selecting detectors for an ensemble14.4.1 Manually inspecting the superset of many detectors14.4.2 Taking the consensus of many detectors14.4.3 Testing random subsets of detectors14.4.4 Greedy methods to identify a set of detectors14.4.5 Selecting detectors to minimize correlation14.5 Scaling scores14.5.1 Min-max scaling14.5.2 z-scores (standard scaling)14.5.3 Robust scaling14.5.4 MAD scaling14.5.5 Ranking14.5.6 Box Cox14.6 Combining scores14.6.1 Mean vs. maximum14.6.2 PyOD ensembling methods14.6.3 Variations of mean and maximum14.6.4 Weighting based on accuracy14.6.5 Stacked ensemblesSummary
15 Working with outlier detection predictions
15.1 Hand-labeling output15.1.1 Hand-labeling specific types of outliers15.1.2 Hand-scoring the outliers15.2 Examining the flagged outliers15.2.1 Manual inspection of the outliers15.2.2 Executing interpretable detectors15.2.3 Examining subspaces of features15.3 Automating the process of sorting outlier detection results15.3.1 Rules to sort the output of detectors15.3.2 Classifiers to sort the output of detectors15.3.3 Executing rules and classifiers on flagged outliers15.4 Semisupervised learning15.4.1 Using labeled data to create a stacked model15.4.2 XGBOD15.5 Regression testingSummary
Part 4.
16 Deep learning-based outlier detection
16.1 Introducing neural networks16.2 PyOD16.2.1 Autoencoders16.2.2 Variational autoencoders16.2.3 Generative adversarial networks16.2.4 SO_GAAL16.2.5 MO_GAAL16.2.6 DeepSVDD16.2.7 Deep IF16.3 Image data16.3.1 Techniques for outlier detection with image data16.3.2 Astronomaly16.4 alibi-detect16.5 Self-supervised learning for outlier detection with tabular data16.5.1 Introducing embeddings16.5.2 Embeddings for outlier detection16.5.3 Transfer learning16.5.4 Self-supervised methods for tabular data16.5.5 DeepODSummary
17 Time-series data
17.1 Cross-sectional, panel, and time-series data17.2 Types of time-series outliers17.2.1 Extreme values17.2.2 Contextual point outliers17.2.3 Outliers based on decomposition17.2.4 Ruptures17.2.5 Outliers based on tabular representations17.2.6 Forecasting-based time-series outlier detection17.2.7 Unusual shapes of windows in timelines17.2.8 Tabular outlier detection on windows of data17.2.9 Mulivariate time-series outliers17.3 Tools for time-series data17.3.1 Anomaly Detection Toolkit17.3.2 DeepOD17.3.3 ProphetSummary
index

Content preview from Outlier Detection in Python

4 The outlier detection process

This chapter covers

Working on an outlier detection project in production
The types of problems we may work with
Where outlier detectors are actually the best option
Collecting and preparing data as well as fitting the models
Evaluating and combining models

We now have a good sense of how outlier detection works generally and how some specific algorithms to identify anomalies work, including statistical and machine learning-based methods. There are, though, a number of steps involved with effectively executing an outlier detection project, which, now that we have a good foundation, we should look at.

In this chapter, we’ll go through the main steps typically involved in outlier detection projects, though they ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Causal Inference and Discovery in Python

Publisher Resources

ISBN: 9781633436473Publisher Support Publisher Website

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Outlier Detection in Python

by Brett Kennedy

4 The outlier detection process

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.