book

Evaluating Machine Learning Models

Name: Evaluating Machine Learning Models
Author: Alice Zheng
ISBN: 9781491932445

by Alice Zheng

September 2015

Intermediate to advanced

20 pages

1h 20m

English

O'Reilly Media, Inc.

Read now

Unlock full access

Preface
1. Orientation
The Machine Learning WorkflowEvaluation MetricsOffline Evaluation MechanismsHyperparameter SearchOnline Testing Mechanisms
2. Evaluation Metrics
Classification MetricsAccuracyConfusion MatrixPer-Class AccuracyLog-LossAUCRanking MetricsPrecision-RecallPrecision-Recall Curve and the F1 ScoreNDCGRegression MetricsRMSEQuantiles of Errors“Almost Correct” PredictionsCaution: The Difference Between Training Metrics and Evaluation MetricsCaution: Skewed Datasets—Imbalanced Classes, Outliers, and Rare DataRelated ReadingSoftware Packages
3. Offline Evaluation Mechanisms: Hold-Out Validation, Cross-Validation, and Bootstrapping
Unpacking the Prototyping Phase: Training, Validation, Model SelectionWhy Not Just Collect More Data?Hold-Out ValidationCross-ValidationBootstrap and JackknifeCaution: The Difference Between Model Validation and TestingSummaryRelated ReadingSoftware Packages
4. Hyperparameter Tuning
Model Parameters Versus HyperparametersWhat Do Hyperparameters Do?Hyperparameter Tuning MechanismHyperparameter Tuning AlgorithmsGrid SearchRandom SearchSmart Hyperparameter TuningThe Case for Nested Cross-ValidationRelated ReadingSoftware Packages
5. The Pitfalls of A/B Testing
A/B Testing: What Is It?Pitfalls of A/B Testing1. Complete Separation of Experiences2. Which Metric?3. How Much Change Counts as Real Change?4. One-Sided or Two-Sided Test?5. How Many False Positives Are You Willing to Tolerate?6. How Many Observations Do You Need?7. Is the Distribution of the Metric Gaussian?8. Are the Variances Equal?9. What Does the p-Value Mean?10. Multiple Models, Multiple Hypotheses11. How Long to Run the Test?12. Catching Distribution DriftMulti-Armed Bandits: An AlternativeRelated ReadingThat’s All, Folks!

404

Not Found

Either this content does not exist or you do not have access to it. Try searching for something new.

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Evaluating Machine Learning Models

by Alice Zheng

404

Not Found