book

Practical Guide to Applied Conformal Prediction in Python

Name: Practical Guide to Applied Conformal Prediction in Python
Author: Valery Manokhin
ISBN: 9781805122760

by Valery Manokhin

December 2023

Intermediate to advanced

240 pages

6h 42m

English

Packt Publishing

Read now

Unlock full access

Practical Guide to Applied Conformal Prediction in Python
ForewordContributorsAbout the authorAbout the reviewers
Preface
Who is this book for?What this book coversTo get the most out of this bookDownload the example code filesConventions usedGet in touchShare Your ThoughtsDownload a free PDF copy of this book
Part 1: Introduction
Chapter 1: Introducing Conformal Prediction
Technical requirementsIntroduction to conformal predictionUnderstanding conformity measuresThe origins of conformal predictionThe future of conformal predictionHow conformal prediction differs from traditional machine learningThe p-value and its role in conformal predictionSummary
Chapter 2: Overview of Conformal Prediction
Understanding uncertainty quantificationAleatoric uncertaintyEpistemic uncertaintyDifferent ways to quantify uncertaintyQuantifying uncertainty using conformal predictionSummary
Part 2: Conformal Prediction Framework
Chapter 3: Fundamentals of Conformal Prediction
Fundamentals of conformal predictionDefinition and principlesBasic components of a conformal predictorTypes of nonconformity measuresSummary
Chapter 4: Validity and Efficiency of Conformal Prediction
The validity of probabilistic predictorsClassifier calibrationThe efficiency of probabilistic predictorsSummary
Chapter 5: Types of Conformal Predictors
Understanding classical predictorsApplying TCP for classification problemsApplying TCP for regression problemsAdvantagesUnderstanding inductive conformal predictorsChoosing the right conformal predictorTransductive conformal predictorsInductive conformal predictorsSummary
Part 3: Applications of Conformal Prediction

Chapter 6: Conformal Prediction for Classification
Classifier calibrationUnderstanding the concepts of classifier calibrationEvaluating calibration performanceVarious approaches to classifier calibrationHistogram binningPlatt scalingIsotonic regressionConformal prediction for classifier calibrationVenn-ABERS conformal predictionComparing calibration methodsOpen source tools for conformal prediction in classification problemsNonconformistSummary
Chapter 7: Conformal Prediction for Regression
Uncertainty quantification for regression problemsUnderstanding the types and sources of uncertainty in regression modelingThe concept of prediction intervalsWhy do we need prediction intervals?How is it different from a confidence interval?Conformal prediction for regression problemsBuilding prediction intervals and predictive distributions using conformal predictionMechanics of CQRQuantile regressionCQRJackknife+Jackknife regressionJackknife+ regressionConformal predictive distributionsSummary
Chapter 8: Conformal Prediction for Time Series and Forecasting
UQ for time series and forecasting problemsThe importance of UQThe history of UQEarly statistical methods – the roots of UQ in time seriesModern machine learning approachesThe concept of PIs in forecasting applicationsDefinition and constructionThe importance of forecasting applicationsChallenges and considerationsVarious approaches to producing PIsParametric approachesNon-parametric approachesBayesian approachesMachine learning approachesConformal predictionConformal prediction for time series and forecastingEnsemble batch PIs (EnbPIs)NeuralProphetSummary
Chapter 9: Conformal Prediction for Computer Vision
Uncertainty quantification for computer visionWhy does uncertainty matter?Types of uncertainty in computer visionQuantifying uncertaintyWhy does deep learning produce miscalibrated predictions?Post-2012 – the deep learning surgeThe "calibration crisis" in deep learning – a turning point in 2017Overconfidence in modern deep learning computer vision modelsVarious approaches to quantify uncertainty in computer vision problemsThe superiority of conformal prediction in uncertainty quantificationConformal prediction for computer visionUncertainty sets for image classifiers using conformal predictionBuilding computer vision classifiers using conformal predictionNaïve Conformal predictionAdaptive Prediction Sets (APS)Regularized Adaptive Prediction Sets (RAPS)Summary
Chapter 10: Conformal Prediction for Natural Language Processing
Uncertainty quantification for NLPWhat is uncertainty in NLP?Benefits of quantifying uncertainty in NLPThe challenges of uncertainty in NLPUnderstanding why deep learning produces miscalibrated predictionsIntroduction to deep learning in NLPChallenges with deep learning predictions in NLPThe implications of miscalibrationVarious approaches to quantify uncertainty in NLP problemsBootstrap methods and ensemble techniquesOut-of-distribution (OOD) detectionConformal prediction for NLPHow conformal prediction works in NLPPractical applications of conformal prediction in NLPAdvantages of using conformal prediction in NLPSummary
Part 4: Advanced Topics
Chapter 11: Handling Imbalanced Data
Introducing imbalanced dataWhy imbalanced data problems are complex to solveMethods for solving imbalanced dataThe methods for solving imbalanced dataSolving imbalanced data problems by applying conformal predictionAddressing imbalanced data with Venn-Abers predictorsKey insights from the Credit Card Fraud Detection notebookSummary
Chapter 12: Multi-Class Conformal Prediction
Multi-class classification problemsAlgorithms for multi-class classificationOne-vs-all and one-vs-one strategiesMetrics for multi-class classification problemsConfusion matrixPrecisionRecallF1 scoreMacro- and micro-averaged metricsArea Under Curve (AUC-ROC)Log loss and its application in measuring calibration of multi-class modelsBrier score and its application in measuring the calibration of multi-class modelsHow conformal prediction can be applied to multi-class classification problemsMulti-class probabilistic classification using inductive and cross-Venn-ABERS predictorsSummary
Index
Why subscribe?
Other Books You May EnjoyPackt is searching for authors like youShare Your ThoughtsDownload a free PDF copy of this book

Content preview from Practical Guide to Applied Conformal Prediction in Python

10 Conformal Prediction for Natural Language Processing

Natural language processing (NLP) grapples with the complexities of human language, where uncertainty is an inherent challenge. As NLP models become integral to risk-sensitive and critical applications, ensuring their reliability is paramount. Conformal prediction emerges as a promising technique, offering a way to quantify the trustworthiness of these models’ predictions, particularly when faced with miscalibrated outputs from deep learning models.

In this chapter, we will navigate the NLP conformal prediction world, understand its significance, and learn how to harness its power for more reliable and confident predictions.

In this chapter, we’re going to cover the following main topics: ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Machine Learning for Time-Series with Python

Publisher Resources

ISBN: 9781805122760

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Practical Guide to Applied Conformal Prediction in Python

by Valery Manokhin

10

Conformal Prediction for Natural Language Processing

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.