book

Human-in-the-Loop Machine Learning

Name: Human-in-the-Loop Machine Learning
Author: Rob Munro
ISBN: 9781617296741

by Rob Munro

July 2021

Intermediate to advanced

424 pages

14h 7m

English

Manning Publications

Read now

Unlock full access

inside front cover
Human-in-the-Loop Machine Learning
Copyright
brief contents
contents
front matter
forewordprefaceacknowledgmentsabout this bookWho should read this bookHow this book is organized: A road mapAbout the codeliveBook discussion forumOther online resourcesabout the author
Part 1 First steps
1 Introduction to human-in-the-loop machine learning
1.1 The basic principles of human-in-the-loop machine learning1.2 Introducing annotation1.2.1 Simple and more complicated annotation strategies1.2.2 Plugging the gap in data science knowledge1.2.3 Quality human annotation: Why is it hard?1.3 Introducing active learning: Improving the speed and reducing the cost of training data1.3.1 Three broad active learning sampling strategies: Uncertainty, diversity, and random1.3.2 What is a random selection of evaluation data?1.3.3 When to use active learning1.4 Machine learning and human–computer interaction1.4.1 User interfaces: How do you create training data?1.4.2 Priming: What can influence human perception?1.4.3 The pros and cons of creating labels by evaluating machine learning predictions1.4.4 Basic principles for designing annotation interfaces1.5 Machine-learning-assisted humans vs. human-assisted machine learning1.6 Transfer learning to kick-start your models1.6.1 Transfer learning in computer vision1.6.2 Transfer learning in NLP1.7 What to expect in this textSummary
2 Getting started with human-in-the-loop machine learning
2.1 Beyond hacktive learning: Your first active learning algorithm2.2 The architecture of your first system2.3 Interpreting model predictions and data to support active learning2.3.1 Confidence ranking2.3.2 Identifying outliers2.3.3 What to expect as you iterate2.4 Building an interface to get human labels2.4.1 A simple interface for labeling text2.4.2 Managing machine learning data2.5 Deploying your first human-in-the-loop machine learning system2.5.1 Always get your evaluation data first2.5.2 Every data point gets a chance2.5.3 Select the right strategies for your data2.5.4 Retrain the model and iterateSummary
Part 2 Active learning

3 Uncertainty sampling
3.1 Interpreting uncertainty in a machine learning model3.1.1 Why look for uncertainty in your model?3.1.2 Softmax and probability distributions3.1.3 Interpreting the success of active learning3.2 Algorithms for uncertainty sampling3.2.1 Least confidence sampling3.2.2 Margin of confidence sampling3.2.3 Ratio sampling3.2.4 Entropy (classification entropy)3.2.5 A deep dive on entropy3.3 Identifying when different types of models are confused3.3.1 Uncertainty sampling with logistic regression and MaxEnt models3.3.2 Uncertainty sampling with SVMs3.3.3 Uncertainty sampling with Bayesian models3.3.4 Uncertainty sampling with decision trees and random forests3.4 Measuring uncertainty across multiple predictions3.4.1 Uncertainty sampling with ensemble models3.4.2 Query by Committee and dropouts3.4.3 The difference between aleatoric and epistemic uncertainty3.4.4 Multilabeled and continuous value classification3.5 Selecting the right number of items for human review3.5.1 Budget-constrained uncertainty sampling3.5.2 Time-constrained uncertainty sampling3.5.3 When do I stop if I’m not time- or budget-constrained?3.6 Evaluating the success of active learning3.6.1 Do I need new test data?3.6.2 Do I need new validation data?3.7 Uncertainty sampling cheat sheet3.8 Further reading3.8.1 Further reading for least confidence sampling3.8.2 Further reading for margin of confidence sampling3.8.3 Further reading for ratio of confidence sampling3.8.4 Further reading for entropy-based sampling3.8.5 Further reading for other machine learning models3.8.6 Further reading for ensemble-based uncertainty samplingSummary
4 Diversity sampling
4.1 Knowing what you don’t know: Identifying gaps in your model’s knowledge4.1.1 Example data for diversity sampling4.1.2 Interpreting neural models for diversity sampling4.1.3 Getting information from hidden layers in PyTorch4.2 Model-based outlier sampling4.2.1 Use validation data to rank activations4.2.2 Which layers should I use to calculate model-based outliers?4.2.3 The limitations of model-based outliers4.3 Cluster-based sampling4.3.1 Cluster members, centroids, and outliers4.3.2 Any clustering algorithm in the universe4.3.3 K-means clustering with cosine similarity4.3.4 Reduced feature dimensions via embeddings or PCA4.3.5 Other clustering algorithms4.4 Representative sampling4.4.1 Representative sampling is rarely used in isolation4.4.2 Simple representative sampling4.4.3 Adaptive representative sampling4.5 Sampling for real-world diversity4.5.1 Common problems in training data diversity4.5.2 Stratified sampling to ensure diversity of demographics4.5.3 Represented and representative: Which matters?4.5.4 Per-demographic accuracy4.5.5 Limitations of sampling for real-world diversity4.6 Diversity sampling with different types of models4.6.1 Model-based outliers with different types of models4.6.2 Clustering with different types of models4.6.3 Representative sampling with different types of models4.6.4 Sampling for real-world diversity with different types of models4.7 Diversity sampling cheat sheet4.8 Further reading4.8.1 Further reading for model-based outliers4.8.2 Further reading for cluster-based sampling4.8.3 Further reading for representative sampling4.8.4 Further reading for sampling for real-world diversitySummary
5 Advanced active learning
5.1 Combining uncertainty sampling and diversity sampling5.1.1 Least confidence sampling with cluster-based sampling5.1.2 Uncertainty sampling with model-based outliers5.1.3 Uncertainty sampling with model-based outliers and clustering5.1.4 Representative sampling cluster-based sampling5.1.5 Sampling from the highest-entropy cluster5.1.6 Other combinations of active learning strategies5.1.7 Combining active learning scores5.1.8 Expected error reduction sampling5.2 Active transfer learning for uncertainty sampling5.2.1 Making your model predict its own errors5.2.2 Implementing active transfer learning5.2.3 Active transfer learning with more layers5.2.4 The pros and cons of active transfer learning5.3 Applying active transfer learning to representative sampling5.3.1 Making your model predict what it doesn’t know5.3.2 Active transfer learning for adaptive representative sampling5.3.3 The pros and cons of active transfer learning for representative sampling5.4 Active transfer learning for adaptive sampling5.4.1 Making uncertainty sampling adaptive by predicting uncertainty5.4.2 The pros and cons of ATLAS5.5 Advanced active learning cheat sheets5.6 Further reading for active transfer learningSummary
6 Applying active learning to different machine learning tasks
6.1 Applying active learning to object detection6.1.1 Accuracy for object detection: Label confidence and localization6.1.2 Uncertainty sampling for label confidence and localization in object detection6.1.3 Diversity sampling for label confidence and localization in object detection6.1.4 Active transfer learning for object detection6.1.5 Setting a low object detection threshold to avoid perpetuating bias6.1.6 Creating training data samples for representative sampling that are similar to your predictions6.1.7 Sampling for image-level diversity in object detection6.1.8 Considering tighter masks when using polygons6.2 Applying active learning to semantic segmentation6.2.1 Accuracy for semantic segmentation6.2.2 Uncertainty sampling for semantic segmentation6.2.3 Diversity sampling for semantic segmentation6.2.4 Active transfer learning for semantic segmentation6.2.5 Sampling for image-level diversity in semantic segmentation6.3 Applying active learning to sequence labeling6.3.1 Accuracy for sequence labeling6.3.2 Uncertainty sampling for sequence labeling6.3.3 Diversity sampling for sequence labeling6.3.4 Active transfer learning for sequence labeling6.3.5 Stratified sampling by confidence and tokens6.3.6 Create training data samples for representative sampling that are similar to your predictions6.3.7 Full-sequence labeling6.3.8 Sampling for document-level diversity in sequence labeling6.4 Applying active learning to language generation6.4.1 Calculating accuracy for language generation systems6.4.2 Uncertainty sampling for language generation6.4.3 Diversity sampling for language generation6.4.4 Active transfer learning for language generation6.5 Applying active learning to other machine learning tasks6.5.1 Active learning for information retrieval6.5.2 Active learning for video6.5.3 Active learning for speech6.6 Choosing the right number of items for human review6.6.1 Active labeling for fully or partially annotated data6.6.2 Combining machine learning with annotation6.7 Further readingSummary
Part 3 Annotation
7 Working with the people annotating your data
7.1 Introduction to annotation7.1.1 Three principles of good data annotation7.1.2 Annotating data and reviewing model predictions7.1.3 Annotations from machine learning-assisted humans7.2 In-house experts7.2.1 Salary for in-house workers7.2.2 Security for in-house workers7.2.3 Ownership for in-house workers7.2.4 Tip: Always run in-house annotation sessions7.3 Outsourced workers7.3.1 Salary for outsourced workers7.3.2 Security for outsourced workers7.3.3 Ownership for outsourced workers7.3.4 Tip: Talk to your outsourced workers7.4 Crowdsourced workers7.4.1 Salary for crowdsourced workers7.4.2 Security for crowdsourced workers7.4.3 Ownership for crowdsourced workers7.4.4 Tip: Create a path to secure work and career advancement7.5 Other workforces7.5.1 End users7.5.2 Volunteers7.5.3 People playing games7.5.4 Model predictions as annotations7.6 Estimating the volume of annotation needed7.6.1 The orders-of-magnitude equation for number of annotations needed7.6.2 Anticipate one to four weeks of annotation training and task refinement7.6.3 Use your pilot annotations and accuracy goal to estimate cost7.6.4 Combining types of workforcesSummary
8 Quality control for data annotation
8.1 Comparing annotations with ground truth answers8.1.1 Annotator agreement with ground truth data8.1.2 Which baseline should you use for expected accuracy?8.2 Interannotator agreement8.2.1 Introduction to interannotator agreement8.2.2 Benefits from calculating interannotator agreement8.2.3 Dataset-level agreement with Krippendorff’s alpha8.2.4 Calculating Krippendorff’s alpha beyond labeling8.2.5 Individual annotator agreement8.2.6 Per-label and per-demographic agreement8.2.7 Extending accuracy with agreement for real-world diversity8.3 Aggregating multiple annotations to create training data8.3.1 Aggregating annotations when everyone agrees8.3.2 The mathematical case for diverse annotators and low agreement8.3.3 Aggregating annotations when annotators disagree8.3.4 Annotator-reported confidences8.3.5 Deciding which labels to trust: Annotation uncertainty8.4 Quality control by expert review8.4.1 Recruiting and training qualified people8.4.2 Training people to become experts8.4.3 Machine-learning-assisted experts8.5 Multistep workflows and review tasks8.6 Further readingSummary
9 Advanced data annotation and augmentation
9.1 Annotation quality for subjective tasks9.1.1 Requesting annotator expectations9.1.2 Assessing viable labels for subjective tasks9.1.3 Trusting an annotator to understand diverse responses9.1.4 Bayesian Truth Serum for subjective judgments9.1.5 Embedding simple tasks in more complicated ones9.2 Machine learning for annotation quality control9.2.1 Calculating annotation confidence as an optimization task9.2.2 Converging on label confidence when annotators disagree9.2.3 Predicting whether a single annotation is correct9.2.4 Predicting whether a single annotation is in agreement9.2.5 Predicting whether an annotator is a bot9.3 Model predictions as annotations9.3.1 Trusting annotations from confident model predictions9.3.2 Treating model predictions as a single annotator9.3.3 Cross-validating to find mislabeled data9.4 Embeddings and contextual representations9.4.1 Transfer learning from an existing model9.4.2 Representations from adjacent easy-to-annotate tasks9.4.3 Self-supervision: Using inherent labels in the data9.5 Search-based and rule-based systems9.5.1 Data filtering with rules9.5.2 Training data search9.5.3 Masked feature filtering9.6 Light supervision on unsupervised models9.6.1 Adapting an unsupervised model to a supervised model9.6.2 Human-guided exploratory data analysis9.7 Synthetic data, data creation, and data augmentation9.7.1 Synthetic data9.7.2 Data creation9.7.3 Data augmentation9.8 Incorporating annotation information into machine learning models9.8.1 Filtering or weighting items by confidence in their labels9.8.2 Including the annotator identity in inputs9.8.3 Incorporating uncertainty into the loss function9.9 Further reading for advanced annotation9.9.1 Further reading for subjective data9.9.2 Further reading for machine learning for annotation quality control9.9.3 Further reading for embeddings/contextual representations9.9.4 Further reading for rule-based systems9.9.5 Further reading for incorporating uncertainty in annotations into the downstream modelsSummary
10 Annotation quality for different machine learning tasks
10.1 Annotation quality for continuous tasks10.1.1 Ground truth for continuous tasks10.1.2 Agreement for continuous tasks10.1.3 Subjectivity in continuous tasks10.1.4 Aggregating continuous judgments to create training data10.1.5 Machine learning for aggregating continuous tasks to create training data10.2 Annotation quality for object detection10.2.1 Ground truth for object detection10.2.2 Agreement for object detection10.2.3 Dimensionality and accuracy in object detection10.2.4 Subjectivity for object detection10.2.5 Aggregating object annotations to create training data10.2.6 Machine learning for object annotations10.3 Annotation quality for semantic segmentation10.3.1 Ground truth for semantic segmentation annotation10.3.2 Agreement for semantic segmentation10.3.3 Subjectivity for semantic segmentation annotations10.3.4 Aggregating semantic segmentation to create training data10.3.5 Machine learning for aggregating semantic segmentation tasks to create training data10.4 Annotation quality for sequence labeling10.4.1 Ground truth for sequence labeling10.4.2 Ground truth for sequence labeling in truly continuous data10.4.3 Agreement for sequence labeling10.4.4 Machine learning and transfer learning for sequence labeling10.4.5 Rule-based, search-based, and synthetic data for sequence labeling10.5 Annotation quality for language generation10.5.1 Ground truth for language generation10.5.2 Agreement and aggregation for language generation10.5.3 Machine learning and transfer learning for language generation10.5.4 Synthetic data for language generation10.6 Annotation quality for other machine learning tasks10.6.1 Annotation for information retrieval10.6.2 Annotation for multifield tasks10.6.3 Annotation for video10.6.4 Annotation for audio data10.7 Further reading for annotation quality for different machine learning tasks10.7.1 Further reading for computer vision10.7.2 Further reading for annotation for natural language processing10.7.3 Further reading for annotation for information retrievalSummary
Part 4 Human–computer interaction for machine learning
11 Interfaces for data annotation
11.1 Basic principles of human–computer interaction11.1.1 Introducing affordance, feedback, and agency11.1.2 Designing interfaces for annotation11.1.3 Minimizing eye movement and scrolling11.1.4 Keyboard shortcuts and input devices11.2 Breaking the rules effectively11.2.1 Scrolling for batch annotation11.2.2 Foot pedals11.2.3 Audio inputs11.3 Priming in annotation interfaces11.3.1 Repetition priming11.3.2 Where priming hurts11.3.3 Where priming helps11.4 Combining human and machine intelligence11.4.1 Annotator feedback11.4.2 Maximizing objectivity by asking what other people would annotate11.4.3 Recasting continuous problems as ranking problems11.5 Smart interfaces for maximizing human intelligence11.5.1 Smart interfaces for semantic segmentation11.5.2 Smart interfaces for object detection11.5.3 Smart interfaces for language generation11.5.4 Smart interfaces for sequence labeling11.6 Machine learning to assist human processes11.6.1 The perception of increased efficiency11.6.2 Active learning for increased efficiency11.6.3 Errors can be better than absence to maximize completeness11.6.4 Keep annotation interfaces separate from daily work interfaces11.7 Further readingSummary
12 Human-in-the-loop machine learning products
12.1 Defining products for human-in-the-loop machine learning applications12.1.1 Start with the problem you are solving12.1.2 Design systems to solve the problem12.1.3 Connecting Python and HTML12.2 Example 1: Exploratory data analysis for news headlines12.2.1 Assumptions12.2.2 Design and implementation12.2.3 Potential extensions12.3 Example 2: Collecting data about food safety events12.3.1 Assumptions12.3.2 Design and implementation12.3.3 Potential extensions12.4 Example 3: Identifying bicycles in images12.4.1 Assumptions12.4.2 Design and implementation12.4.3 Potential extensions12.5 Further reading for building human-in-the-loop machine learning productsSummary
appendix Machine learning refresher
A.1 Interpreting predictions from a modelA.1.1 Probability distributionsA.2 Softmax deep diveA.2.1 Converting the model output to confidences with softmaxA.2.2 The choice of base/temperature for softmaxA.2.3 The result from dividing exponentialsA.3 Measuring human-in-the-loop machine learning systemsA.3.1 Precision, recall, and F-scoreA.3.2 Micro and macro precision, recall, and F-scoreA.3.3 Taking random chance into account: Chance-adjusted accuracyA.3.4 Taking confidence into account: Area under the ROC curve (AUC)A.3.5 Number of model errors spottedA.3.6 Human labor cost savedA.3.7 Other methods for calculating accuracy in this book
index
inside back cover

Overview

Most machine learning systems that are deployed in the world today learn from human feedback. However, most machine learning courses focus almost exclusively on the algorithms, not the human-computer interaction part of the systems. This can leave a big knowledge gap for data scientists working in real-world machine learning, where data scientists spend more time on data management than on building algorithms. Human-in-the-Loop Machine Learning is a practical guide to optimizing the entire machine learning process, including techniques for annotation, active learning, transfer learning, and using machine learning to optimize every step of the process.

About the Technology
Machine learning applications perform better with human feedback. Keeping the right people in the loop improves the accuracy of models, reduces errors in data, lowers costs, and helps you ship models faster.

About the Book
Human-in-the-Loop Machine Learning lays out methods for humans and machines to work together effectively. You’ll find best practices on selecting sample data for human feedback, quality control for human annotations, and designing annotation interfaces. You’ll learn to create training data for labeling, object detection, and semantic segmentation, sequence labeling, and more. The book starts with the basics and progresses to advanced techniques like transfer learning and self-supervision within annotation workflows.

What's Inside

Identifying the right training and evaluation data
Finding and managing people to annotate data
Selecting annotation quality control strategies
Designing interfaces to improve accuracy and efficiency

About the Reader

About the Author
Robert (Munro) Monarch is a data scientist and engineer who has built machine learning data for companies such as Apple, Amazon, Google, and IBM. He holds a PhD from Stanford.

Robert holds a PhD from Stanford focused on Human-in-the-Loop machine learning for healthcare and disaster response, and is a disaster response professional in addition to being a machine learning professional. A worked example throughout this text is classifying disaster-related messages from real disasters that Robert has helped respond to in the past.

Quotes
An excellent reference for learning about active learning from both practical and theoretical perspectives.
- Sayak Paul, PyimageSearch

This book is a must-read for all practitioners who want to integrate active learning in their development cycles. Well written, good coverage, and filled with practical examples.
- Vidhya Vinay, Streamlingo.ai

Very apt real-world examples. I will recommend this book to my colleagues.
- Rajesh Kumar R S, Mindtree

The best reference about supervised machine learning from a real expert and practitioner.
- Michal Rutka, MQL Service

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Publisher Resources

ISBN: 9781617296741Publisher Support Publisher Website Errata Page

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills