book

Training Data for Machine Learning

by Anthony Sarkis

November 2023

Beginner to intermediate

329 pages

9h 3m

English

O'Reilly Media, Inc.

Book available

Read now

Unlock full access

Who Should Read This Book?For the Technical Professional and EngineerFor the Manager and DirectorFor the Subject Matter Expert and Data Annotation SpecialistFor the Data ScientistWhy I Wrote This BookHow This Book Is OrganizedThemesThe Basics and Getting StartedConcepts and TheoriesPutting It All TogetherConventions Used in This BookO’Reilly Online LearningHow to Contact UsAcknowledgments
Training Data IntentsWhat Can You Do With Training Data?What Is Training Data Most Concerned With?Training Data OpportunitiesBusiness TransformationTraining Data EfficiencyTooling ProficiencyProcess Improvement OpportunitiesWhy Training Data MattersML Applications Are Becoming MainstreamThe Foundation of Successful AITraining Data Is Here to StayTraining Data Controls the ML ProgramNew Types of UsersTraining Data in the WildWhat Makes Training Data Difficult?The Art of Supervising MachinesA New Thing for Data ScienceML Program EcosystemData-Centric Machine LearningFailuresHistory of Development Affects Training Data TooWhat Training Data Is NotGenerative AIHuman Alignment Is Human SupervisionSummary
IntroductionGetting Up and RunningInstallationTasks SetupAnnotator SetupData SetupWorkflow SetupData Catalog SetupInitial UsageOptimizationTools OverviewTraining Data for Machine LearningGrowing Selection of ToolsPeople, Process, and DataEmbedded SupervisionHuman Computer SupervisionSeparation of End ConcernsStandardsMany PersonasA Paradigm to Deliver Machine Learning SoftwareTrade-OffsCostsInstalled Versus Software as a ServiceDevelopment SystemScaleInstallation OptionsAnnotation InterfacesModeling IntegrationMulti-User versus Single-User SystemsIntegrationsScopeHidden AssumptionsSecurityOpen Source and Closed SourceHistoryOpen Source StandardsRealizing the Need for Dedicated ToolingSummary
Schema Deep Dive IntroductionLabels and Attributes—What Is It?What Do We Care About?Introduction to LabelsAttributes IntroductionAttribute Complexity Exceeds Spatial ComplexityTechnical OverviewSpatial Representation—Where Is It?Using Spatial Types to Prevent Social BiasTrade-Offs with TypesComputer Vision Spatial Type ExamplesRelationships, Sequences, Time Series: When Is It?Sequences and RelationshipsWhenGuides and InstructionsJudgment CallsRelation of Machine Learning Tasks to Training DataSemantic SegmentationImage Classification (Tags)Object DetectionPose EstimationRelationship of Tasks to Training Data TypesGeneral ConceptsInstance Concept RefresherUpgrading Data Over TimeThe Boundary Between Modeling and Training DataRaw Data ConceptsSummary
IntroductionWho Wants the Data?A Game of TelephonePlanning a Great SystemNaive and Training Data–Centric ApproachesRaw Data StorageBy Reference or by ValueOff-the-Shelf Dedicated Training Data Tooling on Your Own HardwareData Storage: Where Does the Data Rest?External Reference ConnectionRaw Media (BLOB)–Type SpecificFormatting and MappingUser-Defined Types (Compound Files)Defining DataMapsIngest WizardsOrganizing Data and Useful StorageRemote StorageVersioningData AccessDisambiguating Storage, Ingestion, Export, and AccessFile-Based ExportsStreaming DataQueries IntroductionIntegrations with the EcosystemSecurityAccess ControlIdentity and AuthorizationExample of Setting PermissionsSigned URLsPersonally Identifiable InformationPre-LabelingUpdating DataSummary
IntroductionGlue Between Tech and PeopleWhy Are Human Tasks Needed?Partnering with Non-Software Users in New WaysGetting Started with Human TasksBasicsSchemas’ Staying PowerUser RolesTrainingGold Standard TrainingTask Assignment ConceptsDo You Need to Customize the Interface?How Long Will the Average Annotator Be Using It?Tasks and Project StructureQuality AssuranceAnnotator TrustAnnotators Are PartnersCommon Causes of Training Data ErrorsTask Review LoopsAnalyticsAnnotation Metrics ExamplesData ExplorationModelsUsing the Model to Debug the HumansDistinctions Between a Dataset, Model, and Model RunGetting Data to ModelsDataflowOverview of StreamingData OrganizationPipelines and ProcessesDirect AnnotationBusiness Process IntegrationAttributesDepth of LabelingSupervising Existing DataInteractive AutomationsExample: Semantic Segmentation Auto BorderingVideoSummary
IntroductionTheoriesA System Is Only as Useful as Its SchemaWho Supervises the Data MattersIntentionally Chosen Data Is BestWorking with Historical DataTraining Data Is Like CodeSurface Assumptions Around Usage of Your Training DataHuman Supervision Is Different from Classic DatasetsGeneral ConceptsData RelevancyNeed for Both Qualitative and Quantitative EvaluationsIterationsPrioritization: What to LabelTransfer Learning’s Relation to Datasets (Fine-Tuning)Per-Sample Judgment CallsEthical and Privacy ConsiderationsBiasBias Is Hard to EscapeMetadataPreventing Lost MetadataTrain/Val/Test Is the Cherry on TopSample CreationSimple Schema for a Strawberry Picking SystemGeometric RepresentationsBinary ClassificationLet’s Manually Create Our First SetUpgraded ClassificationWhere Is the Traffic Light?MaintenanceActionsNet LiftLevels of System Maturity of Training Data OperationsApplied Versus Research SetsTraining Data ManagementQualityCompleted TasksFreshnessMaintaining Set MetadataTask ManagementSummary
IntroductionAI TransformationSeeing Your Day-to-Day Work as AnnotationThe Creative Revolution of Data-centric AIYou Can Create New DataYou Can Change What Data You CollectYou Can Change the Meaning of the DataYou Can Create!Think Step Function Improvement for Major ProjectsBuild Your AI Data to Secure Your AI Present and FutureAppoint a Leader: The Director of AI DataNew Expectations People Have for the Future of AISometimes Proposals and Corrections, Sometimes ReplacementUpstream Producers and Downstream ConsumersSpectrum of Training Data Team EngagementDedicated Producers and Other TeamsOrganizing Producers from Other TeamsUse Case DiscoveryRubric for Good Use CasesEvaluating a Use Case Against the RubricConceptual Effects of Use CasesThe New “Crowd Sourcing”: Your Own ExpertsKey Levers on Training Data ROIWhat the Annotated Data RepresentsTrade-Offs of Controlling Your Own Training DataThe Need for HardwareCommon Project MistakesModern Training Data ToolsThink Learning Curve, Not PerfectionNew Training and Knowledge Are RequiredHow Companies Produce and Consume DataTrap to Avoid: Premature Optimization in Training DataNo Silver BulletsCulture of Training DataNew Engineering PrinciplesSummary
IntroductionGetting StartedMotivation: When to Use These Methods?Check What Part of the Schema a Method Is Designed to Work OnWhat Do People Actually Use?What Kind of Results Can I Expect?Common ConfusionsUser Interface OptimizationsRisksTrade-OffsNature of AutomationsSetup CostsHow to Benchmark WellHow to Scope the Automation Relative to the ProblemCorrection TimeSubject Matter ExpertsConsider How the Automations StackPre-LabelingStandard Pre-LabelingPre-Labeling a Portion of the Data OnlyInteractive Annotation AutomationCreating Your OwnTechnical Setup NotesWhat Is a Watcher? (Observer Pattern)How to Use a WatcherInteractive Capturing of a Region of InterestInteractive Drawing Box to Polygon Using GrabCutFull Image Model Prediction ExampleExample: Person Detection for Different AttributeQuality Assurance AutomationUsing the Model to Debug the HumansAutomated Checklist ExampleDomain-Specific Reasonableness ChecksData Discovery: What to LabelHuman ExplorationRaw Data ExplorationMetadata ExplorationAdding Pre-Labeling-Based MetadataAugmentationBetter Models Are Better than Better AugmentationTo Augment or Not to AugmentSimulation and Synthetic DataSimulations Still Need Human ReviewMedia SpecificWhat Methods Work with Which Media?ConsiderationsMedia-Specific ResearchDomain SpecificGeometry-Based LabelingHeuristics-Based LabelingSummary
IntroductionIndustryA Security Startup Adopts Training Data ToolsQuality Assurance at a Large-Scale Self-Driving ProjectBig-Tech ChallengesInsurance Tech Startup LessonsStoriesAn Academic Approach to Training DataKaggle TSA CompetitionSummary

Content preview from Training Data for Machine Learning

Chapter 2. Getting Up and Running

Introduction

There are many tools available that help us when we work with data: we have databases to smoothly store data and web servers to smoothly serve data. And now, training data tools to smoothly work with training data.

In addition to tools, there are established processes and expectations for how databases integrate with the rest of your application. But what about training data? How do you get up and running with training data? Throughout this chapter, I’ll cover key considerations, including installation, annotation setup, embedding, end user, workflow, and more.

It’s important to note why I referenced smoothly working with training data earlier. I say “smoothly” because I don’t have to use a database. I could write my data to a file and read from that. Why do I need a database, like Postgres, to build my system? Well, because Postgres brings a vast variety of features, such as guarantees that my data won’t easily get corrupted, that data is recoverable, and that data can be queried efficiently. Training data tools have evolved in a similar way.

In this chapter I will cover:

How to get up and running
The scope of training data tools
The benefits you get from using training data tools
Trade-offs
The history that got us to where we are today

Most of this is focused on things that will be relevant to you today. I also include some brief sections on history to demonstrate why these tools matter. Additionally, I will also answer ...