book

Distributed Machine Learning Patterns

Name: Distributed Machine Learning Patterns
Author: Yuan Tang
ISBN: 9781617299025

by Yuan Tang

January 2024

Intermediate to advanced

248 pages

6h 29m

English

Manning Publications

Audiobook available

Read now

Unlock full access

Distributed Machine Learning Patterns
Copyright
contents
front matter
prefaceacknowledgmentsabout this bookWho should read this book?How this book is organized: A roadmapAbout the codeliveBook discussion forumabout the authorabout the cover illustration
Part 1 Basic concepts and background
1 Introduction to distributed machine learning systems
1.1 Large-scale machine learning1.1.1 The growing scale1.1.2 What can we do?1.2 Distributed systems1.2.1 What is a distributed system?1.2.2 The complexity and patterns1.3 Distributed machine learning systems1.3.1 What is a distributed machine learning system?1.3.2 Are there similar patterns?1.3.3 When should we use a distributed machine learning system?1.3.4 When should we not use a distributed machine learning system?1.4 What we will learn in this bookSummary
Part 2 Patterns of distributed machine learning systems
2 Data ingestion patterns
2.1 What is data ingestion?2.2 The Fashion-MNIST dataset2.3 Batching pattern2.3.1 The problem: Performing expensive operations for Fashion MNIST dataset with limited memory2.3.2 The solution2.3.3 Discussion2.3.4 Exercises2.4 Sharding pattern: Splitting extremely large datasets among multiple machines2.4.1 The problem2.4.2 The solution2.4.3 Discussion2.4.4 Exercises2.5 Caching pattern2.5.1 The problem: Re-accessing previously used data for efficient multi-epoch model training2.5.2 The solution2.5.3 Discussion2.5.4 Exercises2.6 Answers to exercisesSection 2.3.4Section 2.4.4Section 2.5.4Summary
3 Distributed training patterns
3.1 What is distributed training?3.2 Parameter server pattern: Tagging entities in 8 million YouTube videos3.2.1 The problem3.2.2 The solution3.2.3 Discussion3.2.4 Exercises3.3 Collective communication pattern3.3.1 The problem: Improving performance when parameter servers become a bottleneck3.3.2 The solution3.3.3 Discussion3.3.4 Exercises3.4 Elasticity and fault-tolerance pattern3.4.1 The problem: Handling unexpected failures when training with limited computational resources3.4.2 The solution3.4.3 Discussion3.4.4 Exercises3.5 Answers to exercisesSection 3.2.4Section 3.3.4Section 3.4.4Summary
4 Model serving patterns
4.1 What is model serving?4.2 Replicated services pattern: Handling the growing number of serving requests4.2.1 The problem4.2.2 The solution4.2.3 Discussion4.2.4 Exercises4.3 Sharded services pattern4.3.1 The problem: Processing large model serving requests with high-resolution videos4.3.2 The solution4.3.3 Discussion4.3.4 Exercises4.4 The event-driven processing pattern4.4.1 The problem: Responding to model serving requests based on events4.4.2 The solution4.4.3 Discussion4.4.4 Exercises4.5 Answers to exercisesSection 4.2Section 4.3Section 4.4Summary

5 Workflow patterns
5.1 What is workflow?5.2 Fan-in and fan-out patterns: Composing complex machine learning workflows5.2.1 The problem5.2.2 The solution5.2.3 Discussion5.2.4 Exercises5.3 Synchronous and asynchronous patterns: Accelerating workflows with concurrency5.3.1 The problem5.3.2 The solution5.3.3 Discussion5.3.4 Exercises5.4 Step memoization pattern: Skipping redundant workloads via memoized steps5.4.1 The problem5.4.2 The solution5.4.3 Discussion5.4.4 Exercises5.5 Answers to exercisesSection 5.2Section 5.3Section 5.4Summary
6 Operation patterns
6.1 What are operations in machine learning systems?6.2 Scheduling patterns: Assigning resources effectively in a shared cluster6.2.1 The problem6.2.2 The solution6.2.3 Discussion6.2.4 Exercises6.3 Metadata pattern: Handle failures appropriately to minimize the negative effect on users6.3.1 The problem6.3.2 The solution6.3.3 Discussion6.3.4 Exercises6.4 Answers to exercisesSection 6.2Section 6.3Summary
Part 3 Building a distributed machine learning workflow
7 Project overview and system architecture
7.1 Project overview7.1.1 Project background7.1.2 System components7.2 Data ingestion7.2.1 The problem7.2.2 The solution7.2.3 Exercises7.3 Model training7.3.1 The problem7.3.2 The solution7.3.3 Exercises7.4 Model serving7.4.1 The problem7.4.2 The solution7.4.3 Exercises7.5 End-to-end workflow7.5.1 The problems7.5.2 The solutions7.5.3 Exercises7.6 Answers to exercisesSection 7.2Section 7.3Section 7.4Section 7.5Summary
8 Overview of relevant technologies
8.1 TensorFlow: The machine learning framework8.1.1 The basics8.1.2 Exercises8.2 Kubernetes: The distributed container orchestration system8.2.1 The basics8.2.2 Exercises8.3 Kubeflow: Machine learning workloads on Kubernetes8.3.1 The basics8.3.2 Exercises8.4 Argo Workflows: Container-native workflow engine8.4.1 The basics8.4.2 Exercises8.5 Answers to exercisesSection 8.1Section 8.2Section 8.3Section 8.4Summary
9 A complete implementation
9.1 Data ingestion9.1.1 Single-node data pipeline9.1.2 Distributed data pipeline9.2 Model training9.2.1 Model definition and single-node training9.2.2 Distributed model training9.2.3 Model selection9.3 Model serving9.3.1 Single-server model inference9.3.2 Replicated model servers9.4 The end-to-end workflow9.4.1 Sequential steps9.4.2 Step memoizationSummary
index

Content preview from Distributed Machine Learning Patterns

6 Operation patterns

This chapter covers

Recognizing areas of improvement in machine learning systems, such as job scheduling and metadata
Preventing resource starvation and avoiding deadlocks using scheduling techniques, such as fair-share scheduling, priority scheduling, and gang scheduling
Handling failures more effectively to reduce any negative effect on users via the metadata pattern

In chapter 5, we focused on machine learning workflows and the challenges of building them in practice. Workflow is an essential component in machine learning systems as it connects all components in the system. A machine learning workflow can be as easy as chaining data ingestion, model training, and model serving. It can also be very complex when handling ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Publisher Resources

ISBN: 9781617299025Publisher Support Publisher Website

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Distributed Machine Learning Patterns

by Yuan Tang

6 Operation patterns

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.