book

Machine Learning Model Serving Patterns and Best Practices

Name: Machine Learning Model Serving Patterns and Best Practices
Author: Md Johirul Islam
ISBN: 9781803249902

by Md Johirul Islam

December 2022

Intermediate to advanced

336 pages

8h 7m

English

Packt Publishing

Read now

Unlock full access

Machine Learning Model Serving Patterns and Best Practices
ContributorsAbout the authorAbout the reviewer
Preface
Who this book is forWhat this book coversTo get the most out of this bookDownload the example code filesConventions usedGet in touchShare Your ThoughtsDownload a free PDF copy of this book
Part 1:Introduction to Model Serving
Chapter 1: Introducing Model Serving
Technical requirementsWhat is serving?What are models?What is model serving?Understanding the importance of model servingUsing existing tools to serve modelsSummary
Chapter 2: Introducing Model Serving Patterns
Design patterns in software engineeringUnderstanding the value of model serving patternsML serving patternsServing philosophy patternsPatterns of serving approachesSummaryFurther reading
Part 2:Patterns and Best Practices of Model Serving
Chapter 3: Stateless Model Serving
Technical requirementsUnderstanding stateful and stateless functionsStateless functionsStateful functionsExtracting states from stateful functionsUsing stateful functionsStates in machine learning modelsUsing input data as statesMitigating the impact of states from the ML modelSummary
Chapter 4: Continuous Model Evaluation
Technical requirementsIntroducing continuous model evaluationWhat to monitor in model evaluationChallenges of continuous model evaluationThe necessity of continuous model evaluationMonitoring errorsDeciding on retrainingEnhancing serving resourcesUnderstanding business impactCommon metrics for training and monitoringContinuous model evaluation use casesEvaluating a model continuouslyCollecting the ground truthPlotting metrics on a dashboardSelecting the thresholdSetting a notification for performance dropsMonitoring model performance when predicting rare classesSummaryFurther reading
Chapter 5: Keyed Prediction
Technical requirementsIntroducing keyed predictionExploring keyed prediction use casesMulti-threaded programmingMultiple instances of the model running asynchronouslyWhy the keyed prediction model is neededExploring techniques for keyed predictionPassing keys with features from the clientsRemoving keys before the predictionTagging predictions with keysCreating keysSummaryFurther reading
Chapter 6: Batch Model Serving
Technical requirementsIntroducing batch model servingWhat is batch model serving?Different types of batch model servingManual triggersAutomatic periodic triggersUsing continuous model evaluation to retrainServing for offline inferenceServing for on-demand inferenceExample scenarios of batch model servingCase 1 – recommendationCase 2 – sentiment analysisTechniques in batch model servingSetting up a periodic batch updateStoring the predictions in a persistent storePulling predictions by the server applicationLimitations of batch servingSummaryFurther reading

Chapter 7: Online Learning Model Serving
Technical requirementsIntroducing online model servingServing requestsUse cases for online model servingCase 1 – recommending the nearest emergency center during a pandemicCase 2 – predicting the favorite soccer team in a tournamentCase 3 – predicting the path of a hurricane or stormCase 4 – predicting the estimated delivery time of delivery trucksChallenges in online model servingChallenges in using newly arrived data for trainingUnderperforming of the model after online trainingOverfitting and class imbalanceIncreasing of latencyHandling concurrent requestsImplementing online model servingSummaryFurther reading
Chapter 8: Two-Phase Model Serving
Technical requirementsIntroducing two-phase model servingExploring two-phase model serving techniquesQuantized phase one modelTraining and saving an MNIST modelFull integer quantization of the model and saving the converted modelComparing the size and accuracy of the modelsSeparately trained phase one model with reduced featuresSeparately trained different modelsUse cases of two-phase model servingCase 4 – route plannersSummaryFurther reading
Chapter 9: Pipeline Pattern Model Serving
Technical requirementsIntroducing the pipeline patternA DAGStages of the machine learning pipelineIntroducing Apache AirflowGetting started with Apache AirflowCreating and starting a pipeline using Apache AirflowDemonstrating a machine learning pipeline using AirflowAdvantages and disadvantages of the pipeline patternSummaryFurther reading
Chapter 10: Ensemble Model Serving Pattern
Technical requirementsIntroducing the ensemble patternUsing ensemble pattern techniquesModel updateAggregationModel selectionCombining responsesEnd-to-end dummy example of serving the modelSummary
Chapter 11: Business Logic Pattern
Technical requirementsIntroducing the business logic patternType of business logicTechnical approaches to business logic in model servingData validationFeature transformationPrediction post-processingSummary
Part 3:Introduction to Tools for Model Serving
Chapter 12: Exploring TensorFlow Serving
Technical requirementsIntroducing TensorFlow ServingServableLoaderSourceAspired versionsManagerUsing TensorFlow Serving to serve modelsTensorFlow Serving with DockerUsing advanced model configurationsSummaryFurther reading
Chapter 13: Using Ray Serve
Technical requirementsIntroducing Ray ServeDeploymentServeHandleIngress deploymentDeployment graphUsing Ray Serve to serve a modelUsing the ensemble pattern in Ray ServeUsing Ray Serve with the pipeline patternSummaryFurther reading
Chapter 14: Using BentoML
Technical requirementsIntroducing BentoMLPreparing modelsServices and APIsBentoUsing BentoML to serve a modelSummaryFurther reading
Part 4:Exploring Cloud Solutions
Chapter 15: Serving ML Models using a Fully Managed AWS Sagemaker Cloud Solution
Technical requirementsIntroducing Amazon SageMakerAmazon SageMaker featuresUsing Amazon SageMaker to serve a modelCreating a notebook in Amazon SageMakerServing the model using Amazon SageMakerSummary
Index
Why subscribe?
Other Books You May EnjoyPackt is searching for authors like youShare Your ThoughtsDownload a free PDF copy of this book

Content preview from Machine Learning Model Serving Patterns and Best Practices

8 Two-Phase Model Serving

In this chapter, we will discuss the two-phase prediction pattern. In the two-phase prediction pattern, we deploy two different models. The bigger and more complex model is deployed on the server. In most cases, the users of this model are edge devices where the network may fluctuate. So, in the case of bad network access, an edge device can use a lightweight model to get predictions for basic use cases. For broader and more accurate predictions, the devices can get the prediction by calling APIs to the model deployed to the server. We will discuss the serving of models in this scenario of edge devices that exist in unstable networking conditions.

We will cover the following topics in this chapter:

Introducing two-phase ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

PyTorch Recipes: A Problem-Solution Approach to Build, Train and Deploy Neural Network Models

Publisher Resources

ISBN: 9781803249902

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Machine Learning Model Serving Patterns and Best Practices

by Md Johirul Islam

8

Two-Phase Model Serving

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.