book

Pretrain Vision and Large Language Models in Python

Name: Pretrain Vision and Large Language Models in Python
Author: Emily Webber
ISBN: 9781804618257

by Emily Webber

May 2023

Intermediate to advanced

258 pages

7h 31m

English

Packt Publishing

Read now

Unlock full access

Pretrain Vision and Large Language Models in Python
ForewordContributorsAbout the authorAcknowledgmentAbout the reviewer
Preface
Who is this book for?What this book coversTo get the most out of this bookDownload the example code filesConventions usedGet in touchShare Your ThoughtsDownload a free PDF copy of this book
Part 1: Before Pretraining
Chapter 1: An Introduction to Pretraining Foundation Models
The art of pretraining and fine-tuningThe Transformer model architecture and self-attentionState-of-the-art vision and language modelsTop vision models as of April 2023Contrastive pretraining and natural language supervisionTop language models as of April 2023Language technique spotlight – causal modeling and the scaling lawsEncoders and decodersSummaryReferences
Chapter 2: Dataset Preparation: Part One
Finding a dataset and use case for foundation modelingTop pretraining use cases by industryDelta – how different is your dataset?Use the scaling laws to size your datasetsFundamentals – scaling laws of neural language modelsBias detection and mitigationEnhancing your dataset – multilingual, multimodal, and augmentationsSummaryReferences
Chapter 3: Model Preparation
Finding your best base modelStarting with the smallest base model you canTrade-off – simplicity versus complexityFinding your pretraining loss functionPretraining loss functions in vision – ViT and CoCaPretraining loss functions in language – Alexa Teacher ModelChanging your pretraining loss functionSolving for your model sizePractical approaches to solving for your model sizeNot all scaling laws are created equalPlanning future experimentsSummaryReferences
Part 2: Configure Your Environment
Chapter 4: Containers and Accelerators on the Cloud
What are accelerators and why do they matter?Getting ready to use your acceleratorsHow to use accelerators on AWS – Amazon SageMakerOptimizing accelerator performanceHyperparametersInfrastructure optimizations for accelerators on AWSTroubleshooting accelerator performanceSummaryReferences
Chapter 5: Distribution Fundamentals
Understanding key concepts – data and model parallelismWhat data parallel is all aboutWhat model parallel is all aboutCombining model and data parallelDistributed training on Amazon SageMakerDistributed training softwareSM DDPSMP libraryAdvanced techniques to reduce GPU memoryTensor parallelismOptimizer state shardingActivation checkpointingSharded data parallelismBringing it all home with examples from models todayStable Diffusion – data parallelism at scaleGPT-3 – model and data parallelism at scaleSummaryReferences
Chapter 6: Dataset Preparation: Part Two, the Data Loader
Introducing the data loader in PythonBuilding and testing your own data loader – a case study from Stable DiffusionCreating embeddings – tokenizers and other key steps for smart featuresOptimizing your data pipeline on Amazon SageMakerTransforming deep learning datasets at scale on AWSSummaryReferences

Part 3: Train Your Model
Chapter 7: Finding the Right Hyperparameters
Hyperparameters – batch size, learning rate, and moreKey hyperparameters in vision and languageTuning strategiesHyperparameter tuning for foundation modelsScaling up as a function of world size with SageMakerTuning on a sample of your data and updating based on world sizeSummaryReferences
Chapter 8: Large-Scale Training on SageMaker
Optimizing your script for SageMaker trainingImporting packagesArgument parsingTop usability features for SageMaker trainingWarm pools for rapid experimentationSSM and SSH into training instancesTrack jobs and experiments to replicate resultsSummaryReferences
Chapter 9: Advanced Training Concepts
Evaluating and improving throughputCalculating model TFLOPSUsing Flash Attention to speed up your training runsSpeeding up your jobs with compilationIntegrating compilation into your PyTorch scriptsAmazon SageMaker Training Compiler and NeoBest practices for compilationRunning compiled models on Amazon’s Trainium and Inferentia custom hardwareSolving for an optimal training timeSummaryReferences
Part 4: Evaluate Your Model
Chapter 10: Fine-Tuning and Evaluating
Fine-tuning for language, text, and everything in betweenFine-tuning a language-only modelFine-tuning vision-only modelsFine-tuning vision-language modelsEvaluating foundation modelsModel evaluation metrics for visionModel evaluation metrics in languageModel evaluation metrics in joint vision-language tasksIncorporating the human perspective with labeling through SageMaker Ground TruthReinforcement learning from human feedbackSummaryReferences
Chapter 11: Detecting, Mitigating, and Monitoring Bias
Detecting bias in ML modelsDetecting bias in large vision and language modelsMitigating bias in vision and language modelsBias mitigation in language – counterfactual data augmentation and fair loss functionsBias mitigation in vision – reducing correlation dependencies and solving sampling issuesMonitoring bias in ML modelsDetecting, mitigating, and monitoring bias with SageMaker ClarifySummaryReferences
Chapter 12: How to Deploy Your Model
What is model deployment?What is the best way to host my model?Model deployment options on AWS with SageMakerWhy should I shrink my model, and how?Model compilationKnowledge distillationQuantizationHosting distributed models on SageMakerModel servers and end-to-end hosting optimizationsSummaryReferences
Part 5: Deploy Your Model
Chapter 13: Prompt Engineering
Prompt engineering – the art of getting more with lessFrom few- to zero-shot learningText-to-image prompt engineering tipsImage-to-image prompt engineering tipsUpscalingMaskingPrompting for object-to-image with DreamBoothPrompting large language modelsInstruction fine-tuningChain-of-thought promptingSummarizationDefending against prompt injections and jailbreakingAdvanced techniques – prefix and prompt tuningPrefix tuningPrompt tuningSummaryReferences
Chapter 14: MLOps for Vision and Language
What is MLOps?Common MLOps pipelinesContinuous integration and continuous deploymentModel monitoring and human-in-the-loopMLOps for foundation modelsMLOps for visionAWS offerings for MLOpsA quick introduction to SageMaker PipelinesSummaryReferences
Chapter 15: Future Trends in Pretraining Foundation Models
Techniques for building applications for LLMsBuilding interactive dialogue apps with open-source stacksUsing RAG to ensure high accuracy in LLM applicationsIs generation the new classification?Human-centered design for building applications with LLMsOther generative modalitiesAWS offerings in foundation modelsThe future of foundation modelsThe future of pretrainingSummaryReferences
Index
Why subscribe?
Other Books You May EnjoyPackt is searching for authors like youShare Your ThoughtsDownload a free PDF copy of this book

Content preview from Pretrain Vision and Large Language Models in Python

Part 4: Evaluate Your Model

In part 4, you’ll learn how to evaluate your model. You’ll use the scaling laws to identify the shortest possible training time, fine-tune your model to compare with public benchmarks, and identify and mitigate bias.

This section has the following chapters:

Chapter 10, Fine-Tuning and Evaluating
Chapter 11, Detecting, Mitigating, and Monitoring Bias
Chapter 12, How to Deploy Your Model

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Computer Vision Projects with PyTorch: Design and Develop Production-Grade Models

Publisher Resources

ISBN: 9781804618257

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Pretrain Vision and Large Language Models in Python

by Emily Webber

Part 4: Evaluate Your Model

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.