book

LLM Engineer's Handbook

Name: LLM Engineer's Handbook
ISBN: 9781836200079

by Paul Iusztin, Maxime Labonne

October 2024

Intermediate to advanced

522 pages

12h 55m

English

Packt Publishing

Read now

Unlock full access

Forewords
ContributorsJoin our book’s Discord space
Preface
Who this book is forWhat this book coversTo get the most out of this bookGet in touchMaking the Most Out of This Book – Get to Know Your Free Benefits
Understanding the LLM Twin Concept and Architecture
Understanding the LLM Twin conceptWhat is an LLM Twin?Why building an LLM Twin mattersWhy not use ChatGPT (or another similar chatbot)?Planning the MVP of the LLM Twin productWhat is an MVP?Defining the LLM Twin MVPBuilding ML systems with feature/training/inference pipelinesThe problem with building ML systemsThe issue with previous solutionsThe solution – ML pipelines for ML systemsThe feature pipelineThe training pipelineThe inference pipelineBenefits of the FTI architectureDesigning the system architecture of the LLM TwinListing the technical details of the LLM Twin architectureHow to design the LLM Twin architecture using the FTI pipeline designData collection pipelineFeature pipelineTraining pipelineInference pipelineFinal thoughts on the FTI design and the LLM Twin architectureSummaryReferences
Tooling and Installation
Python ecosystem and project installationPoetry: dependency and virtual environment managementPoe the Poet: task execution toolMLOps and LLMOps toolingHugging Face: model registryZenML: orchestrator, artifacts, and metadataOrchestratorArtifacts and metadataHow to run and configure a ZenML pipelineComet ML: experiment trackerOpik: prompt monitoringDatabases for storing unstructured and vector dataMongoDB: NoSQL databaseQdrant: vector databasePreparing for AWSSetting up an AWS account, an access key, and the CLISageMaker: training and inference computeWhy AWS SageMaker?SummaryReferencesJoin our book’s Discord space
Data Engineering
Designing the LLM Twin’s data collection pipelineImplementing the LLM Twin’s data collection pipelineZenML pipeline and stepsThe dispatcher: How do you instantiate the right crawler?The crawlersBase classesGitHubCrawler classCustomArticleCrawler classMediumCrawler classThe NoSQL data warehouse documentsThe ORM and ODM software patternsImplementing the ODM classData categories and user document classesGathering raw data into the data warehouseTroubleshootingSelenium issuesImport our backed-up dataSummaryReferences
RAG Feature Pipeline
Understanding RAGWhy use RAG?HallucinationsOld informationThe vanilla RAG frameworkIngestion pipelineRetrieval pipelineGeneration pipelineWhat are embeddings?Why embeddings are so powerfulHow are embeddings created?Applications of embeddingsMore on vector DBsHow does a vector DB work?Algorithms for creating the vector indexDB operationsAn overview of advanced RAGPre-retrievalRetrievalPost-retrievalExploring the LLM Twin’s RAG feature pipeline architectureThe problem we are solvingThe feature storeWhere does the raw data come from?Designing the architecture of the RAG feature pipelineBatch pipelinesBatch versus streaming pipelinesCore stepsChange data capture: syncing the data warehouse and feature storeWhy is the data stored in two snapshots?OrchestrationImplementing the LLM Twin’s RAG feature pipelineSettingsZenML pipeline and stepsQuerying the data warehouseCleaning the documentsChunk and embed the cleaned documentsLoading the documents to the vector DBPydantic domain entitiesOVMThe dispatcher layerThe handlersThe cleaning handlersThe chunking handlersThe embedding handlersSummaryReferencesJoin our book’s Discord space
Supervised Fine-Tuning
Creating an instruction datasetGeneral frameworkData quantityData curationRule-based filteringData deduplicationData decontaminationData quality evaluationData explorationData generationData augmentationCreating our own instruction datasetExploring SFT and its techniquesWhen to fine-tuneInstruction dataset formatsChat templatesParameter-efficient fine-tuning techniquesFull fine-tuningLoRAQLoRATraining parametersLearning rate and schedulerBatch sizeMaximum length and packingNumber of epochsOptimizersWeight decayGradient checkpointingFine-tuning in practiceSummaryReferences
Fine-Tuning with Preference Alignment
Understanding preference datasetsPreference dataData quantityData generation and evaluationGenerating preferencesTips for data generationEvaluating preferencesCreating our own preference datasetPreference alignmentReinforcement Learning from Human FeedbackDirect Preference OptimizationImplementing DPOSummaryReferencesJoin our book’s Discord space
Evaluating LLMs
Model evaluationComparing ML and LLM evaluationGeneral-purpose LLM evaluationsDomain-specific LLM evaluationsTask-specific LLM evaluationsRAG evaluationRagasARESEvaluating TwinLlama-3.1-8BGenerating answersEvaluating answersAnalyzing resultsSummaryReferences
Inference Optimization
Model optimization strategiesKV cacheContinuous batchingSpeculative decodingOptimized attention mechanismsModel parallelismData parallelismPipeline parallelismTensor parallelismCombining approachesModel quantizationIntroduction to quantizationQuantization with GGUF and llama.cppQuantization with GPTQ and EXL2Other quantization techniquesSummaryReferencesJoin our book’s Discord space

RAG Inference Pipeline
Understanding the LLM Twin’s RAG inference pipelineExploring the LLM Twin’s advanced RAG techniquesAdvanced RAG pre-retrieval optimizations: query expansion and self-queryingQuery expansionSelf-queryingAdvanced RAG retrieval optimization: filtered vector searchAdvanced RAG post-retrieval optimization: rerankingImplementing the LLM Twin’s RAG inference pipelineImplementing the retrieval moduleBringing everything together into the RAG inference pipelineSummaryReferences
Inference Pipeline Deployment
Criteria for choosing deployment typesThroughput and latencyDataUnderstanding inference deployment typesOnline real-time inferenceAsynchronous inferenceOffline batch transformMonolithic versus microservices architecture in model servingMonolithic architectureMicroservices architectureChoosing between monolithic and microservices architecturesExploring the LLM Twin’s inference pipeline deployment strategyThe training versus the inference pipelineDeploying the LLM Twin serviceImplementing the LLM microservice using AWS SageMakerWhat are Hugging Face’s DLCs?Configuring SageMaker rolesDeploying the LLM Twin model to AWS SageMakerCalling the AWS SageMaker Inference endpointBuilding the business microservice using FastAPIAutoscaling capabilities to handle spikes in usageRegistering a scalable targetCreating a scalable policyMinimum and maximum scaling limitsCooldown periodSummaryReferencesJoin our book’s Discord space
MLOps and LLMOps
The path to LLMOps: Understanding its roots in DevOps and MLOpsDevOpsThe DevOps lifecycleThe core DevOps conceptsMLOpsMLOps core componentsMLOps principlesML vs. MLOps engineeringLLMOpsHuman feedbackGuardrailsPrompt monitoringDeploying the LLM Twin’s pipelines to the cloudUnderstanding the infrastructureSetting up MongoDBSetting up QdrantSetting up the ZenML cloudContainerize the code using DockerRun the pipelines on AWSTroubleshooting the ResourceLimitExceeded error after running a ZenML pipeline on SageMakerAdding LLMOps to the LLM TwinLLM Twin’s CI/CD pipeline flowMore on formatting errorsMore on linting errorsQuick overview of GitHub ActionsThe CI pipelineGitHub Actions CI YAML fileThe CD pipelineTest out the CI/CD pipelineThe CT pipelineInitial triggersTrigger downstream pipelinesPrompt monitoringAlertingSummaryReferences
MLOps Principles
1. Automation or operationalization2. Versioning3. Experiment tracking4. TestingTest typesWhat do we test?Test examples5. MonitoringLogsMetricsSystem metricsModel metricsDriftsMonitoring vs. observabilityAlerts6. Reproducibility
Other Books You May Enjoy
Index

Content preview from LLM Engineer's Handbook

6 Fine-Tuning with Preference Alignment

Supervised Fine-Tuning (SFT) has been crucial in adapting LLMs to perform specific tasks. However, SFT struggles to capture the nuances of human preferences and the long tail of potential interactions that a model might encounter. This limitation has led to the development of more advanced techniques for aligning AI systems with human preferences, grouped under the umbrella term preference alignment.

Preference alignment addresses the shortcomings of SFT by incorporating direct human or AI feedback into the training process. This method allows a more nuanced understanding of human preferences, especially in complex scenarios where simple supervised learning falls short. While numerous techniques exist ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Publisher Resources

ISBN: 9781836200079

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

LLM Engineer's Handbook

by Paul Iusztin, Maxime Labonne

6

Fine-Tuning with Preference Alignment

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.