book

Large Language Models: The Hard Parts

by Thársis T. P. Souza, Jonathan K. Regenstein Jr.

May 2026

Intermediate to advanced

340 pages

8h 45m

English

O'Reilly Media, Inc.

Read now

Unlock full access

Foreword
Preface
Who Should Read This BookNavigating This Book by ChapterConventions Used in This BookUsing Code ExamplesO’Reilly Online LearningHow to Contact UsAcknowledgments
1. First Principles: What to Consider Before We Start Building with LLMs
Why Open Source?Strategic ConsiderationsEnterprise RequirementsStakeholder RequirementsCompliance and Security RequirementsPerformance RequirementsOperational RequirementsIntegration RequirementsData Management RequirementsOrganizational AI FrameworksCentralized FrameworkDecentralized FrameworkFederated FrameworkThe Importance of DataModel TypesBase ModelsInstruction Fine-Tuned ModelsDomain Adapted ModelsModel FeaturesContext LengthOutput ControlCachingOutput Token LengthCost and SpeedLicensingCustomizationSmall Language ModelsConclusion
2. The Evals Gap
Nondeterministic Nature of LLMsSource of NondeterminismTemperature10-K LLMBA Temperature TestsThe Evals ChallengeDesigning an LLMBA Evaluation FrameworkConceptual Design: Single LLMBAConceptual Design: Multiple LLMBAsMethodologies for EvaluationQuantitative MetricsCoding BLEU and ROUGECoding an LLM-as-a-JudgeLimitations of LLM-as-a-JudgeEvaluating EvaluatorsA Brief Tour of LLM BenchmarksARC-AGI and the PrizeConclusion
3. Evaluation Tools for LLM-Based Applications
LangSmithBLEU with LangSmithScaling LLM-as-a-Judge with LangSmithPromptfooEvaluating Models with PromptfooEvaluating PromptsLightEvalLightEval SetupEconometric Evaluation with MMLUComparing Multiple Models on EconometricsFrameworks ComparisonConclusion
4. From Data to Context
Preprocessing DocumentsPyPDF2MarkItDownDoclingRAGRAG ProcessChunking DocumentsChunking StrategiesChunking Case StudyVector EmbeddingsRerankingReport GenerationRAG Challenges and LimitationsLong-Context Models: RAG Killer?LCM Case Study: Quiz Generation with CitationsImplementationExample UsageDiscussion and LimitationsConclusion
5. Structured Data Output
Why Structured Output MattersImproving Developer Efficiency and WorkflowMeeting UI and Product RequirementsEnhancing User Trust and ExperienceWhy Is This Hard? The Transformer ArchitectureTraining-Time Constraint TechniquesInference-Time Constraint TechniquesPrompt EngineeringJSON Mode (Fine-Tuned)Combining JSON Mode with PydanticLogit PostprocessingOutlinesOllamaOllama with PydanticLangChainComparing ToolsConclusion
6. LLM Safety Considerations
General AI Safety RisksLLM-Specific Safety RisksJailbreakingPrompt Injection (Prompt Crafting)Stealth EditingAddressing LLMBA Safety RisksMisinformation PreventionUnqualified Advice PreventionBias DetectionPrivacy ProtectionGuidance from AI Labs and Policy CentersOpenAIAnthropicGoogleMLCommons AI Safety BenchmarkCentre for the Governance of AI RubricTwo Specific Approaches to LLM SafetyRed TeamingConstitutional AIConclusion
7. Evaluating LLMs for Safety
Safety Evaluation with Benchmark DatasetsSALAD-BenchTruthfulQAHarmBenchRuntime Guardrails and ModerationNeMo GuardrailsTruLens GuardrailsLlama GuardMistral’s Moderation APIOpenAI’s Moderation APICustom Moderation with LLM-as-a-JudgeCase Study: Implementing a Safety Filter for K–12 StudentsEvals DatasetSafety FiltersBenchmarkingConclusion
8. LLM Alignment: A Case Study
Why Is Alignment Necessary?RLHF: The Breakthrough That Made LLMs TrustworthyProximal Policy OptimizationDPOCase Study: Aligning an LLM to a PolicyAlignment ToolsAcme’s Educational PolicyPreference Dataset—Synthetic Dataset GenerationFine-TuningAlignment Evaluation: LLM-as-a-JudgeDPO Dataset CompositionOur Choice of Base ModelThe Evaluation MethodologyThe Fine-Tuning Process and Parameter ChoiceDesigning a Safety and Alignment PlanPhase 1: Policy DefinitionPhase 2: User Research and Risk IdentificationPhase 3: Evaluation FrameworkPhase 4: Safety Architecture DesignPhase 5: Implementation and Tools SelectionPhase 6: Incident ResponseCommon PitfallsConclusion

9. Epilogue: LLMBAs in the Era of Falling Costs
Appendix. Tools for Local LLM Deployment
Ollamallama.cppGPT-Generated Unified Formatllama.cpp Examplellama-serverllamafileQuantizationPerplexity and KL DivergencePrompts DatasetQuantization ExampleResultsLocal Tools, Real Power
Index
About the Authors

Overview

Large language models (LLMs) have transformed natural language processing, but deploying them in applications introduces numerous technical challenges. Large Language Models: The Hard Parts offers a clear, practical examination of the limitations developers and AI engineers face when building LLM-based applications. With a focus on implementation pitfalls (not just capabilities), this book provides actionable strategies supported by reproducible Python code and open source tools.

Readers will learn how to navigate key obstacles in application evaluation, input management, testing, and safety. Designed for builders and technical product leads, this guide emphasizes practical solutions to real-world problems and promotes a grounded understanding of LLM constraints and trade-offs.

Design testing and evaluation strategies for nondeterministic systems
Manage context, RAG, and long-context retrieval
Address output inconsistency and structural unreliability
Implement safety and content moderation frameworks
Explore alignment challenges and mitigation techniques
Leverage open source models locally

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Publisher Resources

ISBN: 9798341622517Errata Page

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills