book

LLMs in Enterprise

Name: LLMs in Enterprise
ISBN: 9781836203070

by Ahmed Menshawy, Mahmoud Fahmy

September 2025

Intermediate to advanced

564 pages

14h 14m

English

Packt Publishing

Read now

Unlock full access

Contributors
Subscribe for a free eBook
Preface
Who this book is forWhat this book coversTo get the most out of this bookDisclaimer on imagesDisclaimer on AI usageGet in touchShare Your ThoughtsJoin our Discord and Reddit spaceYour Book Comes with Exclusive Perks – Here’s How to Unlock ThemUnlock Your Book’s Exclusive BenefitsHow to unlock these benefits in three easy stepsStep 1Step 2Step 3Need help?
Part 1: Background and Foundational Concepts
Introduction to Large Language Models
Historical context and evolution of language modelsEarly developmentsEvolution over timeComputational advances and increasing data availabilityAdvancements in computational powerAvailability of large datasetsLLMs and transforming user interfaces into natural conversationsEvolution of LLMs architecturesEarly foundations – word embeddingsBreakthrough with transformersThe rise of pre-trained modelsMultimodality and beyondMixture of experts – revolutionizing language model architecturesCore concepts of MoEThe building blocks of MoEsHistorical context and developmentPractical applications and future directionsGPT assistant training recipeBuilding the base modelData collection and assemblyData preprocessing – tokenizationModel configurationComputational requirementsTraining processBuilding base model recapOutcome – the pre-trained base modelSupervised fine-tuning stageData collection – high-quality, task-specificTraining process – refinement and specializationModel adjustments – fine-tuning hyperparametersOutput – the SFT modelReward modeling stageData collection for comparisonHuman judgment and scoringIntegration with the neural networkOutcome – a trained reward modelReinforcement learning stageIntegration of the reward modelTraining processOptimization techniquesOutcome – a reinforced learning modelDecoding the realities and myths of LLMsAda Lovelace’s insightsFailures in simple tasksLimitations compared to human intelligenceObjective-driven AIHuman-technology augmentationSummaryReferences
LLMs in Enterprise: Applications, Challenges, and Design Patterns
From unstructured data to LLMsEnterprise application of LLMsFinancial servicesHealthcareRetail and e-commerceEducation and trainingChallenges in scaling and deploying LLMsTechnical challengesMemory requirementsMemory managementModel pruning and quantizationParallelism strategies in LLM deploymentModel reliabilityMaintenance and continuous updatesNeed for robust infrastructureData privacy and security issuesData privacy concernsSecurity risksRegulatory compliance challengesCost implicationsInitial investment in technologyOngoing costsROI considerationsEthical and societal implicationsBias in AI modelsRegulatory compliance challengesImpact of automation on employmentLLM design patternsDynamic batching for inferenceModel compression and utility vs complexity trade-offEvaluation and monitoringSummaryReferencesSubscribe for a free eBook
Advanced Fine-Tuning Techniques and Strategies for Large Language Models
Technical requirementsTraining foundational modelsPretraining phaseDataset collectionBase model: pretraining and fine-tuning for text generationSupervised fine-tuningDataset collectionTraining phaseReward modeling and reinforcement learningReward modelingReinforcement learningApproaches to integration in business processesUtilizing pre-built APIsIn-context learning using OpenAI APIBuilding the few-shot promptFine-tuning pre-trained modelsFine-tuning the Llama modelFine-tuning strategies for enhanced performanceImplementing RAGExploring RAGConceptual introductionImplementation and use casesEvaluating impactSummaryReferences
Retrieval-Augmented Generation Pattern
Technical requirementsIntroduction to retrieval-augmented generationUnderstanding the need for external information sourcesOvercoming context limitations with retrievalReducing hallucinations and ensuring attributable informationFoundations of retrieval mechanismsHistorical context of information retrievalKey concepts in IRCommon retrieval algorithms and data structuresInverted indexesApproximate nearest neighbor searchHybrid approaches (term-based and embedding-based)Evaluating retrieval qualityPrecision, recall, and relevance metricsEnterprise-specific evaluation considerationsEmbeddings for enhanced retrievalConceptual overview of text embeddingsSemantic vs. lexical similarityEmbedding models and their propertiesDimensionality and vector-space representationsDomain adaptation of embeddingsIndexing, searching, and storing embeddings in vector databasesVector quantization and HNSW graphsFine-tuning embeddings for domain relevanceConstructing a RAG pipelineSystem architecture overviewRetrieval component integrationResponse generation from a retrieved contextChunking strategies for long documentsCombining multiple retrieval methodsEnsuring attribution, reducing hallucinations, and facilitating revisionsSource tracking and document provenanceThresholding mechanisms for irrelevant or uncertain resultsAuditing and updating the retrieval indexHandling stale or outdated informationRevision workflows for enterprise environmentsPolicies and governance for trusted RAG outputsSummaryReferenceSubscribe for a free eBook
Customizing Contextual LLMs
Technical requirementsThe importance of contextual customizationUtilizing external memory for context managementBenefitsRAGs in enterpriseEnterprise examplesLegal industryHealthcare industryFinance industryCustomer service industryEducation industryReal estate industryEnterprise challenges with RAGsIntegration complexityLatency issuesCost considerationsFactual incorrectness and hallucinationTechnical and optimization challengesUnderstanding the “retrieval” aspect of RAGHybrid searchImplementing hybrid searchThe complexity of prompt engineeringAddressing the prompt engineering challengeUse case – using RAG to enhance information retrievalSummaryReferences
Part 2: Advanced Design Patterns and Techniques
The Art of Prompt Engineering for Enterprise LLMs
Technical requirementsFoundations of prompt engineeringThe power of prompt engineeringUnderstanding the prompt-output relationshipThe prompt as a control mechanismPrompts as interfaces to model capabilitiesThe economics of prompt engineeringThe consequences of poor prompt engineeringCase study: healthcare information managementThe iterative nature of prompt engineeringStructured approach to prompt developmentMeasurement and optimizationEnterprise applications of strategic prompt engineeringCustomer service transformationContent generation at scaleData analysis and insight generationUnderstanding the science behind prompt engineeringFoundational conceptsClarity and specificityContextual cuesKey insights for effective prompt engineeringPrompt specificity and output qualityInformation processing in LLMsLeveraging structure and keywords for prompt effectivenessHierarchical organizationDelimiters and separation markersNumbered lists and bullet pointsKeyword optimizationDomain-specific terminologyAction-oriented keywordsPhrasing and syntaxReal-world application: enterprise policy analysisIncorporating context and continuityContinuous improvementZero-shot and few-shot learning promptsExample code for zero-shot and few-shot learning prompts with Hugging FaceManaging and mitigating hallucinations in LLMsUnderstanding hallucinations in LLMsApproaches to reduce hallucinationsIn-depth analysis of hallucination mitigation: a few hypothetical scenariosTools and frameworks for managing hallucinationsSummaryReferencesSubscribe for a free eBook

Enterprise Challenges in Evaluating LLM Applications
Technical requirementsThe role of evaluation patternsOutput accuracy and relevanceResponse variability and robustnessQualitative measures: coherence, fluency, and user experienceEthical considerations and bias detectionLongitudinal and adaptive evaluationManaging common errors in evaluationEnterprise-specific challenges in evaluating LLM applicationsScalability of evaluationEmergent abilities and benchmark creationVariability and model updatesInteraction with real-world tasksReal-world examples of enterprise LLM applicationsRecommendations for approaching LLM evaluationBuilding novel benchmarks and workflowsExample code for a custom benchmarking workflowFocus on interactive and multi-agent evaluationCentering responsible AI in evaluationBridging the gap between evaluation and improvementExample code for bridging the gap between evaluation and improvementReal-world examples of companies following LLM application best practicesBridging evaluation and metrics: a data-driven approachUnderstanding evaluation metrics for LLMsKey metrics and techniques for effective evaluationExample – dialogue systems and BERTScoreHuman evaluation metricsBridging automated and human evaluation techniquesMetrics for evaluating LLM performanceKey performance indicators (KPIs)Quantitative and qualitative measuresContinuous improvement processesSummaryReferences
The Data Blueprint: Crafting Effective Strategies for LLM Development
Technical requirementsImportance of data in LLM developmentImpact on model performanceRole in model generalizationAn overview of data strategy componentsData acquisitionData preprocessingData augmentationData governanceChallenges in data management for LLMsScalability issuesData quality variabilityEthical and legal considerationsCase studies on effective data strategiesExample code – data processing for modern LLMsDeepSeek’s data strategyData quality and preprocessingDefining data quality for LLMsRelevance and diversityAccuracy and consistencyData collection methodsSynthetic data generationBenefits of synthetic dataTools for synthetic data creationData cleaning and filteringDuplicate detection algorithmsNoise reduction techniquesHandling missing dataImputation methodsExclusion criteriaData annotation and labelingHuman annotation strategiesCrowdsourcing platformsExpert labeling teamsAutomated annotation toolsMachine learning-based labelingRule-based systemsData partitioningData partitioning and validation techniquesData sharding for distributed trainingData standardization and normalizationTokenization techniquesHandling multilingual dataAddressing bias and fairnessIdentifying bias in dataMitigation strategiesThe role of data augmentation in LLMsEnhancing model robustnessOvercoming data scarcityTechniques for data augmentationParaphrasing and translationNoise injectionBack-translation methodsLeveraging external knowledge basesIntegrating ontologiesUsing knowledge graphsData enrichment processesEntity recognition and linkingSentiment and semantic analysisEvaluating augmented data qualityAutomated quality metricsHITL evaluationSummaryReferencesSubscribe for a free eBook
Managing Model Deployments in Production
Technical requirementsManaging latency and throughput for business needsRequirements for low latencyInfrastructure and software optimizationsHigh-performance hardwareEfficient model designEdge computingCaching mechanismsThe expected outputThroughput optimization techniquesBalancing speed and accuracyMeeting stringent business and regulatory requirementsCompliance and governanceFrameworks and practices for regulatory complianceSecurity protocolsProtecting data and maintaining trustAuditing and reportingCase study: AI-powered forex rate analysis and query systemProblem statementSolutionCode implementation and explanationSummaryReferences
Accelerated and Optimized Inferencing Patterns
Technical requirementsIntroduction to advanced inferencing enginesThe need for optimized LLM inference Key challenges in LLM deployment Latency versus throughput trade-offs Memory and computational constraintsOverview of inference optimization techniques QuantizationKV cache optimization Dynamic batchingDeployment engines – comparative analysisvLLM – architecture and key featuresPagedAttention reimagining KV cache managementContinuous batching – eliminating static scheduling constraintsUse cases – scalable multi-GPU deploymentsBenchmarks – latency and throughput TensorRT-LLM – NVIDIA’s inference optimizerModel compilation and quantization Integration with Triton Inference ServerPerformance benchmarks (versus native PyTorch) NVIDIA Triton Inference ServerMulti-framework support (PyTorch, TensorFlow, and ONNX) Dynamic batching and model pipelinesCase study – deploying Llama-2 on Triton Machine Learning CompilerJust-in-time compilation for LLMs Cross-platform deployment (edge, cloud, or mobile) Benchmarks: Latency versus CTranslate2/vLLM Performance comparisons and trade-offsBenchmarking methodologyHardware considerations (A100, A6000, and T4)Metrics – time to first token, tokens/secondTool comparison – evaluating inference engine capabilitiesLatency optimization – MLC versus CTranslate2 versus vLLM Throughput optimization – Triton versus vLLM Memory efficiency – TensorRT versus GPTQ Real-world use cases – practical applications of LLM serving solutionsLow-latency chatbots (vLLM and continuous batching) Batch processing with Triton (healthcare NLP) Edge deployment with MLC (on-device LLMs) Advanced topics and emerging trendsDistributed inference Model parallelism in vLLM/TensorRT NVIDIA Dynamo for disaggregated serving Hybrid approaches Combining quantization and KV caching (e.g., TGI + GPTQ)Speculative decoding (faster-than-greedy sampling) Ethical and operational considerations License restrictions (TGI’s licensing shift) Cost optimization (GPU hours versus performance) Summary ReferencesSubscribe for a free eBook
Part 3: GenAI in the Enterprise
Connected LLMs Pattern
Technical requirementsIntroduction to connected LLM systemsScalability issues in monolithic LLMsKnowledge fragmentation in RAGThe case for multi-LLM architectures: specialization versus generalization trade-offsCost-efficiency via task decompositionArchitectures for connected LLMsHierarchical LLM pipelinesRouting logic (expert selection)Case study: OpenAI’s Mixture-of-ExpertsAgentic workflowsAutonomous agents (AutoGPT and BabyAGI)Human-in-the-loop orchestrationDistributed LLM networksCross-model knowledge sharingFederated learning for decentralized LLMsKey enabling technologiesOrchestration frameworksLangChain/LlamaIndex for multi-LLM coordinationDSPy for programmable pipelinesDynamic routingSemantic router (clustering queries by intent)Reinforcement learning-based routingMemory and state managementShared KV caches across LLMsDistributed vector databases for context passingPerformance optimizationLatency reduction techniquesParallel inference with vLLM/TensorRT-LLMPrefetching for dependent LLM callsCost efficiencySmall LLM gatekeepers for query filteringSpot instance utilization for bursty workloadsAdvanced patternsSelf-correcting LLM loopsCritique and refine (e.g., constitutional AI)Cross-model validationRecursive task decompositionTree-of-thought with multi-LLM votingLLM-generated synthetic training dataHybrid symbolic-LLM systemsNeuro-symbolic integration (e.g., Code + LLMs)Verification via formal methodsSummaryReferences
Monitoring LLMs in Production
Technical requirementsStrategies for continuous system monitoringIntroduction to LLM observabilityKey metrics for monitoring LLMsTools and frameworks for observabilityScalability and performance optimizationBuilding reliable and robust LLM systemsFailure modes and fallbacksTesting strategiesRedundancy architecturesUnderstanding observability and monitoringSecuring LLMs – privacy, threats, and complianceData leakage preventionAdversarial threatsAuditabilityOptimizing costs and scaling deploymentsCost drivers and efficiency techniquesScaling architectures and deployment patternsGlobal deployment considerationsField insights and the future of LLM operationsPractical examples across industriesEmerging operational best practicesFuture directions and innovationsSummaryReferencesSubscribe for a free eBook
Responsible AI in LLMs
Technical requirementsDefining responsible AI in LLMsCore pillars: Fairness, transparency, accountability, and safety Why LLMs pose unique ethical challenges The business and societal case for RAI The evolving regulatory landscape for AIReputation and trust in AI systemsRAI implementation frameworkPhase 1 – fairness by designPhase 2 – transparency instrumentationPhase 3 – continuous complianceEthical considerations in LLMsBias and fairnessMitigation strategiesPre-processing – data sanitizationIn-process – adversarial debiasingPost hoc – calibrated output filteringPrivacy risksDifferential privacyFederated learning for sensitive domainsMisinformation and harmful contentJailbreaking attack taxonomyReal-time content moderation systemImplementing transparency and accountabilityModel transparencyExplainability toolsDocumenting model behaviorAuditability and governanceDecision traceability architectureThird-party audit frameworkAudit dashboard metricsSafety and robustness in LLMsAdversarial robustnessRed-teaming LLMs Automated jailbreak detectionDefensive strategiesRLHF for safety alignmentFail-safes and guardrailsConstitutional AI implementationReal-time monitoringHuman oversight protocolSummaryReferences
Emerging Trends and Multimodality
Technical requirementsEmerging trends in AIThe shift beyond text – why multimodality mattersFrom GPT to Gemini – evolution of AI capabilitiesKey drivers (data availability, hardware advances, and user demand)The multimodal landscapeText and image (CLIP and Flamingo)Text and audio (Whisper and AudioLM)Text and video (Phenaki and VideoPoet)Understanding multimodality in LLMsArchitectural foundationsUnified embedding spacesCross-modal attention mechanismsModality-specific encoders/decodersTraining paradigmsContrastive learningJoint pretraining vs modular fine-tuningEmergent properties of multimodal modelsTechnological advances in multimodal AIBreakthrough modelsGPT-4V – vision-language integrationGemini 1.5 – native multimodalityOpen-source alternatives (LlaVA and OpenFlamingo)Key innovationsTokenization across modalitiesEfficient fusion techniques (Q-Former and Perceiver Resampler)Scalability challenges (compute and memory)A multimodal use case in the medical domainSystem architecture overviewMultimodal fusion mechanismEvidence integration and reasoningOutput generation and explanationClinical validation frameworkContinuous learning systemSummaryReferencesSubscribe for a free eBook
Other Books You May Enjoy
Share Your ThoughtsJoin our Discord and Reddit space
Index

Content preview from LLMs in Enterprise

12 Monitoring LLMs in Production

Large language models (LLMs) have moved from research labs into real-world applications, powering tools in customer support, search, education, coding, healthcare, and more. However, deploying LLMs in production isn’t merely a matter of plugging in an API. It involves engineering systems that are reliable, observable, secure, cost-efficient, and resilient to unpredictable behavior.

This chapter explores the key pillars of operating LLMs in production, offering insights from real deployments and highlighting best practices, patterns, and pitfalls.

Technical requirements

Before implementing the data strategies for LLMs discussed in this chapter, ensure you have the necessary hardware and software set up.

Hardware ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Publisher Resources

ISBN: 9781836203070

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

LLMs in Enterprise

by Ahmed Menshawy, Mahmoud Fahmy

12

Monitoring LLMs in Production

Technical requirements

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.