book

AI-Native LLM Security

Name: AI-Native LLM Security
ISBN: 9781836203759

by Vaibhav Malik, Ken Huang, Ads Dawson

December 2025

Intermediate to advanced

416 pages

13h 46m

English

Packt Publishing

Read now

Unlock full access

AI-Native LLM Security
Contributors
About the authors
About the reviewers
Preface
Who this book is forWhat this book coversTo get the most out of this bookDownload the color imagesConventions usedGet in touchFree Benefits with Your Book
Part 1: Foundations of LLM Security
Chapter 1: Fundamentals and Introduction to Large Language Models
The evolution and impact of AI – from foundations to LLMsAIMultimodal modelsMultilingual modelsMachine learning – the foundation of modern AIDeep learning – the power of hierarchical representationGenerative AI – unleashing machine creativityArtificial general intelligence – the next frontierLLMs – an overviewWhat are LLMs?Natural language processing – bridging human and machine communicationHow do LLMs work?Tokenization – the foundation of language processingArchitecture of LLMsTraining process – from raw text to language masteryAdvanced techniques and enhancements in LLMsFew-shot and zero-shot learning – the power of generalizationWhat is RAG?What is fine-tuning?SummaryFurther reading
2
Securing Large Language Models AI-native LLM security – safeguarding the future of AIUnderstanding AI-native securityCritical principles of AI-native securityComponents of AI-native LLM securityThe unique challenges of securing LLMsAdversarial attacksData poisoningModel inversion and extractionPrivacy concernsEthical and legal complianceExplainability and transparencyScalability of security measuresSupply chain securityLLMs – state-of-the-art applications and emerging trendsContent creation and journalismCustomer service and conversational AIEducation and personalized learningHealthcare and biomedical researchLegal tech and complianceFinancial services and quantitative analysisSoftware development and DevOpsCreative industries and digital artsScientific research and interdisciplinary studiesLanguage services and global communicationReal-world applications and case studiesEnhancing customer interaction – using LLMs for chatbots and virtual assistantsStreamlining content production – leveraging LLMs for content creation and summarizationBolstering financial security – implementing LLMs for fraud detection in financial servicesAdvancing medical frontiers – applying LLMs in healthcare and medical researchNavigating legal complexities – employing LLMs for legal and regulatory complianceRevolutionizing learning experiences – deploying LLMs for personalized education and tutoringRecent developments and emerging trendsSummary Further reading
3
The Dual Nature of LLM Risks: Inherent Vulnerabilities and Malicious ActorsInherent vulnerabilities: model opacity, bias, and unpredictabilityModel opacityBiasUnpredictabilityMalicious threats: data poisoning, model stealing, and output manipulationData poisoningModel stealingOutput manipulationReal-world examples: healthcare bias and social media moderationScenario 1: LLM bias in healthcare chatbotScenario 2: malicious attacks on social media moderation LLMCritical principles for securing LLMsProactive threat modelingAdversarial testingData provenance and auditingSecure model training and deploymentRobust monitoring and incident responseTakeaways from critical principles SummaryFurther reading
Chapter 4: Mapping Trust Boundaries in LLM Architectures
Understanding trust boundaries in LLM architecturesData-related attack surfaces in LLM architecturesData poisoning: compromising model integrityData leakage: unintended information disclosurePrivacy risks: protecting sensitive informationModel-related attack surfaces in LLM architectures: an in-depth analysisModel theft: stealing intellectual propertyModel tampering: compromising model integrityAdversarial attacks: exploiting model vulnerabilitiesDeployment-related attack surfaces in LLM architecturesAPI vulnerabilities: the gateway to your LLMAccess control: guarding the keys to your LLMMonitoring: vigilance in LLM deploymentsSupply chain risks in LLM architectures: navigating the complexities of third-party dependenciesLibraries and dependencies: the hidden attack surfaceBest practices for mitigating supply chain risksSummaryFurther reading
5
Aligning LLM Security with Organizational Objectives and Regulatory Landscapes Integrating LLM security into enterprise risk management NIST Risk Management Framework for LLM securityKey LLM security risksImplementing LLM risk management strategiesAddressing challenges in integrating LLM security into ERMBest practices for enhancing LLM security within ERM frameworks Navigating legal and regulatory landscapes for LLM deployment Key legal and regulatory areasDevising a compliance strategy and adapting to future regulationsEthical considerations and responsible AI principles for LLMs Key ethical challengesImplementing responsible AI principlesStakeholder engagement and cross-functional collaboration Measuring and communicating LLM security performance Defining key security metricsImplementing security performance dashboardsImplementing measurement frameworks Analyzing and interpreting resultsCommunicating security performanceAddressing security incidentsContinuous learning and improvementSummary Further reading

Part 2: The OWASP Top 10 for LLM Applications
Chapter 6: Identifying and Prioritizing LLM Security Risks with OWASP
Overview of the OWASP Top 10 methodologyAdapting OWASP for LLM applications: key considerationsCriteria for inclusion in the LLM Top 10Using the LLM Top 10 for risk assessment and prioritizationIntegrating the LLM Top 10 into organizational risk managementSummary
Chapter 7: Diving Deep: Profiles of the Top 10 LLM Security Risks
Injection flaws: prompt injection, data poisoning, and model manipulationPrompt Injection (OWASP LLM01)Training Data Poisoning (OWASP LLM03)Model manipulationBroken authentication and session management in LLM systemsUnderlying vulnerabilitiesInput validation and sanitization challengesSession management complexitiesPotential impactsExample of indirect prompt injection with a pluginCode example of insecure plugin designSensitive data exposure – training data leakage and model inversionUnderlying vulnerabilitiesPotential impactsExample of sensitive data exposureData leakage code exampleBroken access control and unauthorized model accessExcessive Agency (OWASP LLM08)Model Theft (OWASP LLM10)Security misconfigurations in LLM deployment environmentsInsecure Output Handling (OWASP LLM02)Model Denial of Service (OWASP LLM04)Supply Chain Vulnerabilities (OWASP LLM05)Overreliance (OWASP LLM09)Code example illustrating vulnerabilitiesSummary
Chapter 8: Mitigating LLM Risks: Strategies and Techniques for Each OWASP Category
Introducing LLM applications into common system architecture and defense-in-depth approachesAdopting a defense-in-depth approach to LLM applications Which way should you shift in the software development life cycle? Machine learning versus traditional application securityDefense-in-depth considerations for your ecosystemPreventing injection flaws: input validation, sanitization, and robust parsingLLM01: Prompt InjectionLLM03: Training Data PoisoningImplementing strong authentication and session management for LLM systemsLLM02: Insecure Output HandlingProtecting sensitive data: encryption, access controls, and data minimizationLLM06: Sensitive Information DisclosureLLM10: Model TheftEnforcing granular access control for LLM models and endpointsLLM07: Insecure Plugin DesignLLM08: Excessive AgencyHardening LLM deployment environments: configuration management and continuous monitoringLLM05: Supply ChainLLM04: Model Denial of ServiceManaging risks of Overreliance (LLM09) on LLM applicationsSummary
Chapter 9: Adapting the OWASP Top 10 to Diverse Deployment Scenarios
Identifying risk profiles for different LLM application typesAdapting the Top 10 to chatbots and conversational AI systemsSaaSCloud AI platformsPrivate deploymentsApplying the Top 10 to content generation and creative AI applicationsScaling the Top 10 for enterprise-wide LLM deployment and governanceFramework for enterprise governanceImplementing best practices for securitySummary
Part 3: Building Secure LLM Systems
10
Designing LLM Systems for Security: Architecture, Controls, and Best PracticesPrinciples of secure LLM system architectureDefense in depth for LLM systemsZero-trust architecture in LLM deploymentsSecurity by design in LLM developmentLLM-specific security considerationsModel protection strategiesInput/output security frameworkResource control and managementLong-term security considerationsReference architecture componentsClient interface layerAuthentication and authorization layerInput validation and sanitization layerLLM orchestration layerModel serving layerOutput processing and safety checks layerDesigning for isolation and least privilegeComponent isolation strategiesImplementing least privilegeSecuring data flows and communication channelsData flow securityCommunication channel securityImplementing robust access controls and authentication mechanismsAuthentication architectureAuthorization frameworkIntegrating security monitoring and response capabilitiesSecurity monitoring infrastructureIncident response integrationSummaryFurther reading
Chapter 11: Integrating Security into the LLM Development Life Cycle: From Data Curation to Deployment
Secure data collection, curation, and preprocessingSecure data collection strategiesData curation for security and qualitySecure preprocessing techniquesCase study – building a secure data pipeline for financial LLMsProtecting model integrity during training and validationSecuring the training environmentPreventing poisoning and backdoor attacksRobust training methodologies for securityValidation and verification for securityCase study – training a secure healthcare LLMConducting rigorous security testing and evaluationDeveloping a comprehensive testing frameworkTesting for prompt injection and jailbreakingEvaluating data privacy and extraction risksSecurity evaluation metrics and standardsCase study – security testing for a legal research LLMSecure deployment and runtime protection measuresSecure deployment architectureAuthentication, authorization, and access controlMonitoring and anomaly detectionRuntime protection techniquesContinuous monitoring, auditing, and incident responseImplementing continuous security monitoringRegular security auditing and assessmentVulnerability management and remediationIncident response for LLM systemsSummaryFurther reading
Chapter 12: Operational Resilience: Monitoring, Incident Response, and Continuous Improvement
Designing comprehensive monitoring and alerting strategiesThe monitoring hierarchy for LLM systemsKey monitoring metrics and indicatorsBuilding a monitoring architectureImplementing effective alerting strategiesCase study – monitoring architecture for an enterprise LLM platformDetecting anomalies and potential security incidents in LLM systemsUnderstanding LLM anomaly typesDetection techniques for LLM securityConfiguring detectors and alarmsDeveloping and executing effective incident response plansFundamentals of LLM incident response planningInvestigation and analysis techniques for LLM incidentsContainment and remediation strategiesConducting post-incident reviews and root cause analysisStructured post-incident review processesRoot cause analysis for LLM security incidentsIdentifying actionable lessons and improvementsDriving continuous improvement of LLM security postureEstablishing a security improvement frameworkLeveraging threat intelligence and emerging best practicesImplementing feedback loops across the LLM life cycleSummaryFurther reading
Chapter 13: The Future of LLM Security: Emerging Threats, Promising Defenses, and the Path Forward
Emerging threats: the next generation of LLM security challengesAttacks on AI agentsDeepfakeAutomated social engineeringAdvanced supply chain attacks and model poisoningZero-day vulnerabilities in LLM frameworksCross-model attacks and transferabilityQuantum computing threatsAI-driven malware and autonomous attacksEthical manipulation and psychological impactsChallenges in explainability and transparencyIntegration with critical infrastructure Globalization of threat actorsEvolution of attack tools and techniquesPromising defensive innovations: research frontiers and cutting-edge techniquesReinforcement learning for safe and accurate LLMsDifferential privacy for protecting sensitive data in LLMsAdversarial training for strengthening LLM defensesExplainable AI (XAI)Secure model architectures for inherent resistance to attacksFederated learning for decentralized training and enhanced privacyAdditional frontier research topics in LLM securityThe evolving regulatory landscape: navigating changing compliance requirementsData privacy regulationsTransparency and explainabilityBias and fairnessInternational regulationsCompliance automationThe importance of collaboration: engaging with the LLM security communityKnowledge sharingThreat intelligenceOpen source initiativePartnerships between academia and industryGlobal collaborationSummary
Appendices: Latest OWASP Top 10 for LLM and OWASP AIVSS Agentic AI Core Risks
Appendix A: OWASP Top 10 for LLM Applications - 2025 Update
Evolution timelineVersion comparison and mappingComplete mapping tableNew additions in 2025Detailed analysis of changesPromoted OWASP Top 10 ItemsSignificantly modified entriesMerged and consolidated entriesArchitecture evolution impact2023 vs. 2025 Architecture focusNew attack vectors addressedEmerging patterns and trendsMulti-vector attacksSocio-technical risksImplementation recommendationsFor organizations using the 2023 guidelinesFuture considerationsDeep dive into the 2025 newcomersLLM07:2025 System Prompt Leakage LLM08:2025 Vector and Embedding Weaknesses – The RAG Supply-Chain Blind SpotConclusion
Appendix B: OWASP AIVSS Core Agentic AI Security Risks
The what and why of the OWASP AIVSS projectHigh-level overview of the OWASP AIVSS core agentic AI risks Deep dive into OWASP AIVSS’s core agentic AI risks itemsAgentic AI Tool MisuseAgent Access-Control ViolationAgent Orchestration and Multi-Agent Exploitation Agent Identity ImpersonationAgent Memory and Context ManipulationInsecure Agent Critical-Systems InteractionAgent Supply Chain and Dependency AttacksAgent UntraceabilityAgent Goal and Instruction ManipulationConclusion
Appendix C: Unlock Your Exclusive Benefits
Unlock this Book’s Free Benefits in 3 Easy StepsStep 1
Index
Why subscribe?
Other Books You May EnjoyPackt is searching for authors like you

Content preview from AI-Native LLM Security

4 Mapping Trust Boundaries in LLM Architectures

In this chapter, we’ll explore the critical aspect of security within LLM architectures. As LLMs continue to revolutionize AI and find applications across various industries, understanding and mapping their trust boundaries becomes paramount. We’ll delve into the unique security challenges posed by LLMs, examining potential vulnerabilities and attack vectors across different layers of their architecture.

The consequences of poorly defined trust boundaries can be severe, as demonstrated by the 2023 Samsung incident where employees inadvertently leaked sensitive code by uploading it to ChatGPT, highlighting how unclear delineation between trusted and untrusted data flows can lead to significant security ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Publisher Resources

ISBN: 9781836203759

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

AI-Native LLM Security

by Vaibhav Malik, Ken Huang, Ads Dawson

4

Mapping Trust Boundaries in LLM Architectures

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.