book

AI Under Attack

Name: AI Under Attack
ISBN: 9781806119936

by Kris Kimmerle, David Okeyode

June 2026

Intermediate

478 pages

12h 25m

English

Packt Publishing

Read now

Unlock full access

Preface
Free benefits with your book
Part 1: Foundations
Chapter 1: Why AI Security Is Different
The probabilistic nature of AIThe paradigm shift in AI system behaviorThe Replit database deletion storyEmergent behaviors as a security riskThe rise of mythos-class modelsWhen model behavior driftsAlignment decayFine-tuning can destabilize alignmentCompounding effectsWhere AI security fails in practiceRemote code execution in Meta's Llama frameworkPrompt-injection remote code execution in the Vanna.AI text-to-SQL libraryPolicy puppetry universal prompt injectionWhat these incidents teach usThe AI security mindsetSummaryLab activityReferences
Chapter 2: Threat Modeling AI Systems
Understanding threat modelingTime-tested threat modeling frameworksTwelve threat modeling methodsAI-specific attack surfacesPrompt injection and insecure output handlingTraining data poisoning and model theftPrivacy leakageAutonomous agent risksStep-by-step recipe for threat modeling AI systemsStep 1: Define scope and goalsStep 2: Draw a simple data-flow diagramStep 3: Layer on STRIDE plus AI-specific threatsStep 4: Estimate impact and likelihood with quick scoringStep 5: Pick controlsStep 6: Document open questions and residual riskStep 7: Schedule reviews and triggers for updatesRunning a workshopWho to inviteKeeping sessions focusedSample two-hour agendaIllustrated case study: StyleBotStep 1: Define scope and goalsStep 2: Data-flow diagramStep 3: Apply STRIDE with AI enhancementsProduction reality: the same STRIDE pass on a regulated agentStep 4: Risk scoringStep 5: Control selectionStep 6: Open questions and residual risksStep 7: Review scheduleTooling guideManual toolsAutomated platformsTool selection summaryCommon pitfalls and quick winsPitfalls to avoidQuick wins that build momentumWhat success looks likeIndicators of a living threat modelSigns the process is slippingAligning with business goalsSummaryLab activityReferences
Chapter 3: The AI Attack Surface
Mapping AI interfaces, inputs, outputs, memory, and pluginsInput prompt as the primary vectorOutputs beyond text responsesMemory and context: The ephemeral statePlugins and toolsRisks associated with user-facing prompts and APIsThe infinite input space of promptsAPI security in the AI eraUnderstanding tool plugin integrations as attack vectorsThe confused deputy problemInsecure tool design and executionExcessive permissions and scope creepContrasting training time vs inference time attack surfacesTraining time attack surfaceInference time attack surfaceThe interconnected surfaceReal-world examples of AI attack surface exploitationExploiting tool integrationsModel extraction via APITraining data poisoning: Nightshade and beyondPrinciples for minimizing the AI attack surfaceApply least privilege rigorouslyImplement defense in depth for inputs and outputsIsolate and sandbox execution environmentsMaintain provenance and auditabilitySummaryLab activityReferences
Chapter 4: Foundations of AI Defense
The four pillars of AI defenseRobustness against adversarial manipulationContainment and isolation strategiesObservability and monitoringInterpretability for security analysisAdapting traditional security controls for AIControl adaptation at a glanceAuthentication and authorization in AI contextsInput validation and API protectionLogging and auditing considerationsBuilding defense strategies against identified attack surfacesDefending the prompt interfaceSecuring outputs, memory, and stateSecuring tool and plugin integrationsTesting and validating AI defensesRed team exercises for AIAutomated adversarial testingMeasuring defense effectivenessImplementing layered defense strategiesGraceful degradation under attackSummaryLab activityReferences
Part 2: Attacks and Agent Behavior
Chapter 5: Anatomy of an AI System
From classical foundations to generative complexityTraditional ML blueprintGenerative transformationArchitectural inheritance and divergenceThe modern AI training pipelineData preparation and curationMulti-stage training processInfrastructure dependenciesIntegration patterns and their vulnerabilitiesThe API gateway patternRetrieval-augmented generation architectureAgent architectures and tool useMemory and state managementOrchestration frameworksInfrastructure layers and security boundariesTraining infrastructureInference infrastructureData pipeline infrastructureNetwork architecture and segmentationTrust boundaries and critical interfacesPrimary trust boundariesTemporal and state boundariesVisualization and system mappingSummaryLab activityReferences
Chapter 6: Prompt Injection and Jailbreaking
The infinite input space problemDirect prompt injection mechanicsInstruction override attacksRole, delimiter, token, and context attacksIndirect prompt injectionCommon indirect injection channelsJailbreaking techniquesPersona, logic, multi-turn, and obfuscation patternsWhy traditional defenses failThe keyword filtering illusionDefensive prompt engineeringA compact defensive patternTesting your promptsRuntime detection and responseDetection layersResponse and operationsTesting and continuous validationFrom Attack Strings to Regression CasesAdversarial testing and red teamsValidation in productionSummaryLab activityReferences
Chapter 7: Memory, Context, and State Abuse
The statefulness problemSession-level attacksContext poisoningConversation hijackingHistory manipulationCross-session persistence attacksLong-term memory poisoningProfile manipulationKnowledge base contaminationCross-user contaminationShared context leakageModel-level state pollutionCache poisoningWhy memory makes defense harderTemporal obfuscationState explosionDefensive memory architectureSession boundariesUser partitioningMemory tieringMemory forensics and incident responseTrajectory analysisAudit logs and anomaly indicatorsContainment and recoverySummaryLab activityReferences

Chapter 8: Attacks on RAG Systems
RAG architecture and its attack surfaceDocument source poisoningEmbedding and vector database attacksRetrieval manipulation and indirect prompt injectionReal-world case study: when the grounding store is a database row, not a vector indexAccess control and authorization attacksDetection and monitoringDefensive architecture and mitigationsTrade-offs and practical guidanceSummaryLab activityReferences
Chapter 9: Agent Architecture and Vulnerabilities
From responses to actionsFour critical handoffsBetween user inputs and agent instructionsBetween agent and connected toolsBetween agent and memory storageBetween different agentsCommon vulnerabilitiesUnsafe tool execution and privilege escalationPrompt injectionLeaky or unsafe output handlingMemory and data persistence risksOverly broad and long-lived credentialsLayer-by-layer threat analysisInfrastructure layerModel and memory layerOrchestration and tools layerInterface layerSummaryReferences
Chapter 10: Agent Exploitation Techniques
Tool abuse and chaining attacksSequential escalation and inter-tool injectionPermission aggregationIn production, the tools usually live below the modelTool-specific exploitation pathsCritical walkthrough: multi-step attack chainGoal hijacking and reasoning manipulationMulti-agent attack scenariosDetection and responseGoal and action coherenceReasoning and premise checksResource and communication signalsBaselines and responseSummaryLab activityReferences
Chapter 11: Attacks on Training Data and Model Integrity
The upstream compromiseData poisoning attack patternsLabel flipping attacksBackdoor attacksClean-label and availability attacksModel supply chain attacksBackdoor detection techniquesBehavioral and statistical testingAutomated scanning and provider assessmentRuntime monitoringModel extraction and stealingFine-tuning risksDetecting compromised models in productionSummaryLab activityReferences
Chapter 12: AI Red Teaming and Evaluation
Why traditional red teaming fails against AIWhat traditional testing validatesWhat traditional testing missesThe fundamental mismatchBuilding an AI red team programTeam composition and structureScope, authorization, and rules of engagementTesting cadenceManual red team techniquesTesting by attack surfaceDocumenting findingsAutomated adversarial testingTooling and benchmarksCI/CD integration and automated agentsEvaluation metrics for AI securityMetrics to trackComposite risk without false precisionReporting metrics to stakeholdersIndustry frameworks and standards for red teamsReporting and remediationReport structure and severityDriving remediationContinuous red team practiceMaturity modelBuilding adversarial mindsetIndustry collaborationSummaryLab activityReferences
Part 3: Engineering Defenses
Chapter 13: Securing AI Infrastructure, MLOps, and Runtime Environments
Why AI infrastructure differsCloud security foundations for AISegmentation and storageTraining and inference boundariesMLOps pipeline securityData preparation and trainingEvaluation, registry, deployment, and monitoringContainer and orchestration hardeningInfrastructure as code for AISecrets, scanning, and driftModel artifact lifecycle securitySandboxing and isolation techniquesRuntime policy enforcementSummaryReferences
Chapter 14: Building Secure AI Applications and APIs
Secure API design patterns for AIGateway and business logicInference isolation and versioningAuthorization and resource controlsConversation and RAG authorizationRate limits, budgets, and cost controlsInput validation for AI applicationsLayer 1: Structural validationLayer 2: Content-based validationLayer 3: Semantic intent classificationLayer 4: Contextual validationOutput sanitization and leakage preventionPII and prompt leakageSafe renderingSecuring tool and plugin integrationsLeast privilege and sandboxingParameter validationHuman approvalUI, upload, and end-to-end controlsSession and approval safetySecure file upload for RAGAutomated security testing pipelineSummaryLab activityReferences
Chapter 15: Defensive Prompt Engineering
The three lines of defenseDefensive prompt engineering fundamentalsInstruction hierarchyBoundary markers and enclosureMeta-instructions against common attacksPost-promptingContextual constraintsPractical production templateTesting your defensive promptsAdvanced prompt hardening techniquesSigned instructionsMulti-turn consistency checkingRetrieval-augmented safetyAdversarial example integrationConfidence-based refusalPrompt compression resistanceEffectiveness comparisonSummaryLab activityReferences
Chapter 16: Guardrails and Human Oversight
Input guardrails and classificationThe guardrail stackClassifiers, semantic checks, and ensemblesFailure behavior and telemetryOutput guardrails and validationContent safety, secrets, and prompt leakageGrounding and format safetyHuman-centric design patternsHitl vs hotlConfidence-gated escalationFeedback loopsContinuous evaluation and improvementThe red team loopAutomated adversarial testingRegression and deployment decisionsChoosing and integrating guardrail solutionsBuy/build checklistDeployment and cost disciplineSummaryLab activityReferences
Chapter 17: External Dependencies, Supply Chain Integrity and Multi-Tenant Risk
The AI supply chain attack surfaceTrust is transitiveModel provenance and validationInternal model registryCryptographic model signingHeadless signing in CI/CDKubernetes validation without YAML sprawlThe pickle serialization threatScanner limitsMigrating to safetensorsDependency scanning and managementSBOM, VEX, and build evidenceThird-party model and API risk assessmentMulti-tenant architecture securityThe economics of isolationVector database isolation patternsGPU, prompt, and runtime isolation: GPU isolation with NVIDIA MIGPrompt-level tenant enforcementMonitoring external dependenciesIncident response for supply chain compromisesA compact response flowBuilding a secure supply chain programSummaryLab activityReferences
Chapter 18: Securing AI Agents with Zero Trust Architecture
The agent authority problemLegitimate action can still be harmfulTwo identity modes for agentsDelegated human permissionsNon-human agent identitiesThe split-brain failureMap the permission boundaryThe boundary decompositionAuthority inventory questionsContext and memory are part of the boundaryScoped permissions and capability boundariesScope by verb, target, and timeLeast agency when the model looks powerlessCapability tokensApproval gatesPolicy enforcement before actionWhat the policy needs to seePolicy enforcement points and policy decisionsMCP and tool serversA minimal tool-call recordRuntime isolation and segmentationSegmentation patternsContinuous verification and auditAudit evidenceIncident response for authorized agent harmThe containment trapImplementation roadmapWhat good looks likeSummaryLab activityReferences
Part 4: Programs, Operations, and Future State
Chapter 19: AI Governance, Risk, and Compliance
Defining the governance problemGovernanceRisk managementComplianceCreators, adopters, and shared responsibilityCreator responsibilitiesAdopter responsibilitiesProcurement and vendor reviewWhy AI governance differs from IT governanceProbabilistic behaviorOpacityDrift and emerging biasesResponsible AI as an operating floorTranslate principles into evidenceGovernance frameworks that scaleGovernance tiersPolicy frameworkTechnical enforcementException handlingAI risk assessmentAssessment methodologyContinuous monitoringEvidence artifactsAgent authority as a governance problemDelegated human permissionsNon-Human agent identitiesMinimum evidence for agent approvalRisk acceptance for agent authorityNavigating compliance requirementsEU AI act and GPAI obligationsNIST, ISO, and management systemsSector, state, and local requirementsUS federal policyRoles, accountability, and lifecycle integrationAccountability assignmentExecutive risk acceptanceLifecycle gatesMetricsSecuring resourcesSummaryReferences
Chapter 20: AI Security Engineering
GPU infrastructure securityGPU memory architectureMulti-Tenancy choicesSide channels and cluster designGPU isolation decision recordContainer and kubernetes security for ML workloadsResource isolationNetwork policiesSecurity contexts and pod security standardsSecrets in kubernetesAdmission and runtime evidenceModel serving and registry securityInference endpoint securityModel registry securityServing framework hardeningModel loading gateVector database and retrieval securityAccess control patternsEncryption, retention, and monitoringEmbedding and retrieval abuse casesMLOps pipeline securityTraining and artifact controlSoftware, model, and infrastructure supply chainsPromotion evidenceCloud AI platform and secrets securityShared responsibility and identityNetwork and data protectionAgent authority engineeringPolicy before tool executionService account boundariesRevocation pathsObservability and network architectureOpenTelemetry for AI workloadsSecurity monitoring signalsNetwork segmentationService mesh and evidenceEvidence chain for AI actionsSummaryLab activityReferences
Chapter 21: AI Security Operations
Why AI security operations differThe evidence chainAI telemetry collectionPrompt and response loggingBehavioral telemetryInfrastructure and data telemetryRetention and access gatesMinimum action eventDetection strategiesSignature-Based detectionAnomaly and semantic detectionCorrelation and contextDetection quality reviewAlert design and managementSeverity and suppressionIncident response playbooksPrompt injection responseModel extraction and memory poisoning responsesAgent compromise responseEvidence preservation before cleanupRevocation and rollback checksResponding to AI-Enabled threatsDeepfake detection and responseAutomated social engineeringAdaptive and evasive malwareOperational boundaryPost-Incident analysis and improvementLessons learnedControl backlogIntegrating AI security into SOC operationsShared infrastructureAnalyst training and coordinationAI case record fieldsTabletop and drill cadenceContinuous improvementSummaryReferences
Chapter 22: Building Sustainable AI Security Programs
Why AI security programs failChampion dependencyBudget fragilityCapability atrophyIsolationAuthority DriftOrganizational design for sustainabilityReporting StructureProgram charterRisk tiers and minimum evidenceTeam compositionBuilding and retaining talentSkills DevelopmentRole familiesCareer developmentKnowledge retentionMetrics and demonstrating valueSelecting meaningful metricsMetric calibrationDashboards and reportingDemonstrating ROICross-functional collaborationPartnering with engineering and productUsing AI to sustain an AI security programAutomation guardrailsExecutive EngagementChange management and cultureBuilding security cultureImplementing changeAdapting as the field changesTechnology and threat evolutionRegulatory and standards evolutionStandards watch rhythmBuilding program resilienceSuccession planningInstitutional memoryCrisis preparednessContinuous renewalMinimum viable evidenceSummaryReferences
Chapter 23: The Future of AI Security
The converging futureWhat researchers agree onImplications for securityThe scaling era and beyondFrom tools to agents to autonomous systemsAI as a toolAI as an agentAI as an autonomous systemThe transition timelineWhen machines fight machinesThe emerging arms raceDynamics of AI-vs-AI conflictWhat this means for defendersAlignment in adversarial contextsDefensive alignment challengesOffensive alignment challengesAlignment in competitionApproaches to security alignmentThe changing role of human practitionersFrom Execution to OversightSetting objectives and constraintsException handlingStrategic functionsPreparing for the transitionNavigating UncertaintyWhat we can predictWhat we cannot predictStrategies for UncertaintyLooking forwardThe longer horizonYour role in shaping the futureClosing reflectionsReferences
Chapter 24: Unlock Your Exclusive Benefits
Unlock this Book's Free Benefits in 3 Easy Steps
Chapter 25: Practitioner Toolkit
Other Books You May Enjoy
Index

Content preview from AI Under Attack

16 Guardrails and Human Oversight

The prompt did its job until it met content that looked like part of the job. A support assistant retrieved a ticket, saw a hidden instruction inside the customer note, and tried to follow it. The model was not broken in the ordinary software sense. It was doing what language models do: treating text as context and trying to be useful.

That is why Chapter 15 cannot be the whole defense. Defensive prompts reduce ambiguity, but production systems still need controls outside the model: guardrails that inspect inputs and outputs, policy checks that sit in the workflow, and human oversight for actions where automation should not get the last word.

A prompt can tell the model not to reveal customer data. A guardrail ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Publisher Resources

ISBN: 9781806119936

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

AI Under Attack

by Kris Kimmerle, David Okeyode

16

Guardrails and Human Oversight

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.