book

Data Engineering with Generative and Agentic AI on AWS: Building an AI-Augmented Data Practice for the Enterprise

Name: Data Engineering with Generative and Agentic AI on AWS: Building an AI-Augmented Data Practice for the Enterprise
Author: Justin J. Leto
ISBN: 9798868821998

by Justin J. Leto

May 2026

Beginner to intermediate

619 pages

12h 50m

English

Apress

Read now

Unlock full access

Data Engineering with Generative and Agentic AI on AWS
Foreword
Introduction
Why I Wrote This Book?Why Is This Book Different?Who Is This Book For?How the Book Is OrganizedUse of Generative AI in Writing This BookHow I Approach Agentic AIAI Coding for Hands-On LearningDon’t Give Up
Acknowledgments
Table of Contents
About the Author
About the Technical Reviewer
1. Introduction to Data Engineering with Generative and Agentic AI on AWS
What Is CoveredWhat Is Data Engineering?PeopleProcessTechnologyThe Future of Knowledge WorkAWS Cloud for Business AgilityThe Rise of Generative and Agentic AIWhat Is Generative and Agentic AI?The Transformer ArchitectureAnaphora ResolutionLong-Range DependencyEntity ResolutionRetrieval-Augmented Generation (RAG)Model Context Protocol (MCP)Agentic AI Development with KiroAgentic AI for a Modern Data StrategyThe Business of DataData As a Strategic AssetData-Driven Decision MakingData-Driven RevenueDark DataBusiness AlignmentTheoretical Concepts in Data EngineeringDistributed ComputingThe CAP TheoremConsistencyAvailabilityPartition ToleranceConsistency-Partition Tolerance (CP)Availability-Partition Tolerance (AP)ScalabilitySummary
2. Data Security and Governance
Data GovernanceThe Evolving Data Governance ChallengeThe Growth of Unstructured DataGovernment Regulations for Data and AIData Governance for Generative and Agentic AIThe Shared Responsibility ModelAWS Identity and Access Management (AWS IAM)PrincipalsEntities, Roles, and PoliciesThe Anatomy of an IAM PolicyPrinciple of Least PrivilegeIAM Authentication for Data ServicesEncryption for Data ProtectionEncryption Key ManagementAWS-Owned Keys and Customer-Managed KeysCustom Key StoreExternal Key StoreEncryption Across RegionsEncrypting Data-at-RestEncrypting Data-in-TransitSensitive Data Detection and RedactionData Governance on AWSData Governance RolesFine-Grained Access Control for DataGenerative and Agentic AI SecuritySupply Chain AttackSensitive Data LeakageConfused DeputyHallucinationPrompt Injection AttackGuardrailsSafeguards in Amazon BedrockAI-Powered Security with the AWS Security AgentModel Context Protocol (MCP) SecurityInbound and Outbound AuthenticationOAuth for MCPMCP Servers As OAuth Resource ServerMCP Client-Server Request and Authorization FlowsMCP Client-Server Initialization and HandshakeMCP Metadata DiscoveryDynamic Client Registration (optional, but not really)Requesting the Authorization CodeExchange the Authorization Code for the Access TokenRequest for MCP Resource with Access TokenConsent Fatigue from the Human-in-the-Loop Approval FlowClient Credentials Flow for Private MCP ClientsIAM Authentication with SigV4 for MCPCompliance and AuditingAWS ConfigAWS CloudTrailAmazon S3 Server LoggingAWS Security HubAWS Audit ManagerDisaster Recovery, High Availability, and BackupsDisaster RecoveryBackup and RestorePilot LightWarm StandbyMultiregion Active-ActiveHigh AvailabilitySummary
3. Data Lake Design with Apache Iceberg and S3 Tables
Separating Analytical and Operational WorkloadsLock-InScalabilityVendor-Agnostic DataThe Rise of Data LakesSeparation of Data and ComputeSchema-on-Write vs. Schema-on-ReadSerialization FormatsText-Based SerializationAdvanced Serialization FormatsCompression Formats and ThroughputComponents of a Data LakeStorage LayerSecurity and Governance LayerMetadata Management LayerData Lifecycle LayerData Ingestion LayerData Processing LayerServing LayerData Lakes on AWSThe Medallion ArchitectureLanding ZoneAudit StageRaw StageOptimized StageConform StageUnified Analytics StagePublished StageData Cataloging with AWS GlueCreate a DatabaseCreate a CrawlerFine-Grained Access Control with AWS Lake FormationOpen Table FormatsTime TravelSchema EvolutionPartition EvolutionConcurrency ControlChoosing an Open Table FormatApache Iceberg and S3 TablesCreate an S3 Table BucketConfigure IAM Policy for S3 TablesDownload and Stage S3 Tables Catalog JAR in S3Configure an S3 Tables Spark SessionWrite Data to S3 TableSelecting from an S3 Tables Bucket TableUpdating an S3 Tables Bucket TableAdvanced Iceberg Queries on S3 TableInspecting Data FilesInspecting SnapshotsUnmanaged Apache Iceberg in S3Create Apache Iceberg TableUpdate an Apache Iceberg TableTime Travel Feature in Apache IcebergMetadata for Apache Iceberg TablesMaintenance for Apache Iceberg TablesOPTIMIZE Iceberg TablesVACUUM Iceberg TablesUnmanaged Apache Iceberg in AWS GlueAWS Glue Spark ConfigurationApache Iceberg Operations in SparkCreate an Iceberg TableWriting Data to an Iceberg TableUpdate an Iceberg TablePerform Time Travel QueriesMaintenance of Iceberg Tables in SparkExpiring SnapshotsVacuuming an Iceberg TableIceberg Table Maintenance AgentDiscovery and Inventory ToolsMetrics Collection ToolsAnalysis and Planning ToolsMaintenance ActionsSummary

4. Data Mesh Design with Amazon DataZone
Limitations of Centralized Data LakesIntroducing the Data MeshData Mesh on AWS with Amazon DataZoneData Governance with DataZoneCatalog Management with Amazon SageMaker CatalogCatalog Management with Amazon DataZonePublishing and Subscription ManagementData LineageData DiscoverabilityReorganizing for a Data Mesh StrategyData Domain OwnersThe Role of the CoEMatrixed Organizational StructureImplement a Data Mesh with Amazon DataZoneAccount GovernanceDataZone Domain AccountData Publisher AccountsSubscriber AccountsCreate and Configure an Amazon DataZone DomainAssociate AccountsConfigure Lake Formation Hybrid Access Mode for PublishersPublishing DataCreating a Data SourceDiscovering Data AssetsGenerating Metadata for Our Data AssetCreating a Data ProductPublishing the Data ProductConsuming DataDiscovering Data ProductsSubscribing to Data ProductsReviewing the Subscription RequestCreate an Asset FilterUtilizing DataSummary
5. Big Data Processing and Transformation with AWS Glue and AI Agents
Big Data Processing in the CloudEfficiencyScalabilityData Solution Development LifecycleData Lake IngestionChange Data CaptureCDC with AWS Database Migration ServiceAPI ServiceAPI Ingestion with AWS LambdaSFTP Server with AWS Transfer FamilyData Profiling and AnalysisData StructureData QualityDescriptive StatisticsRelationships Between VariablesData IntegrityProfiling and Analysis with Glue DataBrew and AI AgentsCreating an AWS Glue DataBrew Profiling JobData Profiling Analysis with Agentic AIData Profiling Q&A with Amazon Q for BusinessETL Design and DevelopmentVisual ETL Design ToolsAWS Glue DataBrewAWS Glue StudioAmazon SageMaker Unified StudioNotebook-Based Development with Glue Interactive SessionsSageMaker CatalogAI and MLAI-Assisted ETL DevelopmentData Engineering with KiroData Quality with DeequDeequ AnalyzersDeequ ProfilerDeequ Constraint SuggestionCreate a Glue Job with Constraint Verification CodeAuto Scaling for Glue ETL JobsDetecting Schema ChangesAmazon Bedrock Batch InferenceSummary
6. Data Pipeline Orchestration and Observability
Best Practices for Data Pipeline OrchestrationSeparation of ConcernsEncapsulationInteroperability and Control Flow ManagementSchema EvolutionTime to AnswerOrchestrating Data Pipelines with AWSAWS Step FunctionsStandard and Express WorkflowsOrchestrating Workflows with AWS Step FunctionsCreate IAM Policies and RolesCreate AWS Step Functions WorkflowAmazon Managed Workflows for Apache AirflowAmazon MWAA ArchitectureAmazon MWAA EnvironmentOrchestrating Workflows with Amazon MWAAValidating the DAG with an MWAA Test EnvironmentObservability for Data PipelinesLoggingMWAA LoggingAWS Glue LogsCustom LogsMetricsMWAA MetricsAWS Glue MetricsCustom MetricsThe Embedding Metric FormatTracesTraces in AWS GlueDistributing the Trace VariablesAWS Glue JobsAmazon EMR JobsEnhancing Observability with Generative AIAggregated Log Insight ReportingIntroducing the AWS DevOps AgentCreate an Agent SpaceManaging Agent Access to AWS ResourcesEnable Web App AccessManaging Agent CapabilitiesOperator AccessSummary
7. Multimodal Data Extraction and Enrichment with Amazon Bedrock and AWS ML Services
Amazon Bedrock Data AutomationBlueprints for BDABDA ProjectsIntelligent Document Processing (IDP)ClassificationExtractionEnrichmentImage AnalysisObject DetectionImage PropertiesFacial and Emotion RecognitionAudio AnalysisVideo AnalysisSummary
8. Retrieval-Augmented Generation (RAG) with S3 Vectors and Vector Databases
Overview of RAG ArchitectureManaged RAGAmazon Q for BusinessAmazon Bedrock Knowledge BasesData SourcesEmbeddings ModelChunking StrategyCorpus Collection Process and StorageAmazon S3 MetadataCollection Embeddings ProcessAmazon Nova Multimodal EmbeddingsText EmbeddingsFlatten JSON ObjectChunking the ContentDefining the Input ParametersInvoking the ModelParse and Save the ResponseImage EmbeddingsConvert Image to Base64Defining the Input ParametersVideo EmbeddingsDefining the Input ParametersAudio EmbeddingsDefining the Input ParametersVector Databases and Semantic SearchVector Format and PrecisionDistance MetricsEuclidean (L2) DistanceCosine SimilarityInner ProductSemantic Search AlgorithmsHierarchical Navigable Small World (HNSW)Locality-Sensitive Hashing (LSH)Product Quantization (PQ)Similarity Search LibrariesNonmetric Space Library (NMSLib)Facebook AI Similarity Search (FAISS)Common Semantic Search ArchitecturesVector Databases on AWSAmazon S3 VectorsS3 Vectors Buckets and IndexesLoading and Querying Vectors in S3 VectorsIntegrating S3 Vectors with Bedrock Knowledge BasesIntegrating S3 Vectors with Amazon OpenSearchAurora PostgreSQL with Pgvector ExtensionInstall Vector ExtensionCreate Vector TablesContent Metadata TableText Embeddings TableCreate Embeddings IndexInsert EmbeddingsAmazon OpenSearch Serverless with Vector EngineCreate Vector Search CollectionInstall OpenSearch Client LibraryCreate Vector Embeddings IndexSemantic SearchGenerate Embeddings from Query TextSemantic Search in Aurora PostgreSQLSemantic Search in OpenSearch ServerlessConstruct Search QueryExecute SearchOpenSearch Neural Search PluginSummary
9. Streaming and Real-Time Data Processing with Generative AI Enrichment
What Is Streaming Data?Streaming IngestionStreaming Ingestion with Amazon KinesisKinesis Producer Library (KPL)Kinesis Client Library (KCL)Enrich Streaming Data with Generative AILatencyScaling InferenceStreaming Data into Transactional S3 TablesCreate S3Tables Bucket and Managed Iceberg TableLink S3Tables Namespace to Glue Catalog Using a Resource LinkCreate an IAM Role for FirehoseSet Lake Formation PermissionsCreate Firehose Delivery StreamTrend Analysis AgentSummary
10. Data Warehousing with Generative AI and Text-to-SQL Reporting with Amazon Redshift
Impact of Generative AI on Data WarehousesData Warehousing FundamentalsData Scaling and the JOIN ProblemSeparating Analytical WorkloadsOptimized Data Modeling and Organization for ReportingOLAP and Star SchemaColumnar StorageParallelismNetworkingCompressionAmazon RedshiftRedshift ArchitectureRedshift ServerlessRedshift Workload ManagementRedshift Table Design ConsiderationsNo Enforceable ConstraintsSort KeysSingle Sort KeyCompound Sort KeyInterleaved Sort KeyDistribution StylesChange Data Capture (CDC) with ZeroETLAutomated Batch Loading with Auto-CopySales Datamart DemoRedshift Query Editor v2Prompt Engineering for Text-to-SQLStructured Input FormattingText-to-SQL Solution ArchitectureText-to-SQL AgentSQL Evaluation AgentText-to-SQL InterfaceGenerating Report SuggestionsRedshift Integration with Amazon BedrockIAM Policy to Authorize Bedrock AccessCreate External ModelGenerate Sentiment Scores with Bedrock ModelRedshift Integration with Amazon SageMakerIAM Role and Policy to Authorize SageMaker AccessCreate External ModelGenerate Sentiment Scores with Custom ModelRedshift for Retrieval-Augmented Generation (RAG)Redshift ML for Predictive Model TrainingCreate an S3 Bucket for Training Data ExportIAM Role and Policy for SageMaker Model TrainingGrant User Access to Model Creation and SchemaTraining the ModelModel PerformanceBatch InferenceSummary
11. Generative Business Intelligence with Amazon Quick Suite
The Origins of Business IntelligenceAmazon Quick Suite for unified intelligenceChat Agents in Quick SuiteQuick Spaces in Quick SuiteQuick Flows in Quick SuiteQuick Research in Quick SuiteQuick Automate in Quick SuiteGenerative BI with Amazon QuickSightDynamic Reporting with QuickSight QPrerequisitesDatasets in QuickSightBuild a Visual with QStories in QuickSightTopics in QuickSightNamed Entity FeatureAdministering Amazon Quick SuiteCreate an Amazon Quick Suite Account and Sign UpConfigure Authentication Method and Identity IntegrationEstablish SAML Federation ConnectionConfigure User Provisioning and Role AssignmentImplement VPC Integration for Enterprise SecurityConfigure Data Source Connections and SecurityImplement Hierarchical Permission StructureConfigure Data Access Controls and Security BoundariesUser Lifecycle Management and Request ApprovalSummary
12. Building AI Agents with Bedrock AgentCore, Strands Agents, and Model Context Protocol (MCP)
Strands Agents SDKModel Context Protocol (MCP)Amazon Bedrock AgentCoreMCP Server FundamentalsMCP SDKs, Frameworks, and Language SupportLocal and Remote MCP Transport MechanismsMCP Message ProtocolMCP Message StructureMCP RequestsMCP ResponsesMCP NotificationsMCP SecurityMCP Server Deployment on AWSAgentCore Gateway for MCPLambda Function Target for AgentCore GatewayLambda MCP Request EntrypointMCP Request RoutingProcess the MCP RequestInitialize RequestList Tools RequestCall Tools RequestAgentCore Gateway ConfigurationBuild and Deploy Agentic AI Solutions on AWSBuild and Deploy an HTTP MCP ServerCreate Redshift HTTP MCP Server ScriptDeploy and Test Redshift HTTP MCP Server LocallyDeploy the Redshift MCP Server to AWSConfiguring Amazon Cognito for MCP AuthenticationCreate IAM Execution Role for Redshift HTTP MCP ServerConfiguring IAM Authentication for Amazon RedshiftDeploying MCP on AgentCore RuntimeConfigure AgentCore RuntimeLaunch AgentCore RuntimeTesting the Remote HTTP MCP ServerConstruct the URLConstruct the HeadersConstruct the MCP RequestBuild and Deploy Sales Reporting AI AgentCreate Sales Agent ScriptImport StatementsTool and Function DefinitionCreate Memory Hooks for Saving and Injecting User ContextSales Agent DefinitionAuthenticate to the Redshift MCP ServerConstruct the Request to the Redshift MCP ServerTest the Sales Agent Script LocallyDeploy Sales Reporting Agent to AgentCore RuntimeTesting the Sales Reporting Agent on AgentCore RuntimeObservability and Monitoring for AI AgentsSummary
Index

Content preview from Data Engineering with Generative and Agentic AI on AWS: Building an AI-Augmented Data Practice for the Enterprise

J. J. LetoData Engineering with Generative and Agentic AI on AWShttps://doi.org/10.1007/979-8-8688-2199-8_8

8. Retrieval-Augmented Generation (RAG) with S3 Vectors and Vector Databases

Justin J. Leto¹

Justin J. Leto

Email: leto.justin@gmail.com

Retrieval-augmented generation (or RAG) is an advanced technique for augmenting the capabilities of a generative AI model by integrating external knowledge bases that contain up-to-date, task- or domain-specific information. Through this approach, RAG improves the accuracy and relevance of generated content, reduces hallucinations, and enables more context-aware responses.

Due to the resources required (cost and time) to train robust generative AI ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Publisher Resources

ISBN: 9798868821998

Cloud Computing