book

Data Engineering for Multimodal AI

Name: Data Engineering for Multimodal AI
Author: Vasundra Srinivasan
ISBN: 9781098190781

by Vasundra Srinivasan

August 2026

Intermediate to advanced

548 pages

15h 29m

English

O'Reilly Media, Inc.

Read now

Unlock full access

Preface
Why I Wrote This BookHow This Book Was BuiltWho Should Read This BookWhat You Will Get Out of This BookNavigating This BookConventions Used in This BookUsing Code ExamplesO’Reilly Online LearningHow to Contact UsAcknowledgmentsA Note Before You Begin
1. The Multimodal AI Data Engineering Landscape
The Engineering Context of Multimodal AI SystemsData Architecture Components in a Multimodal AI SystemData Contracts: The Foundation of TrustBuilding the Semantic Layer of a Multimodal AI SystemFrom Principle to Practice: Granite’s Layered ArchitectureEntity Resolution and Why It’s HardBuilding the Semantic Layer: A Step-by-Step SummaryKey Challenges in Building Multimodal AI PipelinesIntegration and SynchronizationProcessing at ScaleOperational ResilienceThe Stages of the Multimodal AI Data LifecycleStage 1: Data IngestionStage 2: Staging and OrganizationStage 3: Core ProcessingStage 4: Feature EngineeringStage 5: Model ServingStages 6–8: Continuous Learning and DeploymentStage 9: Monitoring and AnalysisPlaybook: Building a Minimal Semantic Layer PipelineStep 1: Data ExplorationStep 2: Initial Setup and Data PreparationStep 3: Data Ingestion PipelineStep 4: Starting the Semantic API ServerTakeaways
2. Data Architectures for Multimodal AI Systems
Core Patterns for Multimodal Data ArchitecturesVector-Native StorageComposable Data Lakehouse ArchitectureComplementary Strategies: Hybrid Storage and Real-Time ProcessingEmbeddings and Retrieval in Multimodal Data ArchitecturesContainerization and Orchestration for Multimodal AIReliability and Scale in Multimodal AI Data ArchitectureStreaming IngestionScaling StrategiesProduction Patterns for Reliability and ScaleMonitoring and ObservabilityData Fabric and Mesh Architectures for Multimodal SystemsData Fabric FundamentalsData Mesh FundamentalsPlaybook: Building a Multimodal Feature Store for StreamBuyStep 0: Data ExplorationStep 1: Project Setup and ArchitectureStep 2: Data ModelsStep 3: Feature EncodersStep 4: Database ClientsStep 5: Background Vector ProcessingStep 6: API LayerStep 7: Running TestsStep 8: Running the Full StackExtending This PipelineTakeaways
3. Platforms and Patterns for Multimodal AI Data Engineering
Setting the Context: Multimodal Context IntelligenceContext Intelligence in Action: Revault’s Onboarding FlowPlatform Foundations for Multimodal Context InfrastructureManaged Multimodal AI PlatformsComposable Multimodal ContextReal-Time Streaming Infrastructure for Multimodal ContextChoosing Between Archetypes: A Cost and Complexity FrameworkCore System Patterns for Multimodal Context EngineeringMultimodal Representation LayerTemporal State AssemblySelective Context RetrievalCross-Cutting Concerns: Observability, Embeddings, and Agentic Orchestration and Context FlowPlaybook: Building a Serverless Document Retrieval PipelineStage 1: Document Parsing and RepresentationStage 2: Embedding GenerationStage 3: Vector IndexingStage 4: Selective RetrievalScaling for ProductionPlaybook ConclusionTakeaways
4. Ingestion and Transformation for Multimodal AI Systems
Why Is Multimodal Ingestion Fundamentally Harder?Multimodal Ingestion PipelinesTraditional ApproachA Hybrid AI ApproachThe Agentic AI ApproachOperationalizing ELT for Multimodal AIELT Versus ETLHybrid ETL and ELT ArchitecturesPrivileging ELT over ETL at ByteEatsBalancing Cost, Latency, and FidelityValidation, Transformation, Observability, and Annotation Across ModalitiesWhat Happens When Validation FailsFrom Validation to RepresentationObservability: Tracing Signals Through the PipelineAnnotation: Where Multimodal Systems Mature or CollapseIngestion as Context EngineeringPlaybook: Building a Core Multimodal AI Data Operations PipelineStep 1: Order IngestionStep 2: Robot Prep TriggerStep 3: Feature Extraction from Videos to Completion TriggerStep 4: QA TestingStep 5: Vector EmbeddingsStep 6: Similarity SearchTakeaways
5. Feature Engineering and Management for Multimodal Data
Feature Engineering Techniques for Multimodal DataA Modality-Aware Feature-Engineering PipelinePreprocessing as Feature EngineeringExtracting Features Across ModalitiesFeature-Engineering Core TechniquesFeature Selection and Fusion in Multimodal ContextsWhy Fusion Is Not a JoinFusion ArchitecturesCross-Cutting Considerations in FusionImplementing and Managing Cloud-Based Multimodal Feature StoresWhy Feature Stores Are NeededFour Patterns for Feature Store ArchitectureFeature Store as a ServiceFeature Store Implementation: BayGo’s Two ApproachesPlaybook: Building an End-to-End Multimodal Feature-Engineering PipelineGetting Familiar with the DatasetStep 1: Verify Data StructureStep 2: Understand the Main Pipeline CodeStep 3: Run the Multimodal Feature PipelineStep 4: Examine the Generated OutputsStep 5: Understand the Generated OutputsTakeaways
6. Data Storage and Retrieval for Multimodal AI
Storage Solutions for Multimodal ML WorkloadsInfrastructure LayerData Intelligence LayerAI Service LayerApplication LayerDatabases for Cross-Modal ApplicationsVector DatabasesGraph Databases and Knowledge GraphsHybrid Storage and RetrievalCaching Strategies and Inference on the EdgeCaching ZonesStatic CachingDynamic CachingA Comparison of Static and Dynamic CachingCaching in Action: A Field ScenarioCaching Design DecisionsPlaybook: Engineering Graph-Augmented Retrieval and Edge-Aware Caching for Multimodal AISample Data ExplorationOverviewStep 1: Environment Setup and Data PreparationStep 2: Embedding GenerationStep 3: Graph ConstructionStep 4: Hybrid Retrieval TestStep 5: Edge-Aware Cache TestStep 6: Full System TestStep 7: Streamlit UIPlaybook ConclusionTakeaways
7. Streaming and Adaptive Pipelines for Multimodal AI
Designing Streaming Pipelines for Multimodal InferenceEvent-Time AlignmentFrom Lambda to Event Driven: An Architectural ShiftThe Reference PipelineModality-Specific PreprocessingWindowed FusionInference RoutingBack-Pressure ControlFanning Out, Sinks, and System OutputsTuning the Pipeline for ProductionAdaptive Multimodal AI SystemsWhy Does Adaptation Even Matter?Drift Types, Detection, and Engineering ResponseEdge Computing for Multimodal AI Data ProcessingPartitioning Logic: What Runs Where and WhyEdge-Centric Pipeline Architecture WalkthroughFederated LearningCI/CD and Rollback for Edge-Deployed ModelsPlaybook: Building a Real-Time Multimodal Inference RouterStep 1: Data Exploration, Preprocessing, and NormalizationStep 2: Stream Fusion and Inference RoutingStep 3: Inference Serialization and DispatchStep 4: DAG Orchestration and Monitoring
8. Applying MLOps for Multimodal AI Applications
An End-to-End MLOps LifecycleContinuous Integration and Continuous Delivery for Multimodal AIData Versioning and Experiment Tracking for Multimodal AIAutomated Retraining Pipelines for Multimodal AIContinuous Data Integration for Multimodal SystemsStreaming and Batch IngestionThe Feature Store InterfaceTesting and Validation for Multimodal ModelsValidation ArchitectureObservability for Multimodal SystemsCollaborative Dynamics and Best Practices for Multimodal AI TeamsFrom Data Foundations to Feedback: Eight Lessons in Operating Multimodal ML SystemsPlaybook: Implementing an End-to-End MLOps PipelineData ExplorationStep 1: Data IngestionStep 2: Feature ExtractionStep 3: Model TrainingStep 4: Model Evaluation and PredictionStep 5: Model Serving and DeploymentStep 6: Setting Up Model MonitoringPlaybook ConclusionTakeaways
9. Scaling Cloud-Native Multimodal AI Data Pipelines
Distributed Data Processing for Multimodal DataStream-Processing Engines: Apache Beam on DataflowBatch Feature-Engineering Frameworks: Spark on KubernetesDistributed AI Runtimes: Ray + KubeRay + Inference EnginesHigh-Throughput Data Loaders: The Training Input LayerFault Tolerance and RecoveryDistributed DAGs and Bottlenecks in Multimodal PipelinesUnifying the Stack: A Composable Runtime for Multimodal AIAuto-Scaling Strategies for Multimodal WorkloadsModality-Specific Scaling ProfilesCross-Modal Dependency OrchestrationCost Optimization for Cloud-Based Multimodal AI InfrastructurePreemptible Training and Opportunistic ComputeBatching and Inference TieringDynamic Model Parking with Checkpoint StreamingStorage-Aware Cost ControlToken-Efficient InferencePerformance Tuning and Optimization TechniquesMemory-Tiered Sharding in GPU WorkloadsTuning the Inference Engine for LatencyBatch Executor Optimization in SparkData Loader Tuning for Training ThroughputPlaybook: Deploying a Serverless Multimodal Inference Service with On-Demand GPU ScalingUnderstanding the Sample DataStep 1: Prerequisites and SetupStep 2: Deploying the PipelineStep 3: Running a Clinical Workload SimulationStep 4: Running the Predictive Autoscaler DemoTakeaways

10. Model Deployment and Serving for Multimodal AI
Deploying Multimodal Models for Real-Time InferenceThe Inference TriangleThe Inference Framework: Modal ContractsA Production Example at Eighteen LabsScene Text Recognition in Multimodal PipelinesHandling Multi-Stage Inference PipelinesEngineering ChallengesEighteen Labs: Data Center OperationsMonitoring and Managing Deployed Multimodal ModelsMonitoring ArchitectureDetecting DriftFeedback LoopsEvaluation Framework and RetrainingCausal Attribution and Adversarial ResilienceScalable Multimodal ServingConcurrency and SynchronizationBatching Across ModalitiesModel Optimization for DeploymentHardware and Accelerator SchedulingEdge-Cloud Inference TopologyTiered RoutingCost and SustainabilityContinuous Deployment and Lifecycle Orchestration for Multimodal ModelsVersion Control for Composite ModelsCanary DeploymentCI/CD OrchestrationRollbackLarge Behavior ModelsDual-Path ArchitectureSequence-Aware CI/CDGovernancePlaybook: Building and Managing a Multimodal Inference ServiceUnderstanding the Data: RLBenchStep 1: Validating the Model ConfigurationStep 2: Implementing the ModelStep 3: Web Service ImplementationStep 4: Canary Deployment and ContainerizationTakeaways
11. Governance, Security, and Responsible Practices for Multimodal AI
Foundations of Governance in Multimodal AILineage and Provenance TrackingMetadata CatalogsCompliance Across ModalitiesData SovereigntyGovernance ArchitectureExample: Warehouse Governance in PracticeSecuring Multimodal AI PipelinesProtecting Data Across the PipelinePrivacy-Preserving Machine LearningFederated and Distributed LearningAdversarial Threats to Multimodal LLMsExample: Data Center Security in PracticeResponsible and Transparent AIBias Detection Across ModalitiesExplainability for Multimodal SystemsAdversarial RobustnessProduction ObservabilityExample: Responsible AI in PracticePlaybook: Operationalizing Governance, Security, and Responsible AI for Multimodal PipelinesWhat You’ll BuildThe DatasetsStep 1: Design Pipeline ArchitectureStep 2: Data Ingestion ModuleStep 3: Security ModuleStep 4: Inference ModuleStep 5: Responsible AI ModuleStep 6: Running the End-to-End PipelineCost GovernancePlaybook ConclusionTakeaways
12. Putting It All Together: Building End-to-End Multimodal AI Pipelines
Designing and Architecting Complete Multimodal AI SystemsStreamKit in Practice: High-Volume Incident SurgeIntelligent Handling of a Product IncidentKey ConsiderationsIntegration Patterns for Multimodal Data PipelinesPattern 1: On-the-Wire ContractsPattern 2: Transport SelectionPattern 3: Back-Pressure and Fault HandlingPattern 4: Scaling and ElasticityPattern 5: Multimodal RAGWorkflow Orchestration for Multimodal AIOrchestration ResponsibilitiesImplicit Versus Explicit OrchestrationEdge Orchestration in ActionSchema Validation at the EdgeAdaptive Resource ControlThe Edge-Cloud HandoffMonitoring and Troubleshooting End-to-End Multimodal PipelinesSignal Granularity: Instrument Each LaneCross-Modal Correlation: Detect Cascading FailuresActionable Diagnostics: From Alert to FixStreamKit’s Three Observability LayersAutomated Runbooks for Common FailuresBenchmarking Production ReadinessEvolving and Scaling Production Multimodal PipelinesShadow Mode and Safe PromotionScaling Across Sites: Session-Aware FusionDual-Support Deprecation: The Sentiment Analyzer UpgradeAsync Upgrade Bridge: Bridging Embedding IncompatibilitiesResilient Operation Under Modality FailureObservability Binds the Whole PictureEvaluation and Feedback LoopsPlaybook: Building a Multimodal Root Cause Analysis PipelineData ExplorationWhat We Will BuildPipeline ArchitectureRunning the PlaybookWhat to Look For in the OutputExtending the PlaybookTakeaways
Appendix. References and Datasets
Chapter 1. The Multimodal AI Data Engineering LandscapeChapter 2. Data Architectures for Multimodal AI SystemsChapter 3. Platforms and Patterns for Multimodal AI Data EngineeringChapter 4. Ingestion and Transformation for Multimodal AI SystemsChapter 5. Feature Engineering and Management for Multimodal DataDatasetChapter 6. Data Storage and Retrieval for Multimodal AIChapter 7. Streaming and Adaptive Pipelines for Multimodal AIChapter 8. Applying MLOps for Multimodal AI ApplicationsChapter 9. Scaling Cloud-Native Multimodal AI Data PipelinesChapter 10. Model Deployment and Serving for Multimodal AIDatasetChapter 11. Governance, Security, and Responsible Practices for Multimodal AIDatasetsChapter 12. Putting It All Together: Building End-to-End Multimodal AI PipelinesDataset
Index
About the Author

Content preview from Data Engineering for Multimodal AI

Appendix. References and Datasets

Chapter 1. The Multimodal AI Data Engineering Landscape

Anthropic. “Introducing the Model Context Protocol.” November 25, 2024. https://www.anthropic.com/news/model-context-protocol.

Bommasani, R., D. A. Hudson, E. Adeli, et al. On the Opportunities and Risks of Foundation Models. Center for Research on Foundation Models (CRFM) at the Stanford Institute for Human-Centered Artificial Intelligence (HAI), 2021. https://crfm.stanford.edu/assets/report.pdf.

Christophides, V., V. Efthymiou, T. Palpanas, G. Papadakis, and K. Stefanidis. “An Overview of End-to-End Entity Resolution for Big Data.” ACM Computing Surveys 53, no. 6 (2021). https://dl.acm.org/doi/10.1145/3418896.

DeepSeek-AI; D. Guo, D. Yang, H. Zhang, et al. “DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning.” arXiv (2025). https://arxiv.org/abs/2501.12948.

Hogan, A., E. Blomqvist, M. Cochez, et al. “Knowledge Graphs.” ACM Computing Surveys 54, no. 4 (2021). https://arxiv.org/abs/2003.02320.

Liang, P. P., A. Zadeh, and L.-P. Morency. “Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions.” arXiv (2022). https://arxiv.org/abs/2209.03430.

Sculley, D., G. Holt, D. Golovin, et al. “Hidden Technical Debt in Machine Learning Systems.” In Proceedings of the 29th International Conference on Neural Information Processing Systems: Volume 2 (NIPS 2015). https://proceedings.neurips.cc/paper_files/paper/2015/file/86df7dcfd896fcaf2674f757a2463eba-Paper.pdf ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Publisher Resources

ISBN: 9781098190774Errata Page

Cloud Computing