Chapter 11 Data Engineering
Data engineering moves data from fragmented sources into coherent pipelines; structures it with ontologies and semantic layers; governs its quality and lineage; and delivers it to agents, models, dashboards, and simulations with guarantees of timeliness, trust, and integrity. Data engineering for digital twins must support continuous synchronization between the physical world and its computational mirror.
The AI Readiness of Data
AI agents and models are reliable only when the data they consume is engineered to reflect the real world accurately. Data engineering is then upstream AI engineering: a digital twin’s intelligence is bounded by its semantic and operational alignment with reality.
Machine learning (ML) workflows depend on curated, structured, and governed data, including:
- Domain and industry knowledge
- Deduplication and identity resolution
- Model- and rule-based selection
- Ontology alignment
- Type enforcement
- Unit normalization
- Rigorous lineage and provenance.
Rigorous data engineering must serve AI-powered digital twins. AI reasons over engineered data and correctly modeled meaning. Note that:
- We are not claiming data must be perfect—only engineered.
- We are grounding AI reliability in semantics, not scale.
- We are implicitly rejecting “ LLMs will figure it out” thinking.
- We are aligning with MLOps and data governance best practices.
Databases, AI Model Databases, and AI Agent Databases
Databases are the living memory of AI-powered digital ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access