May 2025
Intermediate to advanced
450 pages
4h 56m
Chinese
Content preview from 数据工程基础知识Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.
索引
A
- 抽象，数据工程角色的连续性，从 A 到 B
- 访问策略、日志、监控和警报
- 问责制、数据问责制
- 准确性、数据质量
- ACID（原子性、一致性、隔离性和持久性）事务，融合、下一代数据湖和数据平台，ACID
- 主动安全,主动安全,内部安全研究
- 临时分析、业务分析
- 敏捷架构,原则 5：始终在架构
- 敏捷性、"好 "数据架构
- 人工智能研究人员、机器学习工程师和人工智能研究人员
- Airflow DAG，示例：Spark 中的数据转换
- Airflow 项目，协调示例：气流
- 警报、日志、监控和警报
- 亚马逊弹性块存储（EBS），云虚拟化块存储
- 亚马逊 EMR，示例：使用 S3 和 HDFS 的 AWS EMR
- 模拟数据，数据来源：数据是如何产生的？
- 分析
- Apache Arrow、Apache Arrow 或内存序列化
- Apache Beam 框架、数据流模型和统一批处理与流处理
- Apache Druid，示例：Apache 德鲁伊
- Apache Spark，示例：Apache Spark
- 应用程序接口（见应用程序接口）
- 应用架构，定义的企业架构（另见数据架构；单体架构；技术架构）
- 应用数据库、应用数据库（OLTP 系统）--OLTP和分析
- 应用编程接口 (API)、API、API-RPC和 gRPC、API
- 建筑层级,建筑层级
- 档案存储、高速缓存
- 磁盘驱动器
- 非对称优化,非对称优化
- 异步数据摄取，同步与异步摄取
- 原子事务，应用数据库（OLTP 系统）-原子事务
- 原子性、一致性、隔离性和耐用性（ACID）事务，融合、下一代数据湖和数据平台，ACID
- 自动生成的元数据,元数据
- 自动生命周期策略、存储层考虑因素
- 自动数据生命周期管理、成本
- 自动化,自动化
- 可用性、原则 2：故障预案、分布式系统、可扩展性和故障设计、数据耐用性和可用性
- 可用区,Network+ 和 CPU,可用区
- Avro,Avro
- AWS 良好架构框架，良好数据架构原则
B
- B 树，了解数据库技术的主要考虑因素
- 备份，始终备份您的数据
- 基线架构，原则 5：始终在架构
- bash、技术责任
- 基本可用、软状态、最终一致性 (BASE)、最终一致性与强一致性
- 批处理数据摄取、批处理与流式处理、数据流模型和统一批处理与流式处理、批处理摄取注意事项--数据迁移
- 批量数据转换
- 广播加入，广播加入
- 业务逻辑和派生数据,业务逻辑和派生数据
- 数据整理,数据整理
- 分布式连接,分布式连接
- ETL、ELT 和数据管道，ETL、ELT 和数据管道
- 转型阶段的主要考虑因素
- 地图生成, MapReduce
- 内存缓存，MapReduce 之后
- 模式更新,模式更新
- 洗牌散列连接,洗牌散列连接
- in Spark,示例：Spark 中的数据转换
- SQL 和基于代码的转换工具，SQL 和基于代码的转换工具
- 与流式转换相比，批量转换
- 更新模式,更新模式
- 批处理日志，日志延迟：批处理或实时
- 批量大小、插入、更新和批量大小
- 面向批处理的变更数据采集，面向批处理的 CDC
- 基准、优化、性能和基准战争
- 贝索斯 API 任务，原则 6：构建松耦合系统
- 大数据工程师，2000 和 2010 年代：大数据工程
- 大数据时代，21 世纪初：当代数据工程的诞生》、《大数据......20 世纪 90 年代》、《复杂性的衰落与易用数据工具的崛起》。
- 块存储，块存储-本地实例卷
- 区块链技术，去中心化：区块链与边缘
- 块（HDFS），Hadoop 分布式文件系统
- 有界数据、有界 数据与无界数据
- 博伊斯-科德系统、归一化
- 桥牌桌、卫星
- 广播加入，广播加入
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access
More than 5,000 organizations count on O’Reilly
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.Julian F.
I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.Addison B.
I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.Amir M.
I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.