September 2022
Intermediate to advanced
549 pages
11h 27m
Chinese
Content preview from 基于AWS的数据科学实践
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.
探索数据集
｜
171
5.5
使用
Amazon SageMaker
和
Apache Spark
检测数据质量问题
数据从不是完美的，尤其是拥有
20
年以来超过
1.5
亿条数据的数据集。另外，数据
质量可能由于应用功能的增减，随时间推移而降低。模式会演变，代码会变旧，且
查询会变慢。
由于数据质量并不总是上游应用团队优先考虑的事情，下游的数据工程和数据科学
团队需要处理坏数据和丢失数据。我们要确保为下游消费方提供高质量的数据，包
括商业智能、机器学习工程和数据科学团队。
图
5-13
展示了应用如何为工程师、科学家和分析师生成可消费的数据，以及不同的
团队可能会使用哪些工具来访问这些数据。
应用
机器学习工程师/
数据科学家
商业智能/
数据分析师
数据湖
查询引擎
数据仓库
查看报表
将数据写入
Kinesis
将数据写入
S3
查询S3 训练模型
查询数据
仓库
图 5-13：工程师、科学家和分析师使用不同的工具和服务来访问数据
数据质量可能阻碍正在进行的数据处理流水线。如果不尽早发现这些问题，它们可
以导致误导性的报告（比如误算的双倍收入），有偏见的
AI/ML
模型（偏向或偏离
某个性别或种族），以及其他意料之外的数据产品。
172
｜
第
5
章
为了尽早捕捉这些数据问题，我们使用
AWS
提供的两个开源库，
Deequ
（
https://
oreil.ly/CV ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access
More than 5,000 organizations count on O’Reilly
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.Julian F.
I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.Addison B.
I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.Amir M.
I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.