September 2022
Intermediate to advanced
549 pages
11h 27m
Chinese
Content preview from 基于AWS的数据科学实践
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.
36
｜
第
1
章
1.9.4
为大规模模型训练优化的存储选项
我们已经了解了在
Amazon S3
上构建数据湖的好处。如果我们在分布式模型训练中
需要更快的存储，我们可以使用
Amazon FSx for Lustre
。
Amazon FSx for Lustre
将开源的
Lustre
文件系统提供为一个完全托管的服务。
Lustre
是一个高性能文件系统，提供亚毫秒级别的延时和每秒最高几百吉字节（
GB
）
的吞吐量以及每秒百万级的
IO
操作（
IOPS
）。
我们可以将
FSx for Lustre
文件系统与
Amazon S3
存储桶链接。这让我们可以通过
FSx
文件系统访问并处理
Amazon S3
上的数据。借助
FSx for Lustre
，我们可以配置
模型训练的计算资源从高性能共享存储中访问相同的数据。
Amazon Elastic File System
（
Amazon EFS
）是另一个可以为最多几千台
Amazon
EC2
实例提供文件系统接口的文件存储服务。这个文件系统接口提供标准操作系统
文件
I/O API
并启用了文件系统访问语义，例如强一致性和文件锁。
1.10
使用标签、预算和告警减少开销
纵观全书，我们在使用
Amazon AI
和机器学习栈构建数据科学项目时，为如何降低
开销提供了建议。总体上讲，我们应该为资源打上事业部（
business unit
）名称、
应用名、环境以及用户的标签。我们应使用标签来查看开销的情况。在
AWS
内置
的成本分配标签之外，我们可以提供适用我们领域的自定义分配标签。
AWS
预算
（
Budgets ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access
More than 5,000 organizations count on O’Reilly
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.Julian F.
I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.Addison B.
I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.Amir M.
I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.