November 2025
Intermediate to advanced
1060 pages
14h 20m
Chinese
Content preview from 人工智能系统性能工程 (Chinese Edition)
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.
第五章 GPU 存储 I/O 优化
本作品已使用人工智能进行翻译。欢迎您提供反馈和意见：translation-feedback@oreilly.com
对于人工智能工作负载而言，向GPU输送数据与计算本身同样重要。设想一个场景：一个拥有100万亿参数的模型正在数千台GPU上进行训练。该模型可能需要处理数十亿训练样本，包括文本片段、图像、音频、视频等。
这意味着海量数据必须从存储设备读取并以最快速度输送至GPU。若存储管道效率低下，GPU将陷入饥饿状态而闲置，导致即使采用我们讨论过的复杂通信优化方案，设备利用率仍会偏低。
本章将探讨存储与输入管道的优化方案，重点演示如何高效读取磁盘或远程存储数据、如何预处理数据，以及如何实现I/O操作与GPU计算的重叠执行。
高速存储与数据局部性
大型模型训练任务通常需要读取海量数据集（ ）。大型语言模型常需处理数十亿乃至数万亿训练样本，其文本数据规模达数TB，视觉模型图像数据则达数PB。
在超大规模场景下，存储系统必须持续提供海量吞吐量，以支撑数千乃至数百万个GPU持续运行数月之久。通过在机架内集中部署NVMe SSD，或采用基于机架本地交换拓扑的NVMe over Fabrics（NVMe-oF）技术，可最大限度减少网络跳数并提升性能稳定性。
若数据存储于NFS服务器或云对象存储（如Amazon S3）等网络附加存储中，则需确保所有计算节点的聚合读取带宽充足。假设基于模型和批量大小，每块GPU需200 MB/s训练数据才能保持满负荷运行。若总计8块GPU，则需约1.6 GB/s总带宽。而Blackwell和Rubin等现代高端GPU维持饱和运行所需带宽更高。
以NVIDIA Grace Blackwell GB200/GB300 NVL72机架为例：72块Blackwell GPU通过单一NVLink域互联。若每块GPU需200 MB/s训练数据维持满负荷运行，则需14-20 GB/s的总存储吞吐量才能保障全部GPU高效运作。此类超大规模工作负载要求存储方案具备相应扩展能力。
若您的工作负载涉及更重的媒体流或多模态样本，请根据测得的每样本字节数和每秒样本数进行校准。此类场景下，聚合需求可能大幅提升。
解决方案之一是采用更快的本地存储，例如同机架内的NVMe SSD，或采用NVMe-oF网络拓扑。另一方案是使用Lustre或通用并行文件系统（GPFS）等并行文件系统，将数据缓存至本地SSD。假设存储系统能跟上负载，关键是要配置多个数据加载线程以保持管道饱和。注意Python全局解释器锁（GIL）！
尽可能将数据物理部署在计算节点附近。"附近"可指同一物理节点（如本地NVMe SSD驱动器），或至少通过高速互连（如NVMe over Fabric或先进存储加速器）部署在同一机架内。
对于分布式多节点模型训练， 常用策略是将数据集按节点分片，使每个节点主要从本地磁盘读取子集数据。例如，若拥有100TB数据和10个节点，可预先将10TB数据分配至每个节点本地存储。此时每个节点的数据加载器仅读取本地10TB数据，避免因冗余读取导致网络饱和——尤其当数据集规模难以完全容纳于内存时。
和PyTorch的DistributedSampler 等框架会协调工作进程，确保每个进程在每个 epoch 获得独立的数据片段。这与将数据分片到多个集群节点的目标高度契合。
顺序读取与随机读取模式
GPU在数据处理方面极具速度优势， 但为提升效率，更倾向于读取大块连续数据。同样地，存储系统对大容量顺序读取的吞吐量远高于小规模随机读取。因此在准备数据集或规划存储布局时，应尽可能安排顺序访问模式。 ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access
More than 5,000 organizations count on O’Reilly
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.Julian F.
I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.Addison B.
I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.Amir M.
I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.