book

Programming Massively Parallel Processors, 3rd Edition

by David B. Kirk, Wen-mei W. Hwu

November 2016

Intermediate to advanced

576 pages

18h 22m

English

Morgan Kaufmann

Read now

Unlock full access

Target AudienceHow to Use the BookIllinois–NVIDIA GPU Teaching KitOnline Supplements
Abstract1.1 Heterogeneous Parallel Computing1.2 Architecture of a Modern GPU1.3 Why More Speed or Parallelism?1.4 Speeding Up Real Applications1.5 Challenges in Parallel Programming1.6 Parallel Programming Languages and Models1.7 Overarching Goals1.8 Organization of the BookReferences
Abstract2.1 Data Parallelism2.2 CUDA C Program Structure2.3 A Vector Addition Kernel2.4 Device Global Memory and Data Transfer2.5 Kernel Functions and Threading2.6 Kernel Launch2.7 SummaryReferences
Abstract3.1 CUDA Thread Organization3.2 Mapping Threads to Multidimensional Data3.3 Image Blur: A More Complex Kernel3.4 Synchronization and Transparent Scalability3.5 Resource Assignment3.6 Querying Device Properties3.7 Thread Scheduling and Latency Tolerance3.8 Summary

Abstract4.1 Importance of Memory Access Efficiency4.2 Matrix Multiplication4.3 CUDA Memory Types4.4 Tiling for Reduced Memory Traffic4.5 A Tiled Matrix Multiplication Kernel4.6 Boundary Checks4.7 Memory as a Limiting Factor to Parallelism4.8 Summary
Abstract5.1 Global Memory Bandwidth5.2 More on Memory Parallelism5.3 Warps and SIMD Hardware5.4 Dynamic Partitioning of Resources5.5 Thread Granularity5.6 SummaryReferences
Abstract6.1 Floating-Point Data Representation6.2 Representable Numbers6.3 Special Bit Patterns and Precision in IEEE Format6.4 Arithmetic Accuracy and Rounding6.5 Algorithm Considerations6.6 Linear Solvers and Numerical Stability6.7 SummaryReferences
Abstract7.1 Background7.2 1D Parallel Convolution—A Basic Algorithm7.3 Constant Memory and Caching7.4 Tiled 1D Convolution with Halo Cells7.5 A Simpler Tiled 1D Convolution—General Caching7.6 Tiled 2D Convolution With Halo Cells7.7 Summary7.8 Exercises
Abstract8.1 Background8.2 A Simple Parallel Scan8.3 Speed and Work Efficiency8.4 A More Work-Efficient Parallel Scan8.5 An Even More Work-Efficient Parallel Scan8.6 Hierarchical Parallel Scan for Arbitrary-Length Inputs8.7 Single-Pass Scan for Memory Access Efficiency8.8 Summary8.9 ExercisesReferences
Abstract9.1 Background9.2 Use of Atomic Operations9.3 Block versus Interleaved Partitioning9.4 Latency versus Throughput of Atomic Operations9.5 Atomic Operation in Cache Memory9.6 Privatization9.7 Aggregation9.8 SummaryReference
Abstract10.1 Background10.2 Parallel SpMV Using CSR10.3 Padding and Transposition10.4 Using a Hybrid Approach to Regulate Padding10.5 Sorting and Partitioning for Regularization10.6 SummaryReferences
Abstract11.1 Background11.2 A Sequential Merge Algorithm11.3 A Parallelization Approach11.4 Co-Rank Function Implementation11.5 A Basic Parallel Merge Kernel11.6 A Tiled Merge Kernel11.7 A Circular-Buffer Merge Kernel11.8 SummaryReference
Abstract12.1 Background12.2 Breadth-First Search12.3 A Sequential BFS Function12.4 A Parallel BFS Function12.5 Optimizations12.6 SummaryReferences
Abstract13.1 Background13.2 Dynamic Parallelism Overview13.3 A Simple Example13.4 Memory Data Visibility13.5 Configurations and Memory Management13.6 Synchronization, Streams, and Events13.7 A More Complex Example13.8 A Recursive Example13.9 SummaryReferencesA13.1 Code Appendix
Abstract14.1 Background14.2 Iterative Reconstruction14.3 Computing FHD14.4 Final EvaluationReferences
Abstract15.1 Background15.2 A Simple Kernel Implementation15.3 Thread Granularity Adjustment15.4 Memory Coalescing15.5 SummaryReferences
Abstract16.1 Background16.2 Convolutional Neural Networks16.3 Convolutional Layer: A Basic CUDA Implementation of Forward Propagation16.4 Reduction of Convolutional Layer to Matrix Multiplication16.5 cuDNN LibraryReferences
Abstract17.1 Goals of Parallel Computing17.2 Problem Decomposition17.3 Algorithm Selection17.4 Computational Thinking17.5 Single Program, Multiple Data, Shared Memory and Locality17.6 Strategies for Computational Thinking17.7 A Hypothetical Example: Sodium Map of the Brain17.8 SummaryReferences
Abstract18.1 Background18.2 A Running Example18.3 Message Passing Interface Basics18.4 Message Passing Interface Point-to-Point Communication18.5 Overlapping Computation and Communication18.6 Message Passing Interface Collective Communication18.7 CUDA-Aware Message Passing Interface18.8 SummaryReference
Abstract19.1 The OpenACC Execution Model19.2 OpenACC Directive Format19.3 OpenACC by Example19.4 Comparing OpenACC and CUDA19.5 Interoperability with CUDA and Libraries19.6 The Future of OpenACC
Abstract20.1 Model of Host/Device Interaction20.2 Kernel Execution Control20.3 Memory Bandwidth and Compute Throughput20.4 Programming Environment20.5 Future OutlookReferences
Abstract21.1 Goals Revisited21.2 Future Outlook
A.1 BackgroundA.2 Data Parallelism ModelA.3 Device ArchitectureA.4 Kernel FunctionsA.5 Device Management and Kernel LaunchA.6 Electrostatic Potential Map in OpenCLA.7 Summary
B.1 BackgroundB.2 MotivationB.3 Basic Thrust FeaturesB.4 Generic ProgrammingB.5 Benefits of AbstractionB.6 Best Practices
C.1 CUDA Fortran and CUDA C DifferencesC.2 A First CUDA Fortran ProgramC.3 Multidimensional Array in CUDA FortranC.4 Overloading Host/Device Routines with Generic InterfacesC.5 Calling CUDA C via ISO_C_BindingC.6 Kernel Loop Directives and Reduction OperationsC.7 Dynamic Shared MemoryC.8 Asynchronous Data TransfersC.9 Compilation and ProfilingC.10 Calling Thrust from CUDA Fortran
D.1 Core C++ AMP FeaturesD.2 Details of the C++ AMP Execution ModelD.3 Managing AcceleratorsD.4 Tiled ExecutionD.5 C++ AMP Graphics FeaturesD.6 SummaryReference

Overview

Programming Massively Parallel Processors: A Hands-on Approach, Third Edition shows both student and professional alike the basic concepts of parallel programming and GPU architecture, exploring, in detail, various techniques for constructing parallel programs.

Case studies demonstrate the development process, detailing computational thinking and ending with effective and efficient parallel programs. Topics of performance, floating-point format, parallel patterns, and dynamic parallelism are covered in-depth.

For this new edition, the authors have updated their coverage of CUDA, including coverage of newer libraries, such as CuDNN, moved content that has become less important to appendices, added two new chapters on parallel patterns, and updated case studies to reflect current industry practices.

Teaches computational thinking and problem-solving techniques that facilitate high-performance parallel computing
Utilizes CUDA version 7.5, NVIDIA's software development tool created specifically for massively parallel environments
Contains new and updated case studies
Includes coverage of newer libraries, such as CuDNN for Deep Learning