book

GPU-Accelerated Deep Learning : Essential GPU Ideas, Deep Learning Frameworks, and Optimization Approaches

Name: GPU-Accelerated Deep Learning : Essential GPU Ideas, Deep Learning Frameworks, and Optimization Approaches
ISBN: 9798868820830

by Ramchandra S Mangrulkar, Pallavi Vijay Chavan

January 2026

Intermediate to advanced

158 pages

4h 21m

English

Apress

Read now

Unlock full access

GPU-Accelerated Deep Learning
Introduction
Acknowledgement
Contents
About the Authors
About the Technical Reviewer
1. Introduction to Deep Learning and GPU Acceleration
1.1 Introduction1.1.1 Historical Background1.1.2 Applications of Deep Learning1.1.3 Deep Learning Architectures1.1.3.1 Deep Neural Network1.1.4 Linear Regression as a Single-Layer Neural Network1.2 Convolutional Neural Networks (CNNs)1.2.1 Evolving from Dense Layers to Convolutional Networks1.2.2 Introducing Spatial Structure in MLPs1.2.3 Translation Invariance1.2.4 Locality1.2.5 Convolutions1.3 Recurrent Neural Networks (RNNs)1.4 Mathematical Representation of RNNs1.5 Unrolling an RNN1.6 Challenges in RNNs1.6.1 Vanishing and Exploding Gradients1.6.2 Long Short-Term Memory (LSTM)1.7 Applications of RNNs1.8 GPU Acceleration in Deep Learning1.8.1 Introduction to GPU Acceleration1.8.2 Types of GPU Acceleration1.8.2.1 General-Purpose GPU (GPGPU)1.8.2.2 General-Purpose GPU (GPGPU)1.8.2.3 Dedicated GPUs1.8.2.4 Integrated GPUs1.9 Training a Deep Learning Model1.9.1 Image Classification Problem1.9.2 Exploring the MNIST Data1.9.3 Data Engineering and Dataset Splits1.9.4 Tensors1.9.5 Transforms1.9.5.1 DataLoaders1.9.6 Creating the Model1.9.6.1 The Hidden Layer1.9.6.2 The Output Layer1.9.7 Compiling the Model1.9.8 Calculating Accuracy1.9.9 The Train Function1.10 Summary
2. GPU Architectures, Optimizations, and Deployment
2.1 Deep Learning Frameworks2.1.1 What Is a Deep Learning Framework?2.1.2 Why Use a Deep Learning Framework?2.1.3 Popular Frameworks2.1.3.1 TensorFlow2.1.3.2 PyTorch2.1.3.3 JAX2.1.4 Comparison of Popular Frameworks2.2 Setting GPU Support2.2.1 NVIDIA GPU Driver Requirements2.2.2 Installing CUDA and cuDNN2.2.3 Verifying GPU Availability2.2.4 Managing GPU Memory Usage and Visibility2.3 GPU Acceleration in TensorFlow2.3.1 GPU-Compatible Tensors and Operations2.3.2 Automatic vs. Manual Device Placement2.3.3 Enabling Mixed-Precision Training2.3.4 Using MirroredStrategy for Multi-GPU Training2.4 Multi-GPU Training Strategies2.4.1 Data Parallelism2.4.2 Model Parallelism2.4.3 Hybrid Parallelism2.4.4 Comparison of Multi-GPU Strategies2.5 GPU Memory Optimization Techniques2.5.1 Gradient Checkpointing (Recomputation)2.5.2 Mixed-Precision Training2.5.3 Layer-Wise Memory Reuse2.5.4 Memory Growth and Garbage Collection2.5.5 Comparison of GPU Memory Optimization Techniques2.5.6 TensorRT Integration for Inference Acceleration2.6 Deployment Considerations2.6.1 Edge Devices2.6.2 Containers and Virtual Environments2.7 Benchmarking and Performance Tuning2.7.1 Measuring Speedups and Memory Utilization2.7.2 Common Bottlenecks in GPU Workflows2.7.3 Comparing Training Times Across Frameworks2.8 Summary
3. Advanced Deep Learning Architectures for GPU Workloads
3.1 Transfer Learning with GPU Acceleration3.1.1 Motivation and Need3.1.2 Mathematical Model for Transfer Learning3.1.3 Relevance with GPUs3.2 GPU Benefits in Fine-Tuning Pretrained Models3.2.1 Fine-Tuning vs. Training3.2.2 Fine-Tuning Techniques3.2.2.1 Freezing Bottom Layer3.2.2.2 Parameter-Efficient Fine-Tuning (PEFT)3.2.2.3 Prompt Tuning3.2.3 Role of GPU in Fine-Tuning3.2.4 Forward Pass3.2.5 Loss Function3.2.6 Backpropagation and Gradient Update3.2.7 An Example3.2.8 Forward Pass3.2.9 Loss Calculation (Mean Squared Error)3.2.10 Backpropagation (Gradient Calculation)3.2.11 Parameter Update (Using SGD, η=0.1)3.3 Fine-Tuning in Computer Vision3.4 GPU Acceleration in Deep Network Tuning3.5 GPU-Accelerated Vector Search Indexing Methods3.5.1 IVF-Flat3.5.2 IVF-PQ3.5.3 CAGRA3.5.4 GPU Architecture for Deep Network Tuning3.6 GPU Benefits in Fine-Tuning3.7 Summary
4. Advanced GPU Programming for Deep Learning
4.1 CUDA Programming Model4.1.1 What Is CUDA?4.1.2 CUDA Programming Model4.1.3 Heterogeneous Computing4.1.3.1 Host4.1.3.2 Device4.1.3.3 Interaction Between Host and Device4.1.4 CUDA Extensions to C/C++4.1.4.1 Function Launch and Kernel Execution4.1.4.2 Calling Functions on the GPU4.1.4.3 Memory Management4.1.4.4 Memory Declaration Qualifiers4.1.4.5 Special Instructions4.1.4.6 Thread and Block Identification Keywords4.1.4.7 Function Declarations in CUDA4.1.4.8 CUDA Function Qualifiers Overview4.1.4.9 Function Qualifier Descriptions4.1.4.10 Example: CUDA Function Declarations in Action4.2 Variable Type Qualifiers in CUDA4.2.1 SIMT Architecture4.2.1.1 Mapping Blocks and Threads4.2.1.2 Thread Life Cycle in a GPU4.2.2 Device Memory Management4.2.3 Memory Hierarchy4.2.4 GPU Memory Management4.2.4.1 Types of GPU Memory (Global, Shared, Constant, Texture)4.2.4.2 Memory Growth, Garbage Collection, and Layer-wise Memory Reuse4.2.4.3 Efficient Memory Access and Coalescing Strategies4.2.4.4 Memory Growth, Garbage Collection, and Layer-wise Memory Reuse4.3 Optimization Techniques on GPUs4.3.1 Memory Access Optimization4.3.1.1 Memory Coalescing4.3.2 Thread-Level Optimization4.3.2.1 Occupancy Optimization4.3.2.2 Avoiding Thread Divergence4.3.3 Instruction-Level Optimization4.3.4 Asynchronous Execution and Overlapping4.3.5 Profiling and Analysis Tools4.3.6 Essential GPU Optimization Guidelines4.4 GPU Optimization Techniques4.4.1 Tiled Matrix Multiplication and Parallel Algorithms4.4.2 Gradient Checkpointing and Recomputation4.4.3 Mixed-Precision Training and Tensor Core Usage4.4.4 Implementation of Matrix Multiplication Kernel4.5 Summary

5. Distributed and Multi-GPU Training Strategies
5.1 Distributed Deep Learning5.1.1 Limitations of Single GPU5.2 Single-Node Multi-GPU Training5.2.1 NCCL and CUDA-Aware Collectives5.2.2 GPU Topologies and PCIe/NVLink Considerations5.2.3 Multinode Multi-GPU Training5.2.3.1 Network Interconnects: Ethernet vs. InfiniBand5.2.3.2 Distributed Storage and I/O Constraints5.3 Resource Management in Distributed Environments5.3.1 GPU Memory Optimization in Multi-GPU Setups5.3.1.1 Gradient Accumulation and Checkpointing5.3.1.2 Tensor Sharding and Memory Reuse5.3.2 Scheduling and Load Balancing5.3.2.1 Batch Sizing and Dynamic Allocation5.3.2.2 Stragglers and Synchronization Challenges5.3.3 Communication Overheads5.3.3.1 AllReduce Algorithms5.3.3.2 Latency vs. Bandwidth Optimization5.4 Framework-Level Support and Configurations5.4.1 PyTorch for Distributed Training5.4.1.1 DistributedDataParallel (DDP)5.4.1.2 TorchElastic and TorchRun5.4.1.3 Best Practices and Debugging Tips5.4.2 TensorFlow MultiWorker Strategy5.4.2.1 MirroredStrategy vs. MultiWorkerMirroredStrategy5.4.2.2 Cluster Setup and Role Configuration5.4.2.3 Monitoring with TensorBoard5.4.3 Horovod and DeepSpeed5.4.3.1 Horovod Overview and Integration5.4.3.2 Zero Redundancy Optimizers in DeepSpeed5.5 Cloud and HPC Considerations5.5.1 Distributed Training on AWS/GCP/Azure5.5.1.1 Setting Up GPU Clusters in the Cloud5.5.1.2 Managed Services: SageMaker, Vertex AI, Etc.5.5.2 Running on HPC Infrastructure5.5.2.1 Using SLURM and MPI5.5.2.2 GPU Node Provisioning and Queuing Systems5.6 Monitoring, Profiling, and Debugging Tools5.6.1 NVIDIA Nsight and NVProf5.6.2 TensorBoard and PyTorch Profiler5.6.3 Common Pitfalls in Distributed Training5.6.3.1 Silent Failures5.6.3.2 Incorrect Synchronization5.6.3.3 Resource Underutilization5.7 Future Directions and Research Opportunities5.7.1 Federated and Decentralized Learning5.7.2 Energy-Efficient Multi-GPU Training5.7.3 AI Model Training on Exascale Systems5.8 Summary
6. Recent Applications and Emerging Trends
6.1 GPU Acceleration in Industry6.2 Industrial Case Studies6.2.1 Real-Time Fraud Detection in Financial Services6.2.1.1 Methodology6.2.1.2 Results and Analysis6.2.1.3 Key Takeaways6.2.2 GPU-Accelerated Virtual Screening for Drug Discovery6.2.2.1 Methodology6.2.2.2 Results and Analysis6.2.2.3 Key Takeaways6.2.3 Data-Parallel Perception with Partial GPU Acceleration for Autonomous Driving6.2.3.1 Methodology6.2.3.2 Results and Analysis6.2.3.3 Key Takeaways6.2.4 Edge-Based Video Analytics for Smart Cities6.2.4.1 Methodology6.2.4.2 Results and Analysis6.2.4.3 Key Takeaways6.2.5 GPU-Accelerated Large-Eddy Simulations for Weather Forecasting6.2.5.1 Methodology6.2.5.2 Results and Analysis6.2.5.3 Key Takeaways6.2.6 Graphite—A Graph-Based Extreme Multilabel Classifier for Keyphrase Recommendation6.2.6.1 Methodology6.2.6.2 Results and Analysis6.2.6.3 Key Takeaways6.3 Future Trends6.3.1 Pretraining and Fine-Tuning Synergy6.3.2 Data Considerations6.3.3 Model Compression and Deployment Efficiency6.3.4 Benchmark Evolution6.3.5 Interpretability and Ethics6.4 Summary
Bibliography
Index

Content preview from GPU-Accelerated Deep Learning : Essential GPU Ideas, Deep Learning Frameworks, and Optimization Approaches

Index

A

Activation functions1, 9, 29
Adam optimizer29
Advanced GPU optimization techniques88
- gradient checkpointing and recomputation90, 91
- matrix multiplication92, 93
- mixed-precision training91, 92
- parallel algorithms88, 90
- tensor cores91, 92
- tiled matrix multiplication88, 89, 90
AI model training24, 119
AllReduce operation96, 106, 109, 111
Artificial neural networks (ANNs)2, 67, 68
atomicAdd()83, 85
Augmented realty (AR)5

B

Backpropagation1, 3, 9, 64, 65, 66
Backpropagation through time (BPTT)21
Benchmarking51, 52
BINDSURF123, 124

C

Central processing unit (CPU)22, 23, 25, 38, 43, 52, 78, 7996, 97
- general-purpose computing tasks73
- host73, 74
Cloud platforms112
- AWS SageMaker112, 113
- Azure Machine Learning112
- Google Vertex AI112
- Microsoft Azure112
- setup GPU clusters112
CNNs, ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Publisher Resources

ISBN: 9798868820830Purchase Link Publisher Website

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

GPU-Accelerated Deep Learning : Essential GPU Ideas, Deep Learning Frameworks, and Optimization Approaches

by Ramchandra S Mangrulkar, Pallavi Vijay Chavan

Index

A

B

C

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.