Hands-On GPU Programming with Python and CUDA

Chapter 11, Performance Optimization in CUDA

1. The fact that atomicExch is thread-safe doesn't guarantee that all threads will execute this function at the same time (which is not the case since different blocks in a grid can be executed at different times).
2. A block of size 100 will be executed over multiple warps, which will not be synchronized within the block unless we use __syncthreads. Thus, atomicExch may be called at multiple times.
3. Since a warp executes in lockstep by default, and blocks of size 32 or less are executed with a single warp, __syncthreads would be unnecessary.
4. We use a naïve parallel sum within the warp, but otherwise, we are doing as many sums withatomicAdd as we would do with a serial sum. While CUDA automatically parallelizes ...