Hands-On GPU Programming with Python and CUDA

Conditioning and normalizing data

Before we move on to training and testing our brand-new NN, we need to step back for a moment and talk about conditioning and normalizing data. NNs are highly susceptible to numerical error, especially when inputs have a large variance in scale. This can be mitigated by properly conditioning our training data; this means that for each point in an input sample, we will calculate the mean and variance of each point over all samples, and then subtract the mean and divide by the standard deviation for each point in each sample before it is input into the NN for either training or inference (prediction). This method is known as normalization. Let's put together a small Python function that can do this for us: ...