TensorFlow 1.x Deep Learning Cookbook

Components of TPUs

In all the deep learning models covered in this book, irrespective of the learning paradigm, three basic calculations were necessary: multiplication, addition, and application of an activation function. 

The first two components are part of matrix multiplication: the weight matrix W needs to be multiplied to input matrix X, generally expressed as W^TX; matrix multiplication is computationally expensive on a CPU, and though a GPU parallelizes the operation, still there is scope for improvement.

The TPU has a 65,536 8-bit integer matrix multiplier unit (MXU) that gives a peak throughput of 92 TOPS. The major difference between GPU and TPU multiplication is that GPUs contain floating point multipliers, while TPUs contain 8-bit ...