The Deep Learning Architect's Handbook

6

Understanding Neural Network Transformers

Not to be confused with the electrical devices that are also called transformers, neural network transformers are the jack-of-all-trades variant of NNs. Transformers are capable of processing and capturing patterns from data of any modality, including sequential data such as text data and time-series data, image data, audio data, and video data.

The transformer architecture was introduced in 2017 with the motive of replacing RNN-based sequence-to-sequence architectures and primarily focusing on the machine translation use case of converting text data from one language to another language. The results performed better than the baseline RNN-based model and proved that we don’t need inherent inductive ...