Advanced Deep Learning with Python

Implementing transformers

In this section, we'll implement the transformer model with the help of PyTorch 1.3.1. As the example is relatively complex, we'll simplify it by using a basic training dataset: we'll train the model to copy a randomly generated sequence of integer values—that is, the source and the target sequence are the same and the transformer will learn to replicate the input sequence as the output. We won't include the full source code, but you can find it at https://github.com/PacktPublishing/Advanced-Deep-Learning-with-Python/tree/master/Chapter08/transformer.py.

This example is based on https://github.com/harvardnlp/annotated-transformer. Let's also note that PyTorch 1.2 has introduced native transformer modules (the documentation ...