Advanced Deep Learning with Python

Bidirectional encoder representations from transformers

The bidirectional encoder representations from transformers (BERT) (see BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding at https://arxiv.org/abs/1810.04805) model has a very descriptive name. Let's look at some of the elements that are mentioned:

• Encoder representations: This model uses only the output of the multilayer encoder part of the transformer architecture we described in the The transformer model section.
• Bidirectional: The encoder has an inherent bidirectional nature.

To gain some perspective, let's denote the number of transformer blocks with L, the hidden size with H (previously denoted with d_model), and the number of self-attention heads ...