Advanced Deep Learning with Python

Input data representation

Before going into each training step, let's discuss the input and output data representations, which are shared by the two steps. Somewhat similar to fastText (see Chapter 6, Language Modeling), BERT uses a data-driven tokenization algorithm called WordPiece (https://arxiv.org/abs/1609.08144). This means that, instead of a vocabulary of full words, it creates a vocabulary of subword tokens in an iterative process until that vocabulary reaches a predetermined size (in the case of BERT, the size is 30,000 tokens). This approach has two main advantages:

• It allows us to control the size of the dictionary.
• It handles unknown words by assigning them to the closest existing dictionary subword token.

BERT can handle a ...