Mastering Transformers - Second Edition

3

Autoencoding Language Models

In the previous chapter, we looked at and studied how a typical transformer model can be used by Hugging Face’s Transformers. All of the topics in this book so far have included instructions on how to use pre-trained or pre-built models and less information has been given about specific models and their training.

In this chapter, we will learn how we can train autoencoding language models on any given language from scratch. This training will include pretraining and task-specific training of the models. First, we will start with learning about the BERT model and how it works. Then, we will train the language model using a simple and small corpus. Afterward, we will look at how the model can be used inside any ...