Machine Reading Comprehension

Chapter 6

Pretrained language models

Abstract

The pretrained language model has been a hot topic of natural language processing in recent years. The concept of pretraining derives from transfer learning. A model is first pretrained on related tasks and then fine-tuned on the target task to transfer the learned parameters. This methodology is particularly useful when the target task [e.g., machine reading comprehension (MRC)] has insufficient training data, while the related tasks have ample training data.

In this chapter, we will first introduce the concepts and history of pretrained language models, and then analyze several pretrained language models which have been successfully applied to the MRC task. In particular, we will focus on the phenomenal ...