book

Transfer Learning for Natural Language Processing

Name: Transfer Learning for Natural Language Processing
Author: Paul Azunre
ISBN: 9781617297267

by Paul Azunre

August 2021

Intermediate to advanced

272 pages

7h 54m

English

Manning Publications

Audiobook available

Read now

Unlock full access

Transfer Learning for Natural Language Processing
Copyright
dedication
contents
front matter
prefaceacknowledgmentsabout this bookWho should read this book?Road mapSoftware requirementsAbout the codeliveBook discussion forumabout the authorabout the cover illustration
Part 1 Introduction and overview
1 What is transfer learning?
1.1 Overview of representative NLP tasks1.2 Understanding NLP in the context of AI1.2.1 Artificial intelligence (AI)1.2.2 Machine learning1.2.3 Natural language processing (NLP)1.3 A brief history of NLP advances1.3.1 General overview1.3.2 Recent transfer learning advances1.4 Transfer learning in computer vision1.4.1 General overview1.4.2 Pretrained ImageNet models1.4.3 Fine-tuning pretrained ImageNet models1.5 Why is NLP transfer learning an exciting topic to study now?Summary
2 Getting started with baselines: Data preprocessing
2.1 Preprocessing email spam classification example data2.1.1 Loading and visualizing the Enron corpus2.1.2 Loading and visualizing the fraudulent email corpus2.1.3 Converting the email text into numbers2.2 Preprocessing movie sentiment classification example data2.3 Generalized linear models2.3.1 Logistic regression2.3.2 Support vector machines (SVMs)Summary
3 Getting started with baselines: Benchmarking and optimization
3.1 Decision-tree-based models3.1.1 Random forests (RFs)3.1.2 Gradient-boosting machines (GBMs)3.2 Neural network models3.2.1 Embeddings from Language Models (ELMo)3.2.2 Bidirectional Encoder Representations from Transformers (BERT)3.3 Optimizing performance3.3.1 Manual hyperparameter tuning3.3.2 Systematic hyperparameter tuningSummary
Part 2 Shallow transfer learning and deep transfer learning with recurrent neural networks (RNNs)

4 Shallow transfer learning for NLP
4.1 Semisupervised learning with pretrained word embeddings4.2 Semisupervised learning with higher-level representations4.3 Multitask learning4.3.1 Problem setup and a shallow neural single-task baseline4.3.2 Dual-task experiment4.4 Domain adaptationSummary
5 Preprocessing data for recurrent neural network deep transfer learning experiments
5.1 Preprocessing tabular column-type classification data5.1.1 Obtaining and visualizing tabular data5.1.2 Preprocessing tabular data5.1.3 Encoding preprocessed data as numbers5.2 Preprocessing fact-checking example data5.2.1 Special problem considerations5.2.2 Loading and visualizing fact-checking dataSummary
6 Deep transfer learning for NLP with recurrent neural networks
6.1 Semantic Inference for the Modeling of Ontologies (SIMOn)6.1.1 General neural architecture overview6.1.2 Modeling tabular data6.1.3 Application of SIMOn to tabular column-type classification data6.2 Embeddings from Language Models (ELMo)6.2.1 ELMo bidirectional language modeling6.2.2 Application to fake news detection6.3 Universal Language Model Fine-Tuning (ULMFiT)6.3.1 Target task language model fine-tuning6.3.2 Target task classifier fine-tuningSummary
Part 3 Deep transfer learning with transformers and adaptation strategies
7 Deep transfer learning for NLP with the transformer and GPT
7.1 The transformer7.1.1 An introduction to the transformers library and attention visualization7.1.2 Self-attention7.1.3 Residual connections, encoder-decoder attention, and positional encoding7.1.4 Application of pretrained encoder-decoder to translation7.2 The Generative Pretrained Transformer7.2.1 Architecture overview7.2.2 Transformers pipelines introduction and application to text generation7.2.3 Application to chatbotsSummary
8 Deep transfer learning for NLP with BERT and multilingual BERT
8.1 Bidirectional Encoder Representations from Transformers (BERT)8.1.1 Model architecture8.1.2 Application to question answering8.1.3 Application to fill in the blanks and next-sentence prediction tasks8.2 Cross-lingual learning with multilingual BERT (mBERT)8.2.1 Brief JW300 dataset overview8.2.2 Transfer mBERT to monolingual Twi data with the pretrained tokenizer8.2.3 mBERT and tokenizer trained from scratch on monolingual Twi dataSummary
9 ULMFiT and knowledge distillation adaptation strategies
9.1 Gradual unfreezing and discriminative fine-tuning9.1.1 Pretrained language model fine-tuning9.1.2 Target task classifier fine-tuning9.2 Knowledge distillation9.2.1 Transfer DistilmBERT to monolingual Twi data with pretrained tokenizerSummary
10 ALBERT, adapters, and multitask adaptation strategies
10.1 Embedding factorization and cross-layer parameter sharing10.1.1 Fine-tuning pretrained ALBERT on MDSD book reviews10.2 Multitask fine-tuning10.2.1 General Language Understanding Dataset (GLUE)10.2.2 Fine-tuning on a single GLUE task10.2.3 Sequential adaptation10.3 AdaptersSummary
11 Conclusions
11.1 Overview of key concepts11.2 Other emerging research trends11.2.1 RoBERTa11.2.2 GPT-311.2.3 XLNet11.2.4 BigBird11.2.5 Longformer11.2.6 Reformer11.2.7 T511.2.8 BART11.2.9 XLM11.2.10 TAPAS11.3 Future of transfer learning in NLP11.4 Ethical and environmental considerations11.5 Staying up-to-date11.5.1 Kaggle and Zindi competitions11.5.2 arXiv11.5.3 News and social media (Twitter)11.6 Final wordsSummary
appendix A Kaggle primer
A.1 Free GPUs with Kaggle kernelsA.2 Competitions, discussion, and blog
appendix B Introduction to fundamental deep learning tools
B.1 Stochastic gradient descentB.2 TensorFlowB.3 PyTorchB.4 Keras, fast.ai, and Transformers by Hugging Face
index

Content preview from Transfer Learning for Natural Language Processing

9 ULMFiT and knowledge distillation adaptation strategies

This chapter covers

Implementing the strategies of discriminative fine-tuning and gradual unfreezing
Executing knowledge distillation between teacher and student BERT models

In this chapter and the following chapter, we will cover some adaptation strategies for the deep NLP transfer learning modeling architectures that we have covered so far. In other words, given a pretrained architecture such as ELMo, BERT, or GPT, how can we carry out transfer learning more efficiently? We can employ several measures of efficiency here. We choose to focus on parameter efficiency, where the goal is to yield a model with the fewest parameters possible while suffering minimal reduction in performance. ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Publisher Resources

ISBN: 9781617297267Publisher Support Publisher Website

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Transfer Learning for Natural Language Processing

by Paul Azunre

9 ULMFiT and knowledge distillation adaptation strategies

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.