book

Real-World Natural Language Processing

Name: Real-World Natural Language Processing
Author: Masato Hagiwara
ISBN: 9781617296420

by Masato Hagiwara

November 2021

Beginner to intermediate

336 pages

10h 37m

English

Manning Publications

Read now

Unlock full access

Real-World Natural Language Processing
inside front cover
Copyright
dedication
contents
index
front matter
prefaceacknowledgmentsabout this bookWho should read this bookHow this book is organized: A roadmapAbout the codeliveBook discussion forumOther online resourcesabout the authorabout the cover illustration
Part 1 Basics
1 Introduction to natural language processing
1.1 What is natural language processing (NLP)?1.1.1 What is NLP?1.1.2 What is not NLP?1.1.3 AI, ML, DL, and NLP1.1.4 Why NLP?1.2 How NLP is used1.2.1 NLP applications1.2.2 NLP tasks1.3 Building NLP applications1.3.1 Development of NLP applications1.3.2 Structure of NLP applicationsSummary
2 Your first NLP application
2.1 Introducing sentiment analysis2.2 Working with NLP datasets2.2.1 What is a dataset?2.2.2 Stanford Sentiment Treebank2.2.3 Train, validation, and test sets2.2.4 Loading SST datasets using AllenNLP2.3 Using word embeddings2.3.1 What are word embeddings?2.3.2 Using word embeddings for sentiment analysis2.4 Neural networks2.4.1 What are neural networks?2.4.2 Recurrent neural networks (RNNs) and linear layers2.4.3 Architecture for sentiment analysis2.5 Loss functions and optimization2.6 Training your own classifier2.6.1 Batching2.6.2 Putting everything together2.7 Evaluating your classifier2.8 Deploying your application2.8.1 Making predictions2.8.2 Serving predictionsSummary
3 Word and document embeddings
3.1 Introducing embeddings3.1.1 What are embeddings?3.1.2 Why are embeddings important?3.2 Building blocks of language: Characters, words, and phrases3.2.1 Characters3.2.2 Words, tokens, morphemes, and phrases3.2.3 N-grams3.3 Tokenization, stemming, and lemmatization3.3.1 Tokenization3.3.2 Stemming3.3.3 Lemmatization3.4 Skip-gram and continuous bag of words (CBOW)3.4.1 Where word embeddings come from3.4.2 Using word associations3.4.3 Linear layers3.4.4 Softmax3.4.5 Implementing Skip-gram on AllenNLP3.4.6 Continuous bag of words (CBOW) model3.5 GloVe3.5.1 How GloVe learns word embeddings3.5.2 Using pretrained GloVe vectors3.6 fastText3.6.1 Making use of subword information3.6.2 Using the fastText toolkit3.7 Document-level embeddings3.8 Visualizing embeddingsSummary

4 Sentence classification
4.1 Recurrent neural networks (RNNs)4.1.1 Handling variable-length input4.1.2 RNN abstraction4.1.3 Simple RNNs and nonlinearity4.2 Long short-term memory units (LSTMs) and gated recurrent units (GRUs)4.2.1 Vanishing gradients problem4.2.2 Long short-term memory (LSTM)4.2.3 Gated recurrent units (GRUs)4.3 Accuracy, precision, recall, and F-measure4.3.1 Accuracy4.3.2 Precision and recall4.3.3 F-measure4.4 Building AllenNLP training pipelines4.4.1 Instances and fields4.4.2 Vocabulary and token indexers4.4.3 Token embedders and RNNs4.4.4 Building your own model4.4.5 Putting it all together4.5 Configuring AllenNLP training pipelines4.6 Case study: Language detection4.6.1 Using characters as input4.6.2 Creating a dataset reader4.6.3 Building the training pipeline4.6.4 Running the detector on unseen instancesSummary
5 Sequential labeling and language modeling
5.1 Introducing sequential labeling5.1.1 What is sequential labeling?5.1.2 Using RNNs to encode sequences5.1.3 Implementing a Seq2Seq encoder in AllenNLP5.2 Building a part-of-speech tagger5.2.1 Reading a dataset5.2.2 Defining the model and the loss5.2.3 Building the training pipeline5.3 Multilayer and bidirectional RNNs5.3.1 Multilayer RNNs5.3.2 Bidirectional RNNs5.4 Named entity recognition5.4.1 What is named entity recognition?5.4.2 Tagging spans5.4.3 Implementing a named entity recognizer5.5 Modeling a language5.5.1 What is a language model?5.5.2 Why are language models useful?5.5.3 Training an RNN language model5.6 Text generation using RNNs5.6.1 Feeding characters to an RNN5.6.2 Evaluating text using a language model5.6.3 Generating text using a language modelSummary
Part 2 Advanced models
6 Sequence-to-sequence models
6.1 Introducing sequence-to-sequence models6.2 Machine translation 1016.3 Building your first translator6.3.1 Preparing the datasets6.3.2 Training the model6.3.3 Running the translator6.4 How Seq2Seq models work6.4.1 Encoder6.4.2 Decoder6.4.3 Greedy decoding6.4.4 Beam search decoding6.5 Evaluating translation systems6.5.1 Human evaluation6.5.2 Automatic evaluation6.6 Case study: Building a chatbot6.6.1 Introducing dialogue systems6.6.2 Preparing a dataset6.6.3 Training and running a chatbot6.6.4 Next stepsSummary
7 Convolutional neural networks
7.1 Introducing convolutional neural networks (CNNs)7.1.1 RNNs and their shortcomings7.1.2 Pattern matching for sentence classification7.1.3 Convolutional neural networks (CNNs)7.2 Convolutional layers7.2.1 Pattern matching using filters7.2.2 Rectified linear unit (ReLU)7.2.3 Combining scores7.3 Pooling layers7.4 Case study: Text classification7.4.1 Review: Text classification7.4.2 Using CnnEncoder7.4.3 Training and running the classifierSummary
8 Attention and Transformer
8.1 What is attention?8.1.1 Limitation of vanilla Seq2Seq models8.1.2 Attention mechanism8.2 Sequence-to-sequence with attention8.2.1 Encoder-decoder attention8.2.2 Building a Seq2Seq machine translation with attention8.3 Transformer and self-attention8.3.1 Self-attention8.3.2 Transformer8.3.3 Experiments8.4 Transformer-based language models8.4.1 Transformer as a language model8.4.2 Transformer-XL8.4.3 GPT-28.4.4 XLM8.5 Case study: Spell-checker8.5.1 Spell correction as machine translation8.5.2 Training a spell-checker8.5.3 Improving a spell-checkerSummary
9 Transfer learning with pretrained language models
9.1 Transfer learning9.1.1 Traditional machine learning9.1.2 Word embeddings9.1.3 What is transfer learning?9.2 BERT9.2.1 Limitations of word embeddings9.2.2 Self-supervised learning9.2.3 Pretraining BERT9.2.4 Adapting BERT9.3 Case study 1: Sentiment analysis with BERT9.3.1 Tokenizing input9.3.2 Building the model9.3.3 Training the model9.4 Other pretrained language models9.4.1 ELMo9.4.2 XLNet9.4.3 RoBERTa9.4.4 DistilBERT9.4.5 ALBERT9.5 Case study 2: Natural language inference with BERT9.5.1 What is natural language inference?9.5.2 Using BERT for sentence-pair classification9.5.3 Using Transformers with AllenNLPSummary
Part 3 Putting into production
10 Best practices in developing NLP applications
10.1 Batching instances10.1.1 Padding10.1.2 Sorting10.1.3 Masking10.2 Tokenization for neural models10.2.1 Unknown words10.2.2 Character models10.2.3 Subword models10.3 Avoiding overfitting10.3.1 Regularization10.3.2 Early stopping10.3.3 Cross-validation10.4 Dealing with imbalanced datasets10.4.1 Using appropriate evaluation metrics10.4.2 Upsampling and downsampling10.4.3 Weighting losses10.5 Hyperparameter tuning10.5.1 Examples of hyperparameters10.5.2 Grid search vs. random search10.5.3 Hyperparameter tuning with OptunaSummary
11 Deploying and serving NLP applications
11.1 Architecting your NLP application11.1.1 Before machine learning11.1.2 Choosing the right architecture11.1.3 Project structure11.1.4 Version control11.2 Deploying your NLP model11.2.1 Testing11.2.2 Train-serve skew11.2.3 Monitoring11.2.4 Using GPUs11.3 Case study: Serving and deploying NLP applications11.3.1 Serving models with TorchServe11.3.2 Deploying models with SageMaker11.4 Interpreting and visualizing model predictions11.5 Where to go from hereSummary
index

Content preview from Real-World Natural Language Processing

8 Attention and Transformer

This chapter covers

Using attention to produce summaries of the input and improve the quality of Seq2Seq models
Replacing RNN-style loops with self-attention, a mechanism for the input to summarize itself
Improving machine translation systems with the Transformer model
Building a high-quality spell-checker using the Transformer model and publicly available datasets

Our focus so far in this book has been recurrent neural networks (RNNs), which are a powerful model that can be applied to various NLP tasks such as sentiment analysis, named entity recognition, and machine translation. In this chapter, we will introduce an even more powerful model—the Transformer¹—a new type of encoder-decoder neural network architecture ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Publisher Resources

ISBN: 9781617296420Publisher Support Publisher Website

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Real-World Natural Language Processing

by Masato Hagiwara

8 Attention and Transformer

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.