book

Natural Language Processing with PyTorch

by Delip Rao, Brian McMahan

February 2019

Intermediate to advanced

254 pages

6h 48m

English

O'Reilly Media, Inc.

Read now

Unlock full access

Preface
Conventions Used in This BookUsing Code ExamplesO’Reilly SafariHow to Contact UsAcknowledments
1. Introduction
The Supervised Learning ParadigmObservation and Target EncodingOne-Hot RepresentationTF RepresentationTF-IDF RepresentationTarget EncodingComputational GraphsPyTorch BasicsInstalling PyTorchCreating TensorsTensor Types and SizeTensor OperationsIndexing, Slicing, and JoiningTensors and Computational GraphsCUDA TensorsExercisesSolutionsSummaryReferences
2. A Quick Tour of Traditional NLP
Corpora, Tokens, and TypesUnigrams, Bigrams, Trigrams, …, N-gramsLemmas and StemsCategorizing Sentences and DocumentsCategorizing Words: POS TaggingCategorizing Spans: Chunking and Named Entity RecognitionStructure of SentencesWord Senses and SemanticsSummaryReferences
3. Foundational Components of Neural Networks
The Perceptron: The Simplest Neural NetworkActivation FunctionsSigmoidTanhReLUSoftmaxLoss FunctionsMean Squared Error LossCategorical Cross-Entropy LossBinary Cross-Entropy LossDiving Deep into Supervised TrainingConstructing Toy DataPutting It Together: Gradient-Based Supervised LearningAuxiliary Training ConceptsCorrectly Measuring Model Performance: Evaluation MetricsCorrectly Measuring Model Performance: Splitting the DatasetKnowing When to Stop TrainingFinding the Right HyperparametersRegularizationExample: Classifying Sentiment of Restaurant ReviewsThe Yelp Review DatasetUnderstanding PyTorch’s Dataset RepresentationThe Vocabulary, the Vectorizer, and the DataLoaderA Perceptron ClassifierThe Training RoutineEvaluation, Inference, and InspectionSummaryReferences
4. Feed-Forward Networks for Natural Language Processing
The Multilayer PerceptronA Simple Example: XORImplementing MLPs in PyTorchExample: Surname Classification with an MLPThe Surnames DatasetVocabulary, Vectorizer, and DataLoaderThe SurnameClassifier ModelThe Training RoutineModel Evaluation and PredictionRegularizing MLPs: Weight Regularization and Structural Regularization (or Dropout)Convolutional Neural NetworksCNN HyperparametersImplementing CNNs in PyTorchExample: Classifying Surnames by Using a CNNThe SurnameDataset ClassVocabulary, Vectorizer, and DataLoaderReimplementing the SurnameClassifier with Convolutional NetworksThe Training RoutineModel Evaluation and PredictionMiscellaneous Topics in CNNsPoolingBatch Normalization (BatchNorm)Network-in-Network Connections (1x1 Convolutions)Residual Connections/Residual BlockSummaryReferences
5. Embedding Words and Types
Why Learn Embeddings?Efficiency of EmbeddingsApproaches to Learning Word EmbeddingsThe Practical Use of Pretrained Word EmbeddingsExample: Learning the Continuous Bag of Words EmbeddingsThe Frankenstein DatasetVocabulary, Vectorizer, and DataLoaderThe CBOWClassifier ModelThe Training RoutineModel Evaluation and PredictionExample: Transfer Learning Using Pretrained Embeddings for Document ClassificationThe AG News DatasetVocabulary, Vectorizer, and DataLoaderThe NewsClassifier ModelThe Training RoutineModel Evaluation and PredictionEvaluating on the test datasetSummaryReferences
6. Sequence Modeling for Natural Language Processing
Introduction to Recurrent Neural NetworksImplementing an Elman RNNExample: Classifying Surname Nationality Using a Character RNNThe SurnameDataset ClassThe Vectorization Data StructuresThe SurnameClassifier ModelThe Training Routine and ResultsSummaryReferences
7. Intermediate Sequence Modeling for Natural Language Processing
The Problem with Vanilla RNNs (or Elman RNNs)Gating as a Solution to a Vanilla RNN’s ChallengesExample: A Character RNN for Generating SurnamesThe SurnameDataset ClassThe Vectorization Data StructuresFrom the ElmanRNN to the GRUModel 1: The Unconditioned SurnameGenerationModelModel 2: The Conditioned SurnameGenerationModelThe Training Routine and ResultsTips and Tricks for Training Sequence ModelsReferences
8. Advanced Sequence Modeling for Natural Language Processing
Sequence-to-Sequence Models, Encoder–Decoder Models, and Conditioned GenerationCapturing More from a Sequence: Bidirectional Recurrent ModelsCapturing More from a Sequence: AttentionAttention in Deep Neural NetworksEvaluating Sequence Generation ModelsExample: Neural Machine TranslationThe Machine Translation DatasetA Vectorization Pipeline for NMTEncoding and Decoding in the NMT ModelThe Training Routine and ResultsSummaryReferences
9. Classics, Frontiers, and Next Steps
What Have We Learned so Far?Timeless Topics in NLPDialogue and Interactive SystemsDiscourseInformation Extraction and Text MiningDocument Analysis and RetrievalFrontiers in NLPDesign Patterns for Production NLP SystemsWhere Next?References

Index

Overview

Natural Language Processing (NLP) provides boundless opportunities for solving problems in artificial intelligence, making products such as Amazon Alexa and Google Translate possible. If you’re a developer or data scientist new to NLP and deep learning, this practical guide shows you how to apply these methods using PyTorch, a Python-based deep learning library.

Authors Delip Rao and Brian McMahon provide you with a solid grounding in NLP and deep learning algorithms and demonstrate how to use PyTorch to build applications involving rich representations of text specific to the problems you face. Each chapter includes several code examples and illustrations.

Explore computational graphs and the supervised learning paradigm
Master the basics of the PyTorch optimized tensor manipulation library
Get an overview of traditional NLP concepts and methods
Learn the basic ideas involved in building neural networks
Use embeddings to represent words, sentences, documents, and other features
Explore sequence prediction and generate sequence-to-sequence models
Learn design patterns for building production NLP systems

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Publisher Resources

ISBN: 9781491978221Errata Page

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills