book

Deep Learning for Natural Language Processing

Name: Deep Learning for Natural Language Processing
ISBN: 9781838550295

by Karthiek Reddy Bokka, Shubhangi Hora, Tanuj Jain, Monicah Wambugu

June 2019

Intermediate to advanced

372 pages

6h 34m

English

Packt Publishing

Read now

Unlock full access

Preface
About the BookAbout the AuthorsDescriptionLearning ObjectivesAudienceApproachHardware RequirementsSoftware RequirementsConventionsInstallation and SetupInstall Python on WindowsInstall Python on LinuxInstall Python on macOS XInstalling KerasAdditional Resources
Chapter 1
Introduction to Natural Language ProcessingIntroductionThe Basics of Natural Language ProcessingImportance of natural language processingCapabilities of Natural language processingApplications of Natural Language ProcessingText PreprocessingText Preprocessing TechniquesLowercasing/UppercasingExercise 1: Performing Lowercasing on a SentenceNoise RemovalExercise 2: Removing Noise from WordsText NormalizationStemmingExercise 3: Performing Stemming on WordsLemmatizationExercise 4: Performing Lemmatization on WordsTokenizationExercise 5: Tokenizing WordsExercise 6: Tokenizing SentencesAdditional TechniquesExercise 7: Removing Stop WordsWord EmbeddingsThe Generation of Word EmbeddingsWord2VecFunctioning of Word2VecExercise 8: Generating Word Embeddings Using Word2VecGloVeExercise 9: Generating Word Embeddings Using GloVeActivity 1: Generating Word Embeddings from a Corpus Using Word2Vec.Summary
Chapter 2
Applications of Natural Language ProcessingIntroductionPOS TaggingParts of SpeechPOS TaggerApplications of Parts of Speech TaggingTypes of POS TaggersRule-Based POS TaggersExercise 10: Performing Rule-Based POS TaggingStochastic POS TaggersExercise 11: Performing Stochastic POS TaggingChunkingExercise 12: Performing Chunking with NLTKExercise 13: Performing Chunking with spaCyChinkingExercise 14: Performing ChinkingActivity 2: Building and Training Your Own POS TaggerNamed Entity RecognitionNamed EntitiesNamed Entity RecognizersApplications of Named Entity RecognitionTypes of Named Entity RecognizersRule-Based NERsStochastic NERsExercise 15: Perform Named Entity Recognition with NLTKExercise 16: Performing Named Entity Recognition with spaCyActivity 3: Performing NER on a Tagged CorpusSummary
Chapter 3
Introduction to Neural NetworksIntroductionIntroduction to Deep LearningComparing Machine Learning and Deep LearningNeural NetworksNeural Network ArchitectureThe LayersNodesThe EdgesBiasesActivation FunctionsTraining a Neural NetworkCalculating WeightsThe Loss FunctionThe Gradient Descent AlgorithmBackpropagationDesigning a Neural Network and Its ApplicationsSupervised neural networksUnsupervised neural networksExercise 17: Creating a neural networkFundamentals of Deploying a Model as a ServiceActivity 4: Sentiment Analysis of ReviewsSummary
Chapter 4
Foundations of Convolutional Neural NetworkIntroductionExercise 18: Finding Out How Computers See ImagesUnderstanding the Architecture of a CNNFeature ExtractionConvolutionThe ReLU Activation FunctionExercise 19: Visualizing ReLUPoolingDropoutClassification in Convolutional Neural NetworkExercise 20: Creating a Simple CNN ArchitectureTraining a CNNExercise 21: Training a CNNApplying CNNs to TextExercise 22: Application of a Simple CNN to a Reuters News Topic for ClassificationApplication Areas of CNNsActivity 5: Sentiment Analysis on a Real-life DatasetSummary
Chapter 5
Recurrent Neural NetworksIntroductionPrevious Versions of Neural NetworksRNNsRNN ArchitecturesBPTTUpdates and Gradient FlowAdjusting Weight Matrix WyAdjusting Weight Matrix WsFor Updating WxGradientsExploding GradientsVanishing GradientsRNNs with KerasExercise 23: Building an RNN Model to Show the Stability of Parameters over TimeStateful versus StatelessExercise 24: Turning a Stateless Network into a Stateful Network by Only Changing ArgumentsActivity 6: Solving a Problem with an RNN – Author AttributionSummary
Chapter 6
Gated Recurrent Units (GRUs)IntroductionThe Drawback of Simple RNNsThe Exploding Gradient Problem Gated Recurrent Units (GRUs) Types of GatesThe Update GateThe Reset GateThe Candidate Activation FunctionGRU VariationsSentiment Analysis with GRU Exercise 25: Calculating the Model Validation Accuracy and Loss for Sentiment ClassificationActivity 7: Developing a Sentiment Classification Model Using a Simple RNNText Generation with GRUs Exercise 26: Generating Text Using GRUsActivity 8: Train Your Own Character Generation Model Using a Dataset of Your Choice Summary
Chapter 7
Long Short-Term Memory (LSTM)IntroductionLSTMThe Forget GateThe Input Gate and the Candidate Cell StateCell State UpdateOutput Gate and Current ActivationExercise 27: Building an LSTM-Based Model to Classify an Email as Spam or Not Spam (Ham)Activity 9: Building a Spam or Ham Classifier Using a Simple RNNNeural Language TranslationActivity 10: Creating a French-to-English translation modelSummary
Chapter 8
State-of-the-Art Natural Language ProcessingIntroductionAttention MechanismsAn Attention Mechanism ModelData Normalization Using an Attention MechanismEncoderDecoderAttention mechanismsThe Calculation of AlphaExercise 28: Build a Date Normalization Model for a Database ColumnOther Architectures and DevelopmentsTransformerBERTOpen AI GPT-2Activity 11: Build a Text Summarization ModelSummary
Chapter 9
A Practical NLP Project Workflow in an OrganizationIntroductionGeneral Workflow for the Development of a Machine Learning ProductThe Presentation Workflow:The Research Workflow:The Production-Oriented WorkflowProblem DefinitionData AcquisitionGoogle ColabFlaskDeploymentMaking Changes to a Flask Web AppUse Docker to Wrap the Flask Web Application into a ContainerHost the Container on an Amazon Web Services (AWS) EC2 instanceImprovementsSummary

Appendix
Chapter 1: Introduction to Natural Language ProcessingActivity 1: Generating word embeddings from a corpus using Word2Vec. Chapter 2: Applications of Natural Language ProcessingActivity 2: Building and training your own POS taggerActivity 3: Performing NER on a Tagged CorpusChapter 3: Introduction to Neural NetworksActivity 4: Sentiment Analysis of ReviewsChapter 4: Introduction to convolutional networksActivity 5: Sentiment Analysis on a real-life datasetChapter 5: Foundations of Recurrent Neural NetworkActivity 6: Solve a problem with RNN – Author AttributionPrepare the dataApplying the Model to the Unknown PapersChapter 6: Foundations of GRUsActivity 7: Develop a sentiment classification model using Simple RNNActivity 8: Train your own character generation model with a dataset of your choiceChapter 7: Foundations of LSTMActivity 10: Create a French to English translation modelChapter 8: State of the art in Natural Language ProcessingActivity 11: Build a Text Summarization ModelChapter 9: A practical NLP project workflow in an organisationCode for LSTM modelCode for Flask

Content preview from Deep Learning for Natural Language Processing

Chapter 6 Gated Recurrent Units (GRUs)

Learning Objectives

By the end of this chapter, you will be able to:

Assess the drawback of simple Recurrent Neural Networks (RNNs)
Describe the architecture of Gated Recurrent Units (GRUs)
Perform sentiment analysis using GRUs
Apply GRUs for text generation

The chapter aims to provide a solution to the existing drawbacks of the current architecture of RNNs.

Introduction

In previous chapters, we studied text processing techniques such as word embedding, tokenization, and Term Frequency Inverse Document Frequency (TFIDF). We also learned about a specific network architecture called a Recurrent Neural Network (RNN) that has the drawback of vanishing gradients.

In this chapter, we are going to study ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Deep Learning for Natural Language Processing

Publisher Resources

ISBN: 9781838550295

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Deep Learning for Natural Language Processing

by Karthiek Reddy Bokka, Shubhangi Hora, Tanuj Jain, Monicah Wambugu

Chapter 6

Gated Recurrent Units (GRUs)

Learning Objectives

Introduction

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.