book

Natural Language Processing in Action

Name: Natural Language Processing in Action
ISBN: 9781617294631

by Cole Howard, Hobson Lane, Hannes Hapke

April 2019

Intermediate to advanced

544 pages

17h 29m

English

Manning Publications

Read now

Unlock full access

Inside front cover
Natural Language Processing in Action
Copyright
Brief Table of Contents
Table of Contents
Front matter
ForewordPrefaceAcknowledgmentsHobson LaneHannes Max HapkeCole HowardAbout this BookRoadmapAbout this bookAbout the codeliveBook discussion forumAbout the AuthorsAbout the cover Illustration
Part 1. Wordy machines
1 Packets of thought (NLP overview)
1.1 Natural language vs. programming language1.2 The magic1.2.1 Machines that converse1.2.2 The math1.3 Practical applications1.4 Language through a computer’s “eyes”1.4.1 The language of locks1.4.2 Regular expressions1.4.3 A simple chatbot1.4.4 Another way1.5 A brief overflight of hyperspace1.6 Word order and grammar1.7 A chatbot natural language pipeline1.8 Processing in depth1.9 Natural language IQSummary
2 Build your vocabulary (word tokenization)
2.1 Challenges (a preview of stemming)2.2 Building your vocabulary with a tokenizer2.2.1 Dot product2.2.2 Measuring bag-of-words overlap2.2.3 A token improvement2.2.4 Extending your vocabulary with n-grams2.2.5 Normalizing your vocabulary2.3 Sentiment2.3.1 VADER—A rule-based sentiment analyzer2.3.2 Naive BayesSummary
3 Math with words (TF-IDF vectors)
3.1 Bag of words3.2 Vectorizing3.2.1 Vector spaces3.3 Zipf’s Law3.4 Topic modeling3.4.1 Return of Zipf3.4.2 Relevance ranking3.4.3 Tools3.4.4 Alternatives3.4.5 Okapi BM253.4.6 What’s nextSummary

4 Finding meaning in word counts (semantic analysis)
4.1 From word counts to topic scores4.1.1 TF-IDF vectors and lemmatization4.1.2 Topic vectors4.1.3 Thought experiment4.1.4 An algorithm for scoring topics4.1.5 An LDA classifier4.2 Latent semantic analysis4.2.1 Your thought experiment made real4.3 Singular value decomposition4.3.1 U—left singular vectors4.3.2 S—singular values4.3.3 VT—right singular vectors4.3.4 SVD matrix orientation4.3.5 Truncating the topics4.4 Principal component analysis4.4.1 PCA on 3D vectors4.4.2 Stop horsing around and get back to NLP4.4.3 Using PCA for SMS message semantic analysis4.4.4 Using truncated SVD for SMS message semantic analysis4.4.5 How well does LSA work for spam classification?4.5 Latent Dirichlet allocation (LDiA)4.5.1 The LDiA idea4.5.2 LDiA topic model for SMS messages4.5.3 LDiA + LDA = spam classifier4.5.4 A fairer comparison: 32 LDiA topics4.6 Distance and similarity4.7 Steering with feedback4.7.1 Linear discriminant analysis4.8 Topic vector power4.8.1 Semantic search4.8.2 ImprovementsSummary
Part 2. Deeper learning (neural networks)
5 Baby steps with neural networks (perceptrons and backpropagation)
5.1 Neural networks, the ingredient list5.1.1 Perceptron5.1.2 A numerical perceptron5.1.3 Detour through bias5.1.4 Let’s go skiing—the error surface5.1.5 Off the chair lift, onto the slope5.1.6 Let’s shake things up a bit5.1.7 Keras: Neural networks in Python5.1.8 Onward and deepward5.1.9 Normalization: input with styleSummary
6 Reasoning with word vectors (Word2vec)
6.1 Semantic queries and analogies6.1.1 Analogy questions6.2 Word vectors6.2.1 Vector-oriented reasoning6.2.2 How to compute Word2vec representations6.2.3 How to use the gensim.word2vec module6.2.4 How to generate your own word vector representations6.2.5 Word2vec vs. GloVe (Global Vectors)6.2.6 fastText6.2.7 Word2vec vs. LSA6.2.8 Visualizing word relationships6.2.9 Unnatural words6.2.10 Document similarity with Doc2vecSummary
7 Getting words in order with convolutional neural networks (CNNs)
7.1 Learning meaning7.2 Toolkit7.3 Convolutional neural nets7.3.1 Building blocks7.3.2 Step size (stride)7.3.3 Filter composition7.3.4 Padding7.3.5 Learning7.4 Narrow windows indeed7.4.1 Implementation in Keras: prepping the data7.4.2 Convolutional neural network architecture7.4.3 Pooling7.4.4 Dropout7.4.5 The cherry on the sundae7.4.6 Let’s get to learning (training)7.4.7 Using the model in a pipeline7.4.8 Where do you go from here?Summary
8 Loopy (recurrent) neural networks (RNNs)
8.1 Remembering with recurrent networks8.1.1 Backpropagation through time8.1.2 When do we update what?8.1.3 Recap8.1.4 There’s always a catch8.1.5 Recurrent neural net with Keras8.2 Putting things together8.3 Let’s get to learning our past selves8.4 Hyperparameters8.5 Predicting8.5.1 Statefulness8.5.2 Two-way street8.5.3 What is this thing?Summary
9 Improving retention with long short-term memory networks
9.1 LSTM9.1.1 Backpropagation through time9.1.2 Where does the rubber hit the road?9.1.3 Dirty data9.1.4 Back to the dirty data9.1.5 Words are hard. Letters are easier.9.1.6 My turn to chat9.1.7 My turn to speak more clearly9.1.8 Learned how to say, but not yet what9.1.9 Other kinds of memory9.1.10 Going deeperSummary
10 Sequence-to-sequence models and attention
10.1 Encoder-decoder architecture10.1.1 Decoding thought10.1.2 Look familiar?10.1.3 Sequence-to-sequence conversation10.1.4 LSTM review10.2 Assembling a sequence-to-sequence pipeline10.2.1 Preparing your dataset for the sequence-to-sequence training10.2.2 Sequence-to-sequence model in Keras10.2.3 Sequence encoder10.2.4 Thought decoder10.2.5 Assembling the sequence-to-sequence network10.3 Training the sequence-to-sequence network10.3.1 Generate output sequences10.4 Building a chatbot using sequence-to-sequence networks10.4.1 Preparing the corpus for your training10.4.2 Building your character dictionary10.4.3 Generate one-hot encoded training sets10.4.4 Train your sequence-to-sequence chatbot10.4.5 Assemble the model for sequence generation10.4.6 Predicting a sequence10.4.7 Generating a response10.4.8 Converse with your chatbot10.5 Enhancements10.5.1 Reduce training complexity with bucketing10.5.2 Paying attention10.6 In the real worldSummary
Part 3. Getting real (real-world NLP challenges)
11 Information extraction (named entity extraction and question answering)
11.1 Named entities and relations11.1.1 A knowledge base11.1.2 Information extraction11.2 Regular patterns11.2.1 Regular expressions11.2.2 Information extraction as ML feature extraction11.3 Information worth extracting11.3.1 Extracting GPS locations11.3.2 Extracting dates11.4 Extracting relationships (relations)11.4.1 Part-of-speech (POS) tagging11.4.2 Entity name normalization11.4.3 Relation normalization and extraction11.4.4 Word patterns11.4.5 Segmentation11.4.6 Why won’t split('.!?') work?11.4.7 Sentence segmentation with regular expressions11.5 In the real worldSummary
12 Getting chatty (dialog engines)
12.1 Language skill12.1.1 Modern approaches12.1.2 A hybrid approach12.2 Pattern-matching approach12.2.1 A pattern-matching chatbot with AIML12.2.2 A network view of pattern matching12.3 Grounding12.4 Retrieval (search)12.4.1 The context challenge12.4.2 Example retrieval-based chatbot12.4.3 A search-based chatbot12.5 Generative models12.5.1 Chat about NLPIA12.5.2 Pros and cons of each approach12.6 Four-wheel drive12.6.1 The Will to succeed12.7 Design process12.8 Trickery12.8.1 Ask questions with predictable answers12.8.2 Be entertaining12.8.3 When all else fails, search12.8.4 Being popular12.8.5 Be a connector12.8.6 Getting emotional12.9 In the real worldSummary
13 Scaling up (optimization, parallelization, and batch processing)
13.1 Too much of a good thing (data)13.2 Optimizing NLP algorithms13.2.1 Indexing13.2.2 Advanced indexing13.2.3 Advanced indexing with Annoy13.2.4 Why use approximate indexes at all?13.2.5 An indexing workaround: discretizing13.3 Constant RAM algorithms13.3.1 Gensim13.3.2 Graph computing13.4 Parallelizing your NLP computations13.4.1 Training NLP models on GPUs13.4.2 Renting vs. buying13.4.3 GPU rental options13.4.4 Tensor processing units13.5 Reducing the memory footprint during model training13.6 Gaining model insights with TensorBoard13.6.1 How to visualize word embeddingsSummary
Appendix A. Your NLP tools
A.1 Anaconda3A.2 Install NLPIAA.3 IDEA.4 Ubuntu package managerA.5 MacA.5.1 A Mac package managerA.5.2 Some packagesA.5.3 TuneupsA.6 WindowsA.6.1 Get VirtualA.7 NLPIA automagic
Appendix B. Playful Python and regular expressions
B.1 Working with stringsB.1.1 String types (str and bytes)B.1.2 Templates in Python (.format())B.2 Mapping in Python (dict and OrderedDict)B.3 Regular expressionsB.3.1 |—ORB.3.2 ()—GroupsB.3.3 []—Character classesB.4 StyleB.5 Mastery
Appendix C. Vectors and matrices (linear algebra fundamentals)
C.1 VectorsC.1.1 Distances
Appendix D. Machine learning tools and techniques
D.1 Data selection and avoiding biasD.2 How fit is fit?D.3 Knowing is half the battleD.4 Cross-fit trainingD.5 Holding your model backD.5.1 RegularizationD.5.2 DropoutD.5.3 Batch normalizationD.6 Imbalanced training setsD.6.1 OversamplingD.6.2 UndersamplingD.6.3 Augmenting your dataD.7 Performance metricsD.7.1 Measuring classifier performanceD.7.2 Measuring regressor performanceD.8 Pro tips
Appendix E. Setting up your AWS GPU
E.1 Steps to create your AWS GPU instanceE.1.1 Cost control
Appendix F. Locality sensitive hashing
F.1 High-dimensional vectors are differentF.1.1 Vector space indexes and hashesF.1.2 High-dimensional thinkingF.2 High-dimensional indexingF.2.1 Locality sensitive hashingF.2.2 Approximate nearest neighborsF.3 “Like” prediction
Resources
Applications and project ideasCourses and tutorialsTools and packagesResearch papers and talksVector space models and semantic searchFinanceQuestion answering systemsDeep learningLSTMs and RNNsCompetitions and awardsDatasetsSearch enginesSearch algorithmsOpen source search enginesOpen source full-text indexersManipulative search enginesLess manipulative search enginesDistributed search engines
Glossary
AcronymsTerms
Index
List of Figures
List of Tables
List of Listings

Content preview from Natural Language Processing in Action

6 Reasoning with word vectors (Word2vec)

This chapter covers

Understanding how word vectors are created
Using pretrained models for your applications
Reasoning with word vectors to solve real problems
Visualizing word vectors
Uncovering some surprising uses for word embeddings

One of the most exciting recent advancements in NLP is the “discovery” of word vectors. This chapter will help you understand what they are and how to use them to do some surprisingly powerful things. You’ll learn how to recover some of the fuzziness and subtlety of word meaning that was lost in the approximations of earlier chapters.

In the previous chapters, we ignored the nearby context of a word. We ignored the words around each word. We ignored the effect the neighbors ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Publisher Resources

ISBN: 9781617294631Publisher Support Publisher Website Errata Page

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Natural Language Processing in Action

by Cole Howard, Hobson Lane, Hannes Hapke

6 Reasoning with word vectors (Word2vec)

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.