book

Natural Language Processing in Action, Second Edition

Name: Natural Language Processing in Action, Second Edition
ISBN: 9781617299445

by Cole Howard, Hobson Lane, Maria Dyshel, Hannes Hapke

February 2025

Intermediate to advanced

688 pages

24h 13m

English

Manning Publications

Read now

Unlock full access

Praise for the First Edition
Natural Language Processing in Action, Second Edition
copyright
contents
preface
acknowledgments
about this book
About the technologyAbout this bookWho should read this bookHow this book is organized: A road mapWhere to startAbout the codeliveBook discussion forum
about the authors
about the contributors
about the cover illustration

Part 1 Wordy machines: Vector models of natural language
1 Machines that read and write: A natural language processing overview
1.1 Programming languages vs. NLP1.1.1 Natural language understanding1.1.2 Natural language generation1.1.3 Plumbing it all together for positive-impact AI1.2 The magic of natural language1.2.1 Language and thought1.2.2 Machines that converse1.2.3 The math1.3 Applications1.3.1 Processing programming languages with NLP1.4 Language through a computer’s “eyes”1.4.1 The language of locks1.4.2 Regular expressions1.5 Building a simple chatbot1.5.1 Keyword-based greeting recognizer1.5.2 Pattern-based intent recognition1.5.3 Another way to recognize greetings1.6 A brief overflight of hyperspace1.7 Word order and grammar1.8 A chatbot natural language pipeline1.9 Processing in depth1.10 Natural language IQ1.11 Test yourselfSummary
2 Tokens of thought: Natural language words
2.1 Tokens and tokenization2.1.1 Your tokenizer toolbox2.1.2 The simplest tokenizer2.1.3 Rule-based tokenization2.1.4 SpaCy2.1.5 Finding the fastest word tokenizer2.2 Beyond word tokens2.2.1 WordPiece tokenizers2.3 Improving your vocabulary2.3.1 Extending your vocabulary with n-grams2.3.2 Normalizing your vocabulary2.4 Challenging tokens: Processing logographic languages2.4.1 A complicated picture: Lemmatization and stemming in Chinese2.5 Vectors of tokens2.5.1 One-hot vectors2.5.2 Bag-of-words vectors2.5.3 Why not bag of characters?2.6 Sentiment2.6.1 VADER: A rule-based sentiment analyzer2.6.2 Naive Bayes2.7 Test yourselfSummary
3 Math with words: Term frequency–inverse document frequency vectors
3.1 Bag-of-words vectors3.2 Vectorizing text DataFrame constructor3.2.1 Faster, better, easier token counting3.2.2 Vectorizing your code3.2.3 Vector space TF–IDF (term frequency–inverse document frequency)3.3 Vector distance and similarity3.3.1 Dot product3.4 Counting TF–IDF frequencies3.4.1 Analyzing “this”3.5 Zipf’s law3.6 Inverse document frequency3.6.1 Return of Zipf3.6.2 Relevance ranking3.6.3 Smoothing out the math3.7 Using TF–IDF for your bot3.8 What’s next3.9 Test yourselfSummary
4 Finding meaning in word counts: Semantic analysis
4.1 From word counts to topic scores4.1.1 The limitations of TF–IDF vectors and lemmatization4.1.2 Topic vectors4.1.3 Thought experiment4.1.4 Algorithms for scoring topics4.2 The challenge: Detecting toxicity4.2.1 Linear discriminant analysis classifier4.2.2 Going beyond linear4.3 Reducing dimensions4.3.1 Enter principal component analysis4.3.2 Singular value decomposition4.4 Latent semantic analysis4.4.1 Diving into semantics analysis4.4.2 TruncatedSVD or PCA?4.4.3 How well does LSA perform for toxicity detection?4.4.4 Other ways to reduce dimensions4.5 Latent Dirichlet allocation4.5.1 The LDiA idea4.5.2 LDiA topic model for comments4.5.3 Detecting toxicity with LDiA4.5.4 A fairer comparison: 32 LDiA topics4.6 Distance and similarity4.7 Steering with feedback4.8 Topic vector power4.8.1 Semantic search4.9 Equipping your bot with semantic search4.10 Test yourselfSummary
Part 2 Deeper learning: Neural networks
5 Word brain: Neural networks
5.1 Why neural networks?5.1.1 Neural networks for words5.1.2 Neurons as feature engineers5.1.3 Biological neurons5.1.4 Perceptron5.1.5 A Python perceptron5.2 An example logistic neuron5.2.1 The logistics of clickbait5.2.2 Sex education5.2.3 Pronouns, gender, and sex5.2.4 Sex logistics5.2.5 A sleek, new PyTorch neuron5.3 Skiing down the error slope5.3.1 Off the chair lift, onto the slope: Gradient descent and local minima5.3.2 Shaking things up: Stochastic gradient descent5.4 Test yourselfSummary
6 Reasoning with word embeddings
6.1 This is your brain on words6.2 Applications6.2.1 Search for meaning6.2.2 Combining word embeddings6.2.3 Analogy questions6.2.4 Word2Vec innovation6.2.5 Artificial intelligence relies on embeddings6.3 Word2Vec6.3.1 Analogy reasoning6.3.2 Learning word embeddings6.3.3 Learning meaning without a dictionary6.3.4 Using the gensim.word2vec module6.3.5 Generating your own word vector representations6.4 Word2Vec alternatives6.4.1 GloVe6.4.2 fastText6.4.3 Word2Vec vs. LSA6.4.4 Static vs. contextualized embeddings6.4.5 Visualizing word relationships6.4.6 Making connections6.4.7 Unnatural words6.5 Test yourselfSummary
7 Finding kernels of knowledge in text with CNNs
7.1 Patterns in sequences of words7.2 Convolution7.2.1 Stencils for natural language text7.2.2 A bit more stenciling7.2.3 Correlation vs. convolution7.2.4 Convolution as a mapping function7.2.5 Python convolution example7.2.6 PyTorch 1D CNN on 4D embedding vectors7.2.7 Natural examples7.3 Morse code7.3.1 Decoding Morse with convolution7.4 Building a CNN with PyTorch7.4.1 Clipping and padding7.4.2 Better representation with word embeddings7.4.3 Transfer learning7.4.4 Robustifying your CNN with dropout7.5 PyTorch CNN to process disaster toots7.5.1 Network architecture7.5.2 Pooling7.5.3 Linear layers7.5.4 Getting fit7.5.5 Hyperparameter tuning7.6 Test yourselfSummary
8 Reduce, reuse, and recycle your words: RNNs and LSTMs
8.1 What are RNNs good for?8.1.1 RNN sequence handling8.1.2 RNNs remember everything you tell them8.1.3 RNNs hide their understanding8.1.4 RNNs remember everything you tell them8.2 Predicting nationality with only a last name8.2.1 Building an RNN from scratch8.2.2 Training an RNN, one token at a time8.2.3 Understanding the results8.2.4 Multiclass classifiers vs. multi-label taggers8.3 Backpropagation through time8.3.1 Initializing the hidden layer in an RNN8.4 Remembering with recurrent networks8.4.1 Word-level language models8.4.2 Gated recurrent units8.4.3 Long short-term memory8.4.4 Giving your RNN a tune-up8.5 Predicting8.6 Test yourselfSummary
Part 3 Getting real: Real-world NLP applications
9 Stackable deep learning: Transformers
9.1 Recursion vs. recurrence9.1.1 Attention is not all you need9.1.2 A LEGO set for language9.2 Filling the attention gaps9.2.1 Positional encoding9.2.2 Connecting all the pieces9.2.3 Transformer translation9.3 Bidirectional backpropagation and BERT9.3.1 Tokenization and pretraining9.3.2 Fine-tuning9.3.3 Implementation9.3.4 Fine-tuning a pretrained BERT model for text classification9.4 Test yourselfSummary
10 Large language models in the real world
10.1 Large language models10.1.1 Scaling up10.1.2 Smarter, smaller LLMs10.1.3 Semantic routing and guard rails10.1.4 Red teaming10.2 Generating words with your own LLM10.2.1 Creating your own generative LLM10.2.2 Fine-tuning your generative model10.2.3 Nonsense: Hallucination10.3 Giving LLMs an IQ boost with search10.3.1 Searching for words: Full-text search10.3.2 Searching for meaning: Semantic search10.3.3 Scaling up your semantic search10.3.4 Approximate nearest neighbor search10.3.5 Choosing your index10.3.6 Quantizing the math10.3.7 Pulling it all together with Haystack10.3.8 Getting real10.3.9 A haystack of knowledge10.3.10 Answering questions10.3.11 Combining semantic search with text generation10.3.12 Deploying your app in the cloud10.3.13 Serve your users better10.3.14 AI ethics vs. AI safety10.4 Test yourselfSummary
11 Information extraction and knowledge graphs
11.1 Grounding11.1.1 Going old-fashioned: Information extraction with patterns11.2 First things first: Segmenting your text into sentences11.2.1 Why won’t split('.!?') work?11.2.2 Sentence segmentation with regular expressions11.2.3 Sentence semantics11.3 A knowledge extraction pipeline11.4 Entity recognition11.4.1 Pattern-based entity recognition: Extracting GPS locations11.4.2 Named entity recognition with spaCy11.5 Coreference resolution11.5.1 Coreference resolution with spaCy11.5.2 Entity name normalization11.6 Dependency parsing11.6.1 Constituency parsing with benepar11.7 From dependency parsing to relation extraction11.7.1 Pattern-based relation extraction11.7.2 Neural relation extraction11.8 Building your knowledge base11.8.1 A large knowledge graph11.9 Finding answers in a knowledge graph11.9.1 From questions to queries11.10 Test yourselfSummary
12 Getting chatty with dialog engines
12.1 Chatbots are everywhere12.1.1 Different chatbots, same tools12.1.2 Conversation design12.1.3 Your first conversation diagram12.1.4 What makes a good conversation?12.1.5 Making your chatbot a good listener: Implicit and explicit confirmations12.1.6 Using GUI elements12.2 Making sense of the user’s input: Natural language understanding12.2.1 Intent recognition12.2.2 Multi-label classification12.3 Generating a response12.3.1 Template-based approach12.3.2 Conversation graphs12.3.3 Storing your graph in a relational database12.3.4 Scaling up the content: The search-based approach12.3.5 Designing more complex logic: The programmatic approach12.4 The generative approach12.5 Chatbot frameworks12.5.1 Building an intent-based chatbot with Rasa12.5.2 Adding LLMs to your chatbot with LangChain12.6 Maintaining your chatbot’s design12.7 Evaluating your chatbot12.7.1 Defining your chatbot’s performance metrics12.7.2 Measuring NLU performance12.7.3 Measuring user experience12.7.4 What’s next?12.8 Test yourselfSummary
appendix A Your NLP tools
A.1 Installing nlpia2 in a virtual environmentA.2 Batteries includedA.3 Anaconda3A.4 Installing nlpia2A.5 An integrated development environmentA.6 Debian package managerA.7 MacsA.7.1 A Mac package managerA.7.2 Some useful packagesA.8 Tune-upsA.8.1 Ubuntu tune-upsA.8.2 Mac tune-upsA.9 WindowsA.9.1 ChocolateyA.9.2 Get virtualA.9.3 Installing nlpia2 in a Linux container
appendix B Playful Python and regular expressions
B.1 A playful development environmentB.1.1 Start your REPL enginesB.1.2 Exploring Python landB.2 Working with stringsB.2.1 String types: str and bytesB.2.2 Templates in Python: .format()B.3 Mapping in Python: dict and OrderedDictB.4 Regular expressionB.4.1 The OR operator: |B.4.2 Groups: ()B.4.3 Character classes: []B.5 StyleB.6 Mastery
appendix C Vectors and linear algebra
C.1 VectorsC.1.1 Vector length and normalizationC.1.2 Comparing vectorsC.1.3 Dot productC.2 MatricesC.2.1 Multiplying matricesC.2.2 Matrices as transformationsC.3 TensorsC.4 Diving deeper into linear algebra
appendix D Machine learning tools and techniques
D.1 Change of paradigmD.1.1 Machine learning workflowD.1.2 Problem definitionD.1.3 Data selection and avoiding biasD.2 How fit is “fit”?D.2.1 Knowing is half the battleD.2.2 Cross-validationD.2.3 Imbalanced training setsD.3 Model optimizationD.4 Model evaluation and performance metricsD.4.1 Measuring classifier performanceD.4.2 Measuring regressor performanceD.5 Pro tips
appendix E Deploying NLU containerized microservices
E.1 A multilabel intent classifierE.2 Problem statement and training dataE.3 Microservices architectureE.3.1 Containerizing a microserviceE.3.2 Building a container from a repositoryE.3.3 Scaling an NLU serviceE.4 Running and testing the prediction microserviceE.4.1 Setting up DigitalOcean Spaces and training the modelE.4.2 Downloading the nlu-fastapi container imageE.4.3 Running the containerE.4.4 Interacting with the container API
appendix F Glossary
F.1 AcronymsF.2 Terms
Chapter 1
Chapter 2Chapter 3Chapter 4Chapter 5Chapter 6Chapter 7Chapter 8Chapter 9Chapter 10Chapter 11Chapter 12Appendix AAppendix BAppendix CAppendix DAppendix EAppendix F
index

Content preview from Natural Language Processing in Action, Second Edition

9 Stackable deep learning: Transformers

This chapter covers

Seeing how transformers enable limitless stacking and scaling
Fine-tuning transformers for your application
Applying transformers to extractive and abstraction summarization of long documents
Generating plausible, grammatically correct text with transformers
Estimating the information capacity of a transformer

Transformers are changing the world. The increased intelligence transformers bring to AI is transforming culture, society, and the economy. For the first time, transformers are making us question the long-term economic value of human intelligence and creativity. And the ripple effects of transformers extend beyond just the economy. Transformers are changing not only how ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Publisher Resources

ISBN: 9781617299445Publisher Support Publisher Website

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Natural Language Processing in Action, Second Edition

by Cole Howard, Hobson Lane, Maria Dyshel, Hannes Hapke