book

Applied Natural Language Processing in the Enterprise

by Ankur A. Patel, Ajay Uppili Arasanipalai

May 2021

Beginner to intermediate

333 pages

8h 45m

English

O'Reilly Media, Inc.

Read now

Unlock full access

Preface
What Is Natural Language Processing?Why Should I Read This Book?What Do I Need to Know Already?What Is This Book All About?How Is This Book Organized?Conventions Used in This BookUsing Code ExamplesO’Reilly Online LearningHow to Contact UsAcknowledgmentsAjayAnkur
I. Scratching the Surface
1. Introduction to NLP
What Is NLP?Popular ApplicationsHistoryInflection PointsA Final WordBasic NLPDefining NLP TasksSet Up the Programming EnvironmentspaCy, fast.ai, and Hugging FacePerform NLP Tasks Using spaCyConclusion
2. Transformers and Transfer Learning
Training with fastaiUsing the fastai LibraryULMFiT for Transfer LearningFine-Tuning a Language Model on IMDbTraining a Text ClassifierInference with Hugging FaceLoading ModelsGenerating PredictionsConclusion
3. NLP Tasks and Applications
Pretrained Language ModelsTransfer Learning and Fine-TuningNLP TasksNatural Language DatasetExplore the AG DatasetNLP Task #1: Named Entity RecognitionPerform Inference Using the Original spaCy ModelCustom NERAnnotate via Prodigy: NERTrain the Custom NER Model Using spaCyCustom NER Model Versus Original NER ModelNLP Task #2: Text ClassificationAnnotate via Prodigy: Text ClassificationTrain Text Classification Models Using spaCyConclusion
II. The Cogs in the Machine
4. Tokenization
A Minimal TokenizerHugging Face TokenizersSubword TokenizationBuilding Your Own TokenizerConclusion
5. Embeddings: How Machines “Understand” Words
Understanding Versus Reading TextWord VectorsWord2VecEmbeddings in the Age of Transfer LearningEmbeddings in PracticePreprocessingModelTrainingValidationEmbedding Things That Aren’t WordsMaking Vectorized MusicSome General Tips for Making Custom EmbeddingsConclusion
6. Recurrent Neural Networks and Other Sequence Models
Recurrent Neural NetworksRNNs in PyTorch from ScratchBidirectional RNNSequence to Sequence Using RNNsLong Short-Term MemoryGated Recurrent UnitsConclusion
7. Transformers
Building a Transformer from ScratchAttention MechanismsDot Product AttentionScaled Dot Product AttentionMulti-Head Self-AttentionAdaptive Attention SpanPersistent Memory/All-AttentionProduct-Key MemoryTransformers for Computer VisionConclusion

8. BERTology: Putting It All Together
ImageNetThe Power of Pretrained ModelsThe Path to NLP’s ImageNet MomentPretrained Word EmbeddingsThe Limitations of One-Hot EncodingWord2VecGloVefastTextContext-Aware Pretrained Word EmbeddingsSequential ModelsSequential Data and the Importance of Sequential ModelsRNNsVanilla RNNsLSTM NetworksGRUsAttention MechanismsTransformersTransformer-XLNLP’s ImageNet MomentUniversal Language Model Fine-TuningELMoBERTBERTologyGPT-1, GPT-2, GPT-3Conclusion
III. Outside the Wall
9. Tools of the Trade
Deep Learning FrameworksPyTorchTensorFlowJaxJuliaVisualization and Experiment TrackingTensorBoardWeights & BiasesNeptuneCometMLflowAutoMLH2O.aiDataikuDataRobotML Infrastructure and ComputePaperspaceFloydHubGoogle ColabKaggle KernelsLambda GPU CloudEdge/On-Device InferenceONNXCore MLEdge AcceleratorsCloud Inference and Machine Learning as a ServiceAWSMicrosoft AzureGoogle Cloud PlatformContinuous Integration and DeliveryConclusion
10. Visualization
Our First Streamlit AppBuild the Streamlit AppDeploy the Streamlit AppExplore the Streamlit Web AppBuild and Deploy a Streamlit App for Custom NERBuild and Deploy a Streamlit App for Text Classification on AG News DatasetBuild and Deploy a Streamlit App for Text Classification on Custom TextConclusion
11. Productionization
Data Scientists, Engineers, and AnalystsPrototyping, Deployment, and MaintenanceNotebooks and ScriptsDatabricks: Your Unified Data Analytics PlatformSupport for Big DataSupport for Multiple Programming LanguagesSupport for ML FrameworksSupport for Model Repository, Access Control, Data Lineage, and VersioningDatabricks SetupSet Up Access to S3 BucketSet Up LibrariesCreate ClusterCreate NotebookEnable Init Script and Restart ClusterRun Speed Test: Inference on NER Using spaCyMachine Learning JobsProduction Pipeline NotebookScheduled Machine Learning JobsEvent-Driven Machine Learning PipelineMLflowLog and Register ModelMLflow Model ServingAlternatives to DatabricksAmazon SageMakerSaturn CloudConclusion
12. Conclusion
Ten Final LessonsLesson 1: Start with Simple Approaches FirstLesson 2: Leverage the CommunityLesson 3: Do Not Create from Scratch, When PossibleLesson 4: Intuition and Experience Trounces TheoryLesson 5: Fight Decision FatigueLesson 6: Data Is KingLesson 7: Lean on HumansLesson 8: Pair Yourself with Really Great EngineersLesson 9: EnsembleLesson 10: Have FunFinal Word
A. Scaling
Multi-GPU TrainingDistributed TrainingWhat Makes Deep Training Fast?
B. CUDA
Threads and Thread BlocksWriting CUDA KernelsCUDA in Practice
Index
About the Authors

Content preview from Applied Natural Language Processing in the Enterprise

Part II. The Cogs in the Machine

With a high-level understanding of what NLP is all about, it’s time to start diving into the details to understand how exactly modern NLP works from first principles.

In this section, we’ll cover the following, in order:

Tokenizers
Embeddings
Recurrent neural networks (RNNs)
Transformers
Transfer learning (part 2)

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Publisher Resources

ISBN: 9781492062561Errata Page

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Applied Natural Language Processing in the Enterprise

by Ankur A. Patel, Ajay Uppili Arasanipalai

Part II. The Cogs in the Machine

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.