Applied Natural Language Processing in the Enterprise

Applied Natural Language Processing in the Enterprise

by Ankur A. Patel, Ajay Uppili Arasanipalai
Released May 2021
Publisher(s): O'Reilly Media, Inc.
ISBN: 9781492062578

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Buy on Amazon Buy on ebooks.com
Start your free trial

Book description

NLP has exploded in popularity over the last few years. But while Google, Facebook, OpenAI, and others continue to release larger language models, many teams still struggle with building NLP applications that live up to the hype. This hands-on guide helps you get up to speed on the latest and most promising trends in NLP.

With a basic understanding of machine learning and some Python experience, you'll learn how to build, train, and deploy models for real-world applications in your organization. Authors Ankur Patel and Ajay Uppili Arasanipalai guide you through the process using code and examples that highlight the best practices in modern NLP.

  • Use state-of-the-art NLP models such as BERT and GPT-3 to solve NLP tasks such as named entity recognition, text classification, semantic search, and reading comprehension
  • Train NLP models with performance comparable or superior to that of out-of-the-box systems
  • Learn about Transformer architecture and modern tricks like transfer learning that have taken the NLP world by storm
  • Become familiar with the tools of the trade, including spaCy, Hugging Face, and fast.ai
  • Build core parts of the NLP pipeline--including tokenizers, embeddings, and language models--from scratch using Python and PyTorch
  • Take your models out of Jupyter notebooks and learn how to deploy, monitor, and maintain them in production

Publisher resources

View/Submit Errata

Table of contents

  1. Preface
    1. What Is Natural Language Processing?
    2. Why Should I Read This Book?
    3. What Do I Need to Know Already?
    4. What Is This Book All About?
    5. How Is This Book Organized?
    6. Conventions Used in This Book
    7. Using Code Examples
    8. O’Reilly Online Learning
    9. How to Contact Us
    10. Acknowledgments
      1. Ajay
      2. Ankur
  2. I. Scratching the Surface
  3. 1. Introduction to NLP
    1. What Is NLP?
      1. Popular Applications
      2. History
      3. Inflection Points
      4. A Final Word
    2. Basic NLP
      1. Defining NLP Tasks
      2. Set Up the Programming Environment
      3. spaCy, fast.ai, and Hugging Face
      4. Perform NLP Tasks Using spaCy
    3. Conclusion
  4. 2. Transformers and Transfer Learning
    1. Training with fastai
      1. Using the fastai Library
      2. ULMFiT for Transfer Learning
      3. Fine-Tuning a Language Model on IMDb
      4. Training a Text Classifier
    2. Inference with Hugging Face
      1. Loading Models
      2. Generating Predictions
    3. Conclusion
  5. 3. NLP Tasks and Applications
    1. Pretrained Language Models
    2. Transfer Learning and Fine-Tuning
    3. NLP Tasks
    4. Natural Language Dataset
      1. Explore the AG Dataset
    5. NLP Task #1: Named Entity Recognition
      1. Perform Inference Using the Original spaCy Model
      2. Custom NER
      3. Annotate via Prodigy: NER
      4. Train the Custom NER Model Using spaCy
      5. Custom NER Model Versus Original NER Model
    6. NLP Task #2: Text Classification
      1. Annotate via Prodigy: Text Classification
      2. Train Text Classification Models Using spaCy
    7. Conclusion
  6. II. The Cogs in the Machine
  7. 4. Tokenization
    1. A Minimal Tokenizer
    2. Hugging Face Tokenizers
      1. Subword Tokenization
    3. Building Your Own Tokenizer
    4. Conclusion
  8. 5. Embeddings: How Machines “Understand” Words
    1. Understanding Versus Reading Text
    2. Word Vectors
      1. Word2Vec
      2. Embeddings in the Age of Transfer Learning
    3. Embeddings in Practice
      1. Preprocessing
      2. Model
      3. Training
      4. Validation
    4. Embedding Things That Aren’t Words
      1. Making Vectorized Music
      2. Some General Tips for Making Custom Embeddings
    5. Conclusion
  9. 6. Recurrent Neural Networks and Other Sequence Models
    1. Recurrent Neural Networks
      1. RNNs in PyTorch from Scratch
      2. Bidirectional RNN
      3. Sequence to Sequence Using RNNs
    2. Long Short-Term Memory
    3. Gated Recurrent Units
    4. Conclusion
  10. 7. Transformers
    1. Building a Transformer from Scratch
    2. Attention Mechanisms
      1. Dot Product Attention
      2. Scaled Dot Product Attention
      3. Multi-Head Self-Attention
      4. Adaptive Attention Span
      5. Persistent Memory/All-Attention
      6. Product-Key Memory
    3. Transformers for Computer Vision
    4. Conclusion
  11. 8. BERTology: Putting It All Together
    1. ImageNet
      1. The Power of Pretrained Models
    2. The Path to NLP’s ImageNet Moment
    3. Pretrained Word Embeddings
      1. The Limitations of One-Hot Encoding
      2. Word2Vec
      3. GloVe
      4. fastText
      5. Context-Aware Pretrained Word Embeddings
    4. Sequential Models
      1. Sequential Data and the Importance of Sequential Models
    5. RNNs
      1. Vanilla RNNs
      2. LSTM Networks
      3. GRUs
    6. Attention Mechanisms
    7. Transformers
      1. Transformer-XL
    8. NLP’s ImageNet Moment
      1. Universal Language Model Fine-Tuning
      2. ELMo
      3. BERT
      4. BERTology
      5. GPT-1, GPT-2, GPT-3
    9. Conclusion
  12. III. Outside the Wall
  13. 9. Tools of the Trade
    1. Deep Learning Frameworks
      1. PyTorch
      2. TensorFlow
      3. Jax
      4. Julia
    2. Visualization and Experiment Tracking
      1. TensorBoard
      2. Weights & Biases
      3. Neptune
      4. Comet
      5. MLflow
    3. AutoML
      1. H2O.ai
      2. Dataiku
      3. DataRobot
    4. ML Infrastructure and Compute
      1. Paperspace
      2. FloydHub
      3. Google Colab
      4. Kaggle Kernels
      5. Lambda GPU Cloud
    5. Edge/On-Device Inference
      1. ONNX
      2. Core ML
      3. Edge Accelerators
    6. Cloud Inference and Machine Learning as a Service
      1. AWS
      2. Microsoft Azure
      3. Google Cloud Platform
    7. Continuous Integration and Delivery
    8. Conclusion
  14. 10. Visualization
    1. Our First Streamlit App
      1. Build the Streamlit App
      2. Deploy the Streamlit App
      3. Explore the Streamlit Web App
      4. Build and Deploy a Streamlit App for Custom NER
      5. Build and Deploy a Streamlit App for Text Classification on AG News Dataset
      6. Build and Deploy a Streamlit App for Text Classification on Custom Text
    2. Conclusion
  15. 11. Productionization
    1. Data Scientists, Engineers, and Analysts
      1. Prototyping, Deployment, and Maintenance
      2. Notebooks and Scripts
    2. Databricks: Your Unified Data Analytics Platform
      1. Support for Big Data
      2. Support for Multiple Programming Languages
      3. Support for ML Frameworks
      4. Support for Model Repository, Access Control, Data Lineage, and Versioning
    3. Databricks Setup
      1. Set Up Access to S3 Bucket
      2. Set Up Libraries
      3. Create Cluster
      4. Create Notebook
      5. Enable Init Script and Restart Cluster
      6. Run Speed Test: Inference on NER Using spaCy
    4. Machine Learning Jobs
      1. Production Pipeline Notebook
      2. Scheduled Machine Learning Jobs
      3. Event-Driven Machine Learning Pipeline
    5. MLflow
      1. Log and Register Model
      2. MLflow Model Serving
    6. Alternatives to Databricks
      1. Amazon SageMaker
      2. Saturn Cloud
    7. Conclusion
  16. 12. Conclusion
    1. Ten Final Lessons
      1. Lesson 1: Start with Simple Approaches First
      2. Lesson 2: Leverage the Community
      3. Lesson 3: Do Not Create from Scratch, When Possible
      4. Lesson 4: Intuition and Experience Trounces Theory
      5. Lesson 5: Fight Decision Fatigue
      6. Lesson 6: Data Is King
      7. Lesson 7: Lean on Humans
      8. Lesson 8: Pair Yourself with Really Great Engineers
      9. Lesson 9: Ensemble
      10. Lesson 10: Have Fun
    2. Final Word
  17. A. Scaling
    1. Multi-GPU Training
    2. Distributed Training
    3. What Makes Deep Training Fast?
  18. B. CUDA
    1. Threads and Thread Blocks
    2. Writing CUDA Kernels
    3. CUDA in Practice
  19. Index
  20. About the Authors

Product information

  • Title: Applied Natural Language Processing in the Enterprise
  • Author(s): Ankur A. Patel, Ajay Uppili Arasanipalai
  • Release date: May 2021
  • Publisher(s): O'Reilly Media, Inc.
  • ISBN: 9781492062578