Bayesian Analysis in Natural Language Processing, 2nd Edition

Book description

Natural language processing (NLP) went through a profound transformation in the mid-1980s when it shifted to make heavy use of corpora and data-driven techniques to analyze language.

Since then, the use of statistical techniques in NLP has evolved in several ways. One such example of evolution took place in the late 1990s or early 2000s, when full-fledged Bayesian machinery was introduced to NLP. This Bayesian approach to NLP has come to accommodate various shortcomings in the frequentist approach and to enrich it, especially in the unsupervised setting, where statistical learning is done without target prediction examples.

In this book, we cover the methods and algorithms that are needed to fluently read Bayesian learning papers in NLP and to do research in the area. These methods and algorithms are partially borrowed from both machine learning and statistics and are partially developed "in-house" in NLP. We cover inference techniques such as Markov chain Monte Carlo sampling and variational inference, Bayesian estimation, and nonparametric modeling. In response to rapid changes in the field, this second edition of the book includes a new chapter on representation learning and neural networks in the Bayesian context. We also cover fundamental concepts in Bayesian statistics such as prior distributions, conjugacy, and generative modeling. Finally, we review some of the fundamental modeling techniques in NLP, such as grammar modeling, neural networks and representation learning, and their use with Bayesian analysis.

Table of contents

  1. List of Figures
  2. List of Figures
  3. List of Figures
  4. Preface (First Edition)
  5. Acknowledgments (First Edition)
  6. Preface (Second Edition)
  7. Preliminaries
    1. Probability Measures
    2. Random Variables
      1. Continuous and Discrete Random Variables
      2. Joint Distribution over Multiple Random Variables
    3. Conditional Distributions
      1. Bayes' Rule
      2. Independent and Conditionally Independent Random Variables
      3. Exchangeable Random Variables
    4. Expectations of Random Variables
    5. Models
      1. Parametric vs. Nonparametric Models
      2. Inference with Models
      3. Generative Models
      4. Independence Assumptions in Models
      5. Directed Graphical Models
    6. Learning from Data Scenarios
    7. Bayesian and Frequentist Philosophy (Tip of the Iceberg)
    8. Summary
    9. Exercises
  8. Introduction
    1. Overview: Where Bayesian Statistics and NLP Meet
    2. First Example: The Latent Dirichlet Allocation Model
      1. The Dirichlet Distribution
      2. Inference
      3. Summary
    3. Second Example: Bayesian Text Regression
    4. Conclusion and Summary
    5. Exercises
  9. Priors
    1. Conjugate Priors
      1. Conjugate Priors and Normalization Constants
      2. The Use of Conjugate Priors with Latent Variable Models
      3. Mixture of Conjugate Priors
      4. Renormalized Conjugate Distributions
      5. Discussion: To Be or not to Be Conjugate?
      6. Summary
    2. Priors Over Multinomial and Categorical Distributions
      1. The Dirichlet Distribution Re-Visited
      2. The Logistic Normal Distribution (1/2)
      3. The Logistic Normal Distribution (2/2)
      4. Discussion
      5. Summary
    3. Non-Informative Priors
      1. Uniform and Improper Priors
      2. Jeffreys Prior
      3. Discussion
    4. Conjugacy and Exponential Models
    5. Multiple Parameter Draws in Models
    6. Structural Priors
    7. Conclusion and Summary
    8. Exercises
  10. Bayesian Estimation
    1. Learning with Latent Variables: Two Views
    2. Bayesian Point Estimation
      1. Maximum a Posteriori Estimation
      2. Posterior Approximations Based on the MAP Solution
      3. Decision-Theoretic Point Estimation
      4. Discussion and Summary
    3. Empirical Bayes
    4. Asymptotic Behavior of the Posterior
    5. Summary
    6. Exercises
  11. Sampling Methods
    1. MCMC Algorithms: Overview
    2. NLP Model Structure for MCMC Inference
      1. Partitioning the Latent Variables
    3. Gibbs Sampling
      1. Collapsed Gibbs Sampling
      2. Operator View
      3. Parallelizing the Gibbs Sampler
      4. Summary
    4. The Metropolis–Hastings Algorithm
      1. Variants of Metropolis–Hastings
    5. Slice Sampling
      1. Auxiliary Variable Sampling
      2. The Use of Slice Sampling and Auxiliary Variable Sampling in NLP
    6. Simulated Annealing
    7. Convergence of MCMC Algorithms
    8. Markov Chain: Basic Theory
    9. Sampling Algorithms Not in the MCMC Realm
    10. Monte Carlo Integration
    11. Discussion
      1. Computability of Distribution vs. Sampling
      2. Nested MCMC Sampling
      3. Runtime of MCMC Samplers
      4. Particle Filtering
    12. Conclusion and Summary
    13. Exercises
  12. Variational Inference
    1. Variational Bound on Marginal Log-Likelihood
    2. Mean-Field Approximation
    3. Mean-Field Variational Inference Algorithm
      1. Dirichlet-Multinomial Variational Inference
      2. Connection to the Expectation-Maximization Algorithm
    4. Empirical Bayes with Variational Inference
    5. Discussion
      1. Initialization of the Inference Algorithms
      2. Convergence Diagnosis
      3. The Use of Variational Inference for Decoding
      4. Variational Inference as KL Divergence Minimization
      5. Online Variational Inference
    6. Summary
    7. Exercises
  13. Nonparametric Priors
    1. The Dirichlet Process: Three Views
      1. The Stick-Breaking Process
      2. The Chinese Restaurant Process
    2. Dirichlet Process Mixtures
      1. Inference with Dirichlet Process Mixtures
      2. Dirichlet Process Mixture as a Limit of Mixture Models
    3. The Hierarchical Dirichlet Process
    4. The Pitman–Yor Process
      1. Pitman–Yor Process for Language Modeling
      2. Power-Law Behavior of the Pitman–Yor Process
    5. Discussion
      1. Gaussian Processes
      2. The Indian Buffet Process
      3. Nested Chinese Restaurant Process
      4. Distance-Dependent Chinese Restaurant Process
      5. Sequence Memoizers
    6. Summary
    7. Exercises
  14. Bayesian Grammar Models
    1. Bayesian Hidden Markov Models
      1. Hidden Markov Models with an Infinite State Space
    2. Probabilistic Context-Free Grammars
      1. PCFGs as a Collection of Multinomials
      2. Basic Inference Algorithms for PCFGs
      3. Hidden Markov Models as PCFGs
    3. Bayesian Probabilistic Context-Free Grammars
      1. Priors on PCFGs
      2. Monte Carlo Inference with Bayesian PCFGs
      3. Variational Inference with Bayesian PCFGs
    4. Adaptor Grammars
      1. Pitman–Yor Adaptor Grammars
      2. Stick-Breaking View of PYAG
      3. Inference with PYAG
    5. Hierarchical Dirichlet Process PCFGs (HDP-PCFGs)
      1. Extensions to the HDP-PCFG Model
    6. Dependency Grammars
      1. State-Split Nonparametric Dependency Models
    7. Synchronous Grammars
    8. Multilingual Learning
      1. Part-of-Speech Tagging
      2. Grammar Induction
    9. Further Reading
    10. Summary
    11. Exercises
  15. Representation Learning and Neural Networks
    1. Neural Networks and Representation Learning: Why Now?
    2. Word Embeddings
      1. Skip-Gram Models for Word Embeddings
      2. Bayesian Skip-Gram Word Embeddings
      3. Discussion
    3. Neural Networks
      1. Frequentist Estimation and the Backpropagation Algorithm
      2. Priors on Neural Network Weights
    4. Modern Use of Neural Networks in NLP
      1. Recurrent and Recursive Neural Networks
      2. Vanishing and Exploding Gradient Problem
      3. Neural Encoder-Decoder Models
      4. Convolutional Neural Networks
    5. Tuning Neural Networks
      1. Regularization
      2. Hyperparameter Tuning
    6. Generative Modeling with Neural Networks
      1. Variational Autoencoders (1/2)
      2. Variational Autoencoders (2/2)
      3. Generative Adversarial Networks
    7. Conclusion
    8. Exercises
  16. Closing Remarks
  17. Basic Concepts
    1. Basic Concepts in Information Theory
      1. Entropy and Cross Entropy
      2. Kullback–Leibler Divergence
    2. Other Basic Concepts
      1. Jensen's Inequality
      2. The Chain Rule for Differentiation
      3. Transformation of Continuous Random Variables
      4. The Expectation-Maximization Algorithm
    3. Basic Concepts in Optimization
      1. Stochastic Gradient Descent
      2. Constrained Optimization
  18. Distribution Catalog
    1. The Multinomial Distribution
    2. The Dirichlet Distribution
    3. The Poisson Distribution
    4. The Gamma Distribution
    5. The Multivariate Normal Distribution
    6. The Laplace Distribution
    7. The Logistic Normal Distribution
    8. The Inverse Wishart Distribution
    9. The Gumbel Distribution
  19. Bibliography (1/6)
  20. Bibliography (2/6)
  21. Bibliography (3/6)
  22. Bibliography (4/6)
  23. Bibliography (5/6)
  24. Bibliography (6/6)
  25. Author's Biography
  26. Blank Page (1/4)
  27. Blank Page (2/4)
  28. Blank Page (3/4)
  29. Blank Page (4/4)

Product information

  • Title: Bayesian Analysis in Natural Language Processing, 2nd Edition
  • Author(s): Shay Cohen, Graeme Hirst
  • Release date: April 2019
  • Publisher(s): Morgan & Claypool Publishers
  • ISBN: 9781681735276