book

Natural Language Processing with TensorFlow - Second Edition

Name: Natural Language Processing with TensorFlow - Second Edition
Author: Thushan Ganegedara
ISBN: 9781838641351

by Thushan Ganegedara

July 2022

Intermediate to advanced

514 pages

12h 24m

English

Packt Publishing

Read now

Unlock full access

Preface
Who this book is forWhat this book coversTo get the most out of this bookGet in touch
Introduction to Natural Language Processing
What is Natural Language Processing?Tasks of Natural Language ProcessingThe traditional approach to Natural Language ProcessingUnderstanding the traditional approachExample – generating football game summariesDrawbacks of the traditional approachThe deep learning approach to Natural Language ProcessingHistory of deep learningThe current state of deep learning and NLPUnderstanding a simple deep model – a fully connected neural networkIntroduction to the technical toolsDescription of the toolsInstalling Anaconda and PythonCreating a Conda environmentTensorFlow (GPU) software requirementsAccessing Jupyter NotebookVerifying the TensorFlow installationSummary
Understanding TensorFlow 2
What is TensorFlow?Getting started with TensorFlow 2TensorFlow 2 architecture – What happens during graph build?TensorFlow architecture – what happens when you execute the graph?Café Le TensorFlow 2 – understanding TensorFlow 2 with an analogyFlashback: TensorFlow 1Inputs, variables, outputs, and operationsDefining inputs in TensorFlowFeeding data as NumPy arraysFeeding data as tensorsBuilding a data pipeline using the tf.data APIDefining variables in TensorFlowDefining outputs in TensorFlowDefining operations in TensorFlowComparison operationsMathematical operationsUpdating (scattering) values in tensorsCollecting (gathering) values from a tensorNeural network-related operationsNonlinear activations used by neural networksThe convolution operationThe pooling operationDefining lossKeras: The model building API of TensorFlowSequential APIFunctional APISub-classing APIImplementing our first neural networkPreparing the dataImplementing the neural network with KerasTraining the modelTesting the modelSummary
Word2vec – Learning Word Embeddings
What is a word representation or meaning?Classical approaches to learning word representationOne-hot encoded representationThe TF-IDF methodCo-occurrence matrixAn intuitive understanding of Word2vec – an approach to learning word representationExercise: does queen = king – he + she?The skip-gram algorithmFrom raw text to semi-structured textUnderstanding the skip-gram algorithmImplementing and running the skip-gram algorithm with TensorFlowImplementing the data generators with TensorFlowImplementing the skip-gram architecture with TensorFlowTraining and evaluating the modelThe Continuous Bag-of-Words algorithmGenerating data for the CBOW algorithmImplementing CBOW in TensorFlowTraining and evaluating the modelSummary
Advanced Word Vector Algorithms
GloVe – Global Vectors representationUnderstanding GloVeImplementing GloVeGenerating data for GloVeTraining and evaluating GloVeELMo – Taking ambiguities out of word vectorsDownloading ELMo from TensorFlow HubPreparing inputs for ELMoGenerating embeddings with ELMoDocument classification with ELMoDatasetGenerating document embeddingsClassifying documents with document embeddingsSummary
Sentence Classification with Convolutional Neural Networks
Introducing CNNsCNN fundamentalsThe power of CNNsUnderstanding CNNsConvolution operationStandard convolution operationConvolving with strideConvolving with paddingTransposed convolutionPooling operationMax poolingMax pooling with strideAverage poolingFully connected layersPutting everything togetherExercise – image classification on Fashion-MNIST with CNNAbout the dataDownloading and exploring the dataImplementing the CNNAnalyzing the predictions produced with a CNNUsing CNNs for sentence classificationHow data is transformed for sentence classificationImplementation – downloading and preparing dataImplementation – building a tokenizerThe sentence classification CNN modelThe convolution operationPooling over timeImplementation – sentence classification with CNNsTraining the model Summary
Recurrent Neural Networks
Understanding RNNsThe problem with feed-forward neural networksModeling with RNNsTechnical description of an RNNBackpropagation Through TimeHow backpropagation worksWhy we cannot use BP directly for RNNsBackpropagation Through Time – training RNNsTruncated BPTT – training RNNs efficientlyLimitations of BPTT – vanishing and exploding gradientsApplications of RNNsOne-to-one RNNsOne-to-many RNNsMany-to-one RNNsMany-to-many RNNsNamed Entity Recognition with RNNsUnderstanding the dataProcessing dataDefining hyperparametersDefining the modelIntroduction to the TextVectorization layerDefining the rest of the modelEvaluation metrics and the loss functionTraining and evaluating RNN on NER taskVisually analyzing outputsNER with character and token embeddingsUsing convolution to generate token embeddingsImplementing the new NER modelDefining hyperparametersDefining the input layerDefining the token-based TextVectorization layerDefining the character-based TextVectorization layerProcessing the inputs for the char_vectorize_layerPerforming convolution on the character embeddingsModel training and evaluationOther improvements you can makeSummary
Understanding Long Short-Term Memory Networks
Understanding Long Short-Term Memory NetworksWhat is an LSTM?LSTMs in more detailHow LSTMs differ from standard RNNsHow LSTMs solve the vanishing gradient problemImproving LSTMsGreedy samplingBeam searchUsing word vectorsBidirectional LSTMs (BiLSTMs)Other variants of LSTMsPeephole connectionsGated Recurrent UnitsSummary
Applications of LSTM – Generating Text
Our dataAbout the datasetGenerating training, validation, and test setsAnalyzing the vocabulary sizeDefining the tf.data pipelineImplementing the language modelDefining the TextVectorization layerDefining the LSTM modelDefining metrics and compiling the modelTraining the modelDefining the inference modelGenerating new text with the modelComparing LSTMs to LSTMs with peephole connections and GRUsStandard LSTMReviewGated Recurrent Units (GRUs)ReviewThe modelLSTMs with peepholesReviewThe codeTraining and validation perplexities over timeImproving sequential models – beam searchImplementing beam searchGenerating text with beam searchImproving LSTMs – generating text with words instead of n-gramsThe curse of dimensionalityWord2vec to the rescueGenerating text with Word2vecSummary
Sequence-to-Sequence Learning – Neural Machine Translation
Machine translationA brief historical tour of machine translationRule-based translationStatistical Machine Translation (SMT)Neural Machine Translation (NMT)Understanding neural machine translationIntuition behind NMT systemsNMT architectureThe embedding layerThe encoderThe context vectorThe decoderPreparing data for the NMT systemThe datasetAdding special tokensSplitting training, validation, and testing datasetsDefining sequence lengths for the two languagesPadding the sentencesDefining the modelConverting tokens to IDsDefining the encoderDefining the decoderAttention: Analyzing the encoder statesComputing AttentionImplementing AttentionDefining the final modelTraining the NMTThe BLEU score – evaluating the machine translation systemsModified precisionBrevity penaltyThe final BLEU scoreVisualizing Attention patternsInference with NMTOther applications of Seq2Seq models – chatbotsTraining a chatbotEvaluating chatbots – the Turing testSummary

Transformers
Transformer architectureThe encoder and the decoderComputing the output of the self-attention layerEmbedding layers in the TransformerResiduals and normalizationUnderstanding BERTInput processing for BERTTasks solved by BERTHow BERT is pre-trainedMasked Language Modeling (MLM)Next Sentence Prediction (NSP)Use case: Using BERT to answer questionsIntroduction to the Hugging Face transformers libraryExploring the dataImplementing BERTImplementing and using the TokenizerDefining a TensorFlow datasetBERT for answering questionsDefining the config and the modelTraining and evaluating the modelAnswering questions with BertSummary
Image Captioning with Transformers
Getting to know the dataILSVRC ImageNet datasetThe MS-COCO datasetDownloading the dataProcessing and tokenizing dataPreprocessing dataTokenizing dataDefining a tf.data.DatasetThe machine learning pipeline for image caption generationVision Transformer (ViT)Text-based decoder TransformerPutting everything togetherImplementing the model with TensorFlowImplementing the ViT modelImplementing the text-based decoderDefining the self-attention layerDefining the Transformer layerDefining the full decoderTraining the modelEvaluating the results quantitativelyBLEUROUGEMETEORCIDErEvaluating the modelCaptions generated for test imagesSummary
Appendix A: Mathematical Foundations and Advanced TensorFlow
Basic data structuresScalarVectorsMatricesIndexing of a matrixSpecial types of matricesIdentity matrixSquare diagonal matrixTensorsTensor/matrix operationsTransposeMatrix multiplicationElement-wise multiplicationInverseFinding the matrix inverse – Singular Value Decomposition (SVD)NormsDeterminantProbabilityRandom variablesDiscrete random variablesContinuous random variablesThe probability mass/density functionConditional probabilityJoint probabilityMarginal probabilityBayes’ ruleVisualizing word embeddings with TensorBoardStarting TensorBoardSaving word embeddings and visualizing via TensorBoardSummary
Other Books You May Enjoy
Index

Content preview from Natural Language Processing with TensorFlow - Second Edition

10 Transformers

Transformer models changed the playing field for most machine learning problems that involve sequential data. They have advanced the state of the art by a significant margin compared to the previous leaders, RNN-based models. One of the primary reasons that the Transformer model is so performant is that it has access to the whole sequence of items (e.g. sequence of tokens), as opposed to RNN-based models, which look at one item at a time. The term Transformer has come up several times in our conversations as a method that has outperformed other sequential models such as LSTMs and GRUs. Now, we will learn more about Transformer models.

In this chapter, we will first learn about the Transformer model in detail. Then we will discuss ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Advanced Natural Language Processing with TensorFlow 2

Publisher Resources

ISBN: 9781838641351

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Natural Language Processing with TensorFlow - Second Edition

by Thushan Ganegedara

10

Transformers

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.