book

Transformers for Natural Language Processing

Name: Transformers for Natural Language Processing
Author: Denis Rothman
ISBN: 9781800565791

by Denis Rothman

January 2021

Intermediate to advanced

384 pages

8h 9m

English

Packt Publishing

Read now

Unlock full access

Preface
Who this book is forWhat this book coversTo get the most out of this bookGet in touch
Getting Started with the Model Architecture of the Transformer
The background of the TransformerThe rise of the Transformer: Attention Is All You NeedThe encoder stackInput embeddingPositional encodingSub-layer 1: Multi-head attentionSub-layer 2: Feedforward networkThe decoder stackOutput embedding and position encodingThe attention layersThe FFN sub-layer, the Post-LN, and the linear layerTraining and performanceBefore we end the chapterSummaryQuestionsReferences
Fine-Tuning BERT Models
The architecture of BERTThe encoder stackPreparing the pretraining input environmentPretraining and fine-tuning a BERT modelFine-tuning BERTActivating the GPUInstalling the Hugging Face PyTorch interface for BERTImporting the modulesSpecifying CUDA as the device for torchLoading the datasetCreating sentences, label lists, and adding BERT tokensActivating the BERT tokenizerProcessing the dataCreating attention masksSplitting data into training and validation setsConverting all the data into torch tensorsSelecting a batch size and creating an iteratorBERT model configurationLoading the Hugging Face BERT uncased base modelOptimizer grouped parametersThe hyperparameters for the training loopThe training loopTraining evaluationPredicting and evaluating using the holdout datasetEvaluating using Matthews Correlation CoefficientThe score of individual batchesMatthews evaluation for the whole datasetSummaryQuestionsReferences
Pretraining a RoBERTa Model from Scratch
Training a tokenizer and pretraining a transformerBuilding KantaiBERT from scratchStep 1: Loading the datasetStep 2: Installing Hugging Face transformersStep 3: Training a tokenizerStep 4: Saving the files to diskStep 5: Loading the trained tokenizer filesStep 6: Checking resource constraints: GPU and CUDAStep 7: Defining the configuration of the modelStep 8: Reloading the tokenizer in transformersStep 9: Initializing a model from scratchExploring the parametersStep 10: Building the datasetStep 11: Defining a data collatorStep 12: Initializing the trainerStep 13: Pretraining the modelStep 14: Saving the final model (+tokenizer + config) to diskStep 15: Language modeling with FillMaskPipelineNext stepsSummaryQuestionsReferences
Downstream NLP Tasks with Transformers
Transduction and the inductive inheritance of transformersThe human intelligence stackThe machine intelligence stackTransformer performances versus Human BaselinesEvaluating models with metricsAccuracy scoreF1-scoreMatthews Correlation Coefficient (MCC)Benchmark tasks and datasetsFrom GLUE to SuperGLUEIntroducing higher Human Baseline standardsThe SuperGLUE evaluation processDefining the SuperGLUE benchmark tasksBoolQCommitment Bank (CB)Multi-Sentence Reading Comprehension (MultiRC)Reading Comprehension with Commonsense Reasoning Dataset (ReCoRD)Recognizing Textual Entailment (RTE)Words in Context (WiC)The Winograd Schema Challenge (WSC)Running downstream tasksThe Corpus of Linguistic Acceptability (CoLA)Stanford Sentiment TreeBank (SST-2)Microsoft Research Paraphrase Corpus (MRPC)Winograd schemasSummaryQuestionsReferences
Machine Translation with the Transformer
Defining machine translationHuman transductions and translationsMachine transductions and translationsPreprocessing a WMT datasetPreprocessing the raw dataFinalizing the preprocessing of the datasetsEvaluating machine translation with BLEUGeometric evaluationsApplying a smoothing techniqueChencherry smoothingTranslations with TraxInstalling TraxCreating a Transformer modelInitializing the model using pretrained weightsTokenizing a sentenceDecoding from the TransformerDe-tokenizing and displaying the translationSummaryQuestionsReferences
Text Generation with OpenAI GPT-2 and GPT-3 Models
The rise of billion-parameter transformer modelsThe increasing size of transformer modelsContext size and maximum path lengthTransformers, reformers, PET, or GPT?The limits of the original Transformer architectureRunning BertVizThe ReformerPattern-Exploiting Training (PET)The philosophy of Pattern-Exploiting Training (PET)It's time to make a decisionThe architecture of OpenAI GPT modelsFrom fine-tuning to zero-shot modelsStacking decoder layersText completion with GPT-2Step 1: Activating the GPUStep 2: Cloning the OpenAI GPT-2 repositoryStep 3: Installing the requirementsStep 4: Checking the version of TensorFlowStep 5: Downloading the 345M parameter GPT-2 modelSteps 6-7: Intermediate instructionsSteps 7b-8: Importing and defining the modelStep 9: Interacting with GPT-2Training a GPT-2 language modelStep 1: PrerequisitesSteps 2 to 6: Initial steps of the training processStep 7: The N Shepperd training filesStep 8: Encoding the datasetStep 9: Training the modelStep 10: Creating a training model directoryContext and completion examplesGenerating music with transformersSummaryQuestionsReferences
Applying Transformers to Legal and Financial Documents for AI Text Summarization
Designing a universal text-to-text modelThe rise of text-to-text transformer modelsA prefix instead of task-specific formatsThe T5 modelText summarization with T5Hugging FaceHugging Face transformer resourcesInitializing the T5-large transformer modelGetting started with T5Exploring the architecture of the T5 modelSummarizing documents with T5-largeCreating a summarization functionA general topic sampleThe Bill of Rights sampleA corporate law sampleSummaryQuestionsReferences
Matching Tokenizers and Datasets
Matching datasets and tokenizersBest practicesStep 1: PreprocessingStep 2: Post-processingContinuous human quality controlWord2Vec tokenizationCase 0: Words in the dataset and the dictionaryCase 1: Words not in the dataset or the dictionaryCase 2: Noisy relationshipsCase 3: Rare wordsCase 4: Replacing rare wordsCase 5: EntailmentStandard NLP tasks with specific vocabularyGenerating unconditional samples with GPT-2Controlling tokenized dataGenerating trained conditional samplesT5 Bill of Rights SampleSummarizing the Bill of Rights, version 1Summarizing the Bill of Rights, version 2SummaryQuestionsReferences
Semantic Role Labeling with BERT-Based Transformers
Getting started with SRLDefining Semantic Role LabelingVisualizing SRLRunning a pretrained BERT-based modelThe architecture of the BERT-based modelSetting up the BERT SRL environmentSRL experiments with the BERT-based modelBasic samplesSample 1Sample 2Sample 3Difficult samplesSample 4Sample 5Sample 6SummaryQuestionsReferences

Let Your Data Do the Talking: Story, Questions, and Answers
MethodologyTransformers and methodsMethod 0: Trial and errorMethod 1: NER firstUsing NER to find questionsLocation entity questionsPerson entity questionsMethod 2: SRL firstQuestion-answering with ELECTRAProject management constraintsUsing SRL to find questionsNext stepsExploring Haystack with a RoBERTa modelSummaryQuestionsReferences
Detecting Customer Emotions to Make Predictions
Getting started: Sentiment analysis transformersThe Stanford Sentiment Treebank (SST)Sentiment analysis with RoBERTa-largePredicting customer behavior with sentiment analysisSentiment analysis with DistilBERTSentiment analysis with Hugging Face's models listDistilBERT for SSTMiniLM-L12-H384-uncasedRoBERTa-large-mnliBERT-base multilingual modelSummaryQuestionsReferences
Analyzing Fake News with Transformers
Emotional reactions to fake newsCognitive dissonance triggers emotional reactionsAnalyzing a conflictual TweetBehavioral representation of fake newsA rational approach to fake newsDefining a fake news resolution roadmapGun controlSentiment analysisNamed entity recognition (NER)Semantic Role Labeling (SRL)Reference sitesCOVID-19 and former President Trump's TweetsSemantic Role Labeling (SRL)Before we goLooking for the silver bulletLooking for reliable training methodsSummaryQuestionsReferences
Appendix: Answers to the Questions
Chapter 1, Getting Started with the Model Architecture of the TransformerChapter 2, Fine-Tuning BERT ModelsChapter 3, Pretraining a RoBERTa Model from ScratchChapter 4, Downstream NLP Tasks with TransformersChapter 5, Machine Translation with the TransformerChapter 6, Text Generation with OpenAI GPT-2 and GPT-3 ModelsChapter 7, Applying Transformers to Legal and Financial Documents for AI Text SummarizationChapter 8, Matching Tokenizers and DatasetsChapter 9, Semantic Role Labeling with BERT-Based TransformersChapter 10, Let Your Data Do the Talking: Story, Questions, and AnswersChapter 11, Detecting Customer Emotions to Make PredictionsChapter 12, Analyzing Fake News with Transformers
Other Books You May Enjoy
Index

Overview

Discover the revolutionary transformer architectures that are shaping the future of Natural Language Processing (NLP). This comprehensive book provides a deep dive into state-of-the-art language models such as GPT-2, BERT, and T5, offering practical implementation guidance using Python and popular frameworks like PyTorch and TensorFlow. Master next-generation NLP techniques with hands-on examples and real-world applications.

What this Book will help me do

Implement transformer models including BERT, GPT-2, and T5 for NLP tasks.
Leverage Python frameworks such as PyTorch and TensorFlow for advanced language modeling.
Execute sentiment analysis, text summarization, and speech recognition projects.
Optimize transformer-based solutions for specific datasets and scenarios.
Understand the principles and cognitive insights behind transformer architectures.

Author(s)

Denis Rothman is a pioneer in the field of AI and NLP with years of experience in designing and deploying machine learning systems. As a passionate educator, he brings cutting-edge technical concepts to readers in an accessible and engaging manner. Drawing on his expertise, Denis has authored several influential works in the AI domain, bridging complex theories with practical applications.

Who is it for?

This book is designed for intermediate to advanced learners in the field of deep learning and NLP who wish to gain a better understanding of transformer architectures and their applications. Ideal readers are data scientists, AI practitioners, and software engineers with some experience in Python and Neural Networks, aiming to enhance their skills in modern NLP technologies. If you're familiar with frameworks like PyTorch or TensorFlow and seek to expand your knowledge of state-of-the-art models, this book is for you.

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Transformers for Natural Language Processing - Second Edition

Publisher Resources

ISBN: 9781800565791

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills