book

Transformers for Natural Language Processing

Name: Transformers for Natural Language Processing
Author: Denis Rothman
ISBN: 9781800565791

by Denis Rothman

January 2021

Intermediate to advanced

384 pages

8h 9m

English

Packt Publishing

Read now

Unlock full access

Preface
Who this book is forWhat this book coversTo get the most out of this bookGet in touch
Getting Started with the Model Architecture of the Transformer
The background of the TransformerThe rise of the Transformer: Attention Is All You NeedThe encoder stackInput embeddingPositional encodingSub-layer 1: Multi-head attentionSub-layer 2: Feedforward networkThe decoder stackOutput embedding and position encodingThe attention layersThe FFN sub-layer, the Post-LN, and the linear layerTraining and performanceBefore we end the chapterSummaryQuestionsReferences
Fine-Tuning BERT Models
The architecture of BERTThe encoder stackPreparing the pretraining input environmentPretraining and fine-tuning a BERT modelFine-tuning BERTActivating the GPUInstalling the Hugging Face PyTorch interface for BERTImporting the modulesSpecifying CUDA as the device for torchLoading the datasetCreating sentences, label lists, and adding BERT tokensActivating the BERT tokenizerProcessing the dataCreating attention masksSplitting data into training and validation setsConverting all the data into torch tensorsSelecting a batch size and creating an iteratorBERT model configurationLoading the Hugging Face BERT uncased base modelOptimizer grouped parametersThe hyperparameters for the training loopThe training loopTraining evaluationPredicting and evaluating using the holdout datasetEvaluating using Matthews Correlation CoefficientThe score of individual batchesMatthews evaluation for the whole datasetSummaryQuestionsReferences
Pretraining a RoBERTa Model from Scratch
Training a tokenizer and pretraining a transformerBuilding KantaiBERT from scratchStep 1: Loading the datasetStep 2: Installing Hugging Face transformersStep 3: Training a tokenizerStep 4: Saving the files to diskStep 5: Loading the trained tokenizer filesStep 6: Checking resource constraints: GPU and CUDAStep 7: Defining the configuration of the modelStep 8: Reloading the tokenizer in transformersStep 9: Initializing a model from scratchExploring the parametersStep 10: Building the datasetStep 11: Defining a data collatorStep 12: Initializing the trainerStep 13: Pretraining the modelStep 14: Saving the final model (+tokenizer + config) to diskStep 15: Language modeling with FillMaskPipelineNext stepsSummaryQuestionsReferences
Downstream NLP Tasks with Transformers
Transduction and the inductive inheritance of transformersThe human intelligence stackThe machine intelligence stackTransformer performances versus Human BaselinesEvaluating models with metricsAccuracy scoreF1-scoreMatthews Correlation Coefficient (MCC)Benchmark tasks and datasetsFrom GLUE to SuperGLUEIntroducing higher Human Baseline standardsThe SuperGLUE evaluation processDefining the SuperGLUE benchmark tasksBoolQCommitment Bank (CB)Multi-Sentence Reading Comprehension (MultiRC)Reading Comprehension with Commonsense Reasoning Dataset (ReCoRD)Recognizing Textual Entailment (RTE)Words in Context (WiC)The Winograd Schema Challenge (WSC)Running downstream tasksThe Corpus of Linguistic Acceptability (CoLA)Stanford Sentiment TreeBank (SST-2)Microsoft Research Paraphrase Corpus (MRPC)Winograd schemasSummaryQuestionsReferences
Machine Translation with the Transformer
Defining machine translationHuman transductions and translationsMachine transductions and translationsPreprocessing a WMT datasetPreprocessing the raw dataFinalizing the preprocessing of the datasetsEvaluating machine translation with BLEUGeometric evaluationsApplying a smoothing techniqueChencherry smoothingTranslations with TraxInstalling TraxCreating a Transformer modelInitializing the model using pretrained weightsTokenizing a sentenceDecoding from the TransformerDe-tokenizing and displaying the translationSummaryQuestionsReferences
Text Generation with OpenAI GPT-2 and GPT-3 Models
The rise of billion-parameter transformer modelsThe increasing size of transformer modelsContext size and maximum path lengthTransformers, reformers, PET, or GPT?The limits of the original Transformer architectureRunning BertVizThe ReformerPattern-Exploiting Training (PET)The philosophy of Pattern-Exploiting Training (PET)It's time to make a decisionThe architecture of OpenAI GPT modelsFrom fine-tuning to zero-shot modelsStacking decoder layersText completion with GPT-2Step 1: Activating the GPUStep 2: Cloning the OpenAI GPT-2 repositoryStep 3: Installing the requirementsStep 4: Checking the version of TensorFlowStep 5: Downloading the 345M parameter GPT-2 modelSteps 6-7: Intermediate instructionsSteps 7b-8: Importing and defining the modelStep 9: Interacting with GPT-2Training a GPT-2 language modelStep 1: PrerequisitesSteps 2 to 6: Initial steps of the training processStep 7: The N Shepperd training filesStep 8: Encoding the datasetStep 9: Training the modelStep 10: Creating a training model directoryContext and completion examplesGenerating music with transformersSummaryQuestionsReferences
Applying Transformers to Legal and Financial Documents for AI Text Summarization
Designing a universal text-to-text modelThe rise of text-to-text transformer modelsA prefix instead of task-specific formatsThe T5 modelText summarization with T5Hugging FaceHugging Face transformer resourcesInitializing the T5-large transformer modelGetting started with T5Exploring the architecture of the T5 modelSummarizing documents with T5-largeCreating a summarization functionA general topic sampleThe Bill of Rights sampleA corporate law sampleSummaryQuestionsReferences
Matching Tokenizers and Datasets
Matching datasets and tokenizersBest practicesStep 1: PreprocessingStep 2: Post-processingContinuous human quality controlWord2Vec tokenizationCase 0: Words in the dataset and the dictionaryCase 1: Words not in the dataset or the dictionaryCase 2: Noisy relationshipsCase 3: Rare wordsCase 4: Replacing rare wordsCase 5: EntailmentStandard NLP tasks with specific vocabularyGenerating unconditional samples with GPT-2Controlling tokenized dataGenerating trained conditional samplesT5 Bill of Rights SampleSummarizing the Bill of Rights, version 1Summarizing the Bill of Rights, version 2SummaryQuestionsReferences
Semantic Role Labeling with BERT-Based Transformers
Getting started with SRLDefining Semantic Role LabelingVisualizing SRLRunning a pretrained BERT-based modelThe architecture of the BERT-based modelSetting up the BERT SRL environmentSRL experiments with the BERT-based modelBasic samplesSample 1Sample 2Sample 3Difficult samplesSample 4Sample 5Sample 6SummaryQuestionsReferences

Let Your Data Do the Talking: Story, Questions, and Answers
MethodologyTransformers and methodsMethod 0: Trial and errorMethod 1: NER firstUsing NER to find questionsLocation entity questionsPerson entity questionsMethod 2: SRL firstQuestion-answering with ELECTRAProject management constraintsUsing SRL to find questionsNext stepsExploring Haystack with a RoBERTa modelSummaryQuestionsReferences
Detecting Customer Emotions to Make Predictions
Getting started: Sentiment analysis transformersThe Stanford Sentiment Treebank (SST)Sentiment analysis with RoBERTa-largePredicting customer behavior with sentiment analysisSentiment analysis with DistilBERTSentiment analysis with Hugging Face's models listDistilBERT for SSTMiniLM-L12-H384-uncasedRoBERTa-large-mnliBERT-base multilingual modelSummaryQuestionsReferences
Analyzing Fake News with Transformers
Emotional reactions to fake newsCognitive dissonance triggers emotional reactionsAnalyzing a conflictual TweetBehavioral representation of fake newsA rational approach to fake newsDefining a fake news resolution roadmapGun controlSentiment analysisNamed entity recognition (NER)Semantic Role Labeling (SRL)Reference sitesCOVID-19 and former President Trump's TweetsSemantic Role Labeling (SRL)Before we goLooking for the silver bulletLooking for reliable training methodsSummaryQuestionsReferences
Appendix: Answers to the Questions
Chapter 1, Getting Started with the Model Architecture of the TransformerChapter 2, Fine-Tuning BERT ModelsChapter 3, Pretraining a RoBERTa Model from ScratchChapter 4, Downstream NLP Tasks with TransformersChapter 5, Machine Translation with the TransformerChapter 6, Text Generation with OpenAI GPT-2 and GPT-3 ModelsChapter 7, Applying Transformers to Legal and Financial Documents for AI Text SummarizationChapter 8, Matching Tokenizers and DatasetsChapter 9, Semantic Role Labeling with BERT-Based TransformersChapter 10, Let Your Data Do the Talking: Story, Questions, and AnswersChapter 11, Detecting Customer Emotions to Make PredictionsChapter 12, Analyzing Fake News with Transformers
Other Books You May Enjoy
Index

Content preview from Transformers for Natural Language Processing

6 Text Generation with OpenAI GPT-2 and GPT-3 Models

In 2020, Brown et al. (2020) described the training of an OpenAI GPT-3 model with 175 billion parameters trained with approximately one trillion words in 50 petaflop/s days. This represents about 50*1020 operations per day for 400 billion byte-pair-encoded tokens. At the same time, we learned that OpenAI had access to a tailor-made supercomputer that contained 280,000 CPUs and 10,000 GPUs.

A new era had begun. A battle of giants had begun with the recent ground-breaking intelligence of transformers and the power of supercomputers. Microsoft, Google, Facebook, Baidu, IBM, and others produce game-changing AI resources several times a year. AI project managers and developers need to continually ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Transformers for Natural Language Processing - Second Edition

Publisher Resources

ISBN: 9781800565791

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Transformers for Natural Language Processing

by Denis Rothman

6

Text Generation with OpenAI GPT-2 and GPT-3 Models

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.