book

Transformers for Natural Language Processing - Second Edition

Name: Transformers for Natural Language Processing - Second Edition
Author: Denis Rothman
ISBN: 9781803247335

by Denis Rothman

March 2022

Beginner to intermediate

602 pages

12h 42m

English

Packt Publishing

Read now

Unlock full access

Preface
Who this book is forWhat this book coversTo get the most out of this bookGet in touch
What are Transformers?
The ecosystem of transformersIndustry 4.0Foundation modelsIs programming becoming a sub-domain of NLP?The future of artificial intelligence specialistsOptimizing NLP models with transformersThe background of transformersWhat resources should we use?The rise of Transformer 4.0 seamless APIsChoosing ready-to-use API-driven librariesChoosing a Transformer ModelThe role of Industry 4.0 artificial intelligence specialistsSummaryQuestionsReferences
Getting Started with the Architecture of the Transformer Model
The rise of the Transformer: Attention is All You NeedThe encoder stackInput embeddingPositional encodingSublayer 1: Multi-head attentionSublayer 2: Feedforward networkThe decoder stackOutput embedding and position encodingThe attention layersThe FFN sublayer, the post-LN, and the linear layerTraining and performanceTranformer models in Hugging FaceSummaryQuestionsReferences
Fine-Tuning BERT Models
The architecture of BERTThe encoder stackPreparing the pretraining input environmentPretraining and fine-tuning a BERT modelFine-tuning BERTHardware constraintsInstalling the Hugging Face PyTorch interface for BERTImporting the modulesSpecifying CUDA as the device for torchLoading the datasetCreating sentences, label lists, and adding BERT tokensActivating the BERT tokenizerProcessing the dataCreating attention masksSplitting the data into training and validation setsConverting all the data into torch tensorsSelecting a batch size and creating an iteratorBERT model configurationLoading the Hugging Face BERT uncased base modelOptimizer grouped parametersThe hyperparameters for the training loopThe training loopTraining evaluationPredicting and evaluating using the holdout datasetEvaluating using the Matthews Correlation CoefficientThe scores of individual batchesMatthews evaluation for the whole datasetSummaryQuestionsReferences
Pretraining a RoBERTa Model from Scratch
Training a tokenizer and pretraining a transformerBuilding KantaiBERT from scratchStep 1: Loading the datasetStep 2: Installing Hugging Face transformersStep 3: Training a tokenizerStep 4: Saving the files to diskStep 5: Loading the trained tokenizer filesStep 6: Checking resource constraints: GPU and CUDAStep 7: Defining the configuration of the modelStep 8: Reloading the tokenizer in transformersStep 9: Initializing a model from scratchExploring the parametersStep 10: Building the datasetStep 11: Defining a data collatorStep 12: Initializing the trainerStep 13: Pretraining the modelStep 14: Saving the final model (+tokenizer + config) to diskStep 15: Language modeling with FillMaskPipelineNext stepsSummaryQuestionsReferences
Downstream NLP Tasks with Transformers
Transduction and the inductive inheritance of transformersThe human intelligence stackThe machine intelligence stackTransformer performances versus Human BaselinesEvaluating models with metricsAccuracy scoreF1-scoreMatthews Correlation Coefficient (MCC)Benchmark tasks and datasetsFrom GLUE to SuperGLUEIntroducing higher Human Baselines standardsThe SuperGLUE evaluation processDefining the SuperGLUE benchmark tasksBoolQCommitment Bank (CB)Multi-Sentence Reading Comprehension (MultiRC)Reading Comprehension with Commonsense Reasoning Dataset (ReCoRD)Recognizing Textual Entailment (RTE)Words in Context (WiC)The Winograd schema challenge (WSC)Running downstream tasksThe Corpus of Linguistic Acceptability (CoLA)Stanford Sentiment TreeBank (SST-2)Microsoft Research Paraphrase Corpus (MRPC)Winograd schemasSummaryQuestionsReferences
Machine Translation with the Transformer
Defining machine translationHuman transductions and translationsMachine transductions and translationsPreprocessing a WMT datasetPreprocessing the raw dataFinalizing the preprocessing of the datasetsEvaluating machine translation with BLEUGeometric evaluationsApplying a smoothing techniqueChencherry smoothingTranslation with Google TranslateTranslations with TraxInstalling TraxCreating the original Transformer modelInitializing the model using pretrained weightsTokenizing a sentenceDecoding from the TransformerDe-tokenizing and displaying the translationSummaryQuestionsReferences
The Rise of Suprahuman Transformers with GPT-3 Engines
Suprahuman NLP with GPT-3 transformer modelsThe architecture of OpenAI GPT transformer modelsThe rise of billion-parameter transformer modelsThe increasing size of transformer modelsContext size and maximum path lengthFrom fine-tuning to zero-shot modelsStacking decoder layersGPT-3 enginesGeneric text completion with GPT-2Step 9: Interacting with GPT-2Training a custom GPT-2 language modelStep 12: Interactive context and completion examplesRunning OpenAI GPT-3 tasksRunning NLP tasks onlineGetting started with GPT-3 enginesRunning our first NLP task with GPT-3NLP tasks and examplesComparing the output of GPT-2 and GPT-3Fine-tuning GPT-3Preparing the dataStep 1: Installing OpenAIStep 2: Entering the API keyStep 3: Activating OpenAI’s data preparation moduleFine-tuning GPT-3Step 4: Creating an OS environmentStep 5: Fine-tuning OpenAI’s Ada engineStep 6: Interacting with the fine-tuned modelThe role of an Industry 4.0 AI specialistInitial conclusionsSummaryQuestionsReferences
Applying Transformers to Legal and Financial Documents for AI Text Summarization
Designing a universal text-to-text modelThe rise of text-to-text transformer modelsA prefix instead of task-specific formatsThe T5 modelText summarization with T5Hugging FaceHugging Face transformer resourcesInitializing the T5-large transformer modelGetting started with T5Exploring the architecture of the T5 modelSummarizing documents with T5-largeCreating a summarization functionA general topic sampleThe Bill of Rights sampleA corporate law sampleSummarization with GPT-3SummaryQuestionsReferences
Matching Tokenizers and Datasets
Matching datasets and tokenizersBest practicesStep 1: PreprocessingStep 2: Quality controlContinuous human quality controlWord2Vec tokenizationCase 0: Words in the dataset and the dictionaryCase 1: Words not in the dataset or the dictionaryCase 2: Noisy relationshipsCase 3: Words in the text but not in the dictionaryCase 4: Rare wordsCase 5: Replacing rare wordsCase 6: EntailmentStandard NLP tasks with specific vocabularyGenerating unconditional samples with GPT-2Generating trained conditional samplesControlling tokenized dataExploring the scope of GPT-3SummaryQuestionsReferences

Semantic Role Labeling with BERT-Based Transformers
Getting started with SRLDefining semantic role labelingVisualizing SRLRunning a pretrained BERT-based modelThe architecture of the BERT-based modelSetting up the BERT SRL environmentSRL experiments with the BERT-based modelBasic samplesSample 1Sample 2Sample 3Difficult samplesSample 4Sample 5Sample 6Questioning the scope of SRLThe limit of predicate analysisRedefining SRLSummaryQuestionsReferences
Let Your Data Do the Talking: Story, Questions, and Answers
MethodologyTransformers and methodsMethod 0: Trial and errorMethod 1: NER firstUsing NER to find questionsLocation entity questionsPerson entity questionsMethod 2: SRL firstQuestion-answering with ELECTRAProject management constraintsUsing SRL to find questionsNext stepsExploring Haystack with a RoBERTa modelExploring Q&A with a GTP-3 engineSummaryQuestionsReferences
Detecting Customer Emotions to Make Predictions
Getting started: Sentiment analysis transformersThe Stanford Sentiment Treebank (SST)Sentiment analysis with RoBERTa-largePredicting customer behavior with sentiment analysisSentiment analysis with DistilBERTSentiment analysis with Hugging Face’s models’ listDistilBERT for SSTMiniLM-L12-H384-uncasedRoBERTa-large-mnliBERT-base multilingual modelSentiment analysis with GPT-3Some Pragmatic I4.0 thinking before we leaveInvestigating with SRLInvestigating with Hugging FaceInvestigating with the GPT-3 playgroundGPT-3 codeSummaryQuestionsReferences
Analyzing Fake News with Transformers
Emotional reactions to fake newsCognitive dissonance triggers emotional reactionsAnalyzing a conflictual TweetBehavioral representation of fake newsA rational approach to fake newsDefining a fake news resolution roadmapThe gun control debateSentiment analysisNamed entity recognition (NER)Semantic Role Labeling (SRL)Gun control SRLReference sitesCOVID-19 and former President Trump’s TweetsSemantic Role Labeling (SRL)Before we goSummaryQuestionsReferences
Interpreting Black Box Transformer Models
Transformer visualization with BertVizRunning BertVizStep 1: Installing BertViz and importing the modulesStep 2: Load the models and retrieve attentionStep 3: Head viewStep 4: Processing and displaying attention headsStep 5: Model viewLITPCARunning LITTransformer visualization via dictionary learningTransformer factorsIntroducing LIMEThe visualization interfaceExploring models we cannot accessSummaryQuestionsReferences
From NLP to Task-Agnostic Transformer Models
Choosing a model and an ecosystemThe ReformerRunning an exampleDeBERTaRunning an exampleFrom Task-Agnostic Models to Vision TransformersViT – Vision TransformersThe Basic Architecture of ViTVision transformers in codeCLIPThe Basic Architecture of CLIPCLIP in codeDALL-EThe Basic Architecture of DALL-EDALL-E in codeAn expanding universe of modelsSummaryQuestionsReferences
The Emergence of Transformer-Driven Copilots
Prompt engineeringCasual English with a meaningful contextCasual English with a metonymyCasual English with an ellipsisCasual English with vague contextCasual English with sensorsCasual English with sensors but no visible contextFormal English conversation with no contextPrompt engineering trainingCopilotsGitHub CopilotDomain-specific GPT-3 enginesEmbedding2MLStep 1: Installing and importing OpenAIStep 2: Loading the datasetStep 3: Combining the columnsStep 4: Running the GPT-3 embeddingStep 5: Clustering (k-means clustering) with the embeddingsStep 6: Visualizing the clusters (t-SNE)Instruct seriesContent filterTransformer-based recommender systemsGeneral-purpose sequencesDataset pipeline simulation with RL using an MDPTraining customer behaviors with an MDPSimulating consumer behavior with an MDPMaking recommendationsComputer visionHumans and AI copilots in metaversesFrom looking at to being inSummaryQuestionsReferences
The Consolidation of Suprahuman Transformers with OpenAI’s ChatGPT and GPT-4
Consolidating suprahuman NLP with ChatGPT and GPT-4 transformer modelsHow to get the most out of this chapterBuild on your knowledgeFocus on the innovationsGitHub repositoryAccepting the limitations of a disruptive innovationWho owns AI-created content?OpportunitiesList of models in this chapter Jump-starting the ChatGPT API Content generationResponseExtracting the text of the response ChatGPT Plus writes and comments on a programDesigning the promptChatGPT Plus writes the programChatGPT Plus plots the outputGetting started with the GPT-4 API1. GPT-4 helps to explain how to write source code2. GPT-4 creates a function to show the YouTube presentation of GPT-4 by Greg Brockman on March 14th, 20233. GPT-4 creates an application for WikiArt to display images4. GPT-4 creates an application to display IMDb reviews5. GPT-4 creates an application to display a news feed6. GPT-4 creates a k-means clustering (kmc) algorithm7. Talking with GPT-4 about GPT-4 and GPT model architectureAdvanced prompt engineering1. Building a knowledge base for ChatGPT and GPT-42. Adding keywords and parsing user requests3. Building prompts that pilot ChatGPT4. Moderation, quality controlExplainable AI (XAI)Speech-to-text with WhisperText-to-speech with gTTSText-to-speech with gTTS and IPythonTranscribing audio files with WhisperGetting started with the DALL-E 2 APICreating a new imageCreating a variation of an imagePutting it all togetherSummaryQuestionsReferences
Appendix I — Terminology of Transformer Models
StackSublayerAttention heads
Appendix II — Hardware Constraints for Transformer Models
The Architecture and Scale of TransformersWhy GPUs are so specialGPUs are designed for parallel computingGPUs are also designed for matrix multiplicationImplementing GPUs in codeTesting GPUs with Google ColabGoogle Colab Free with a CPUGoogle Colab Free with a GPUGoogle Colab Pro with a GPU
Appendix III — Generic Text Completion with GPT-2
Step 1: Activating the GPUStep 2: Cloning the OpenAI GPT-2 repositoryStep 3: Installing the requirementsStep 4: Checking the version of TensorFlowStep 5: Downloading the 345M-parameter GPT-2 modelSteps 6-7: Intermediate instructionsSteps 7b-8: Importing and defining the modelStep 9: Interacting with GPT-2References
Appendix IV — Custom Text Completion with GPT-2
Training a GPT-2 language modelStep 1: PrerequisitesSteps 2 to 6: Initial steps of the training processStep 7: The N Shepperd training filesStep 8: Encoding the datasetStep 9: Training a GPT-2 modelStep 10: Creating a training model directoryStep 11: Generating unconditional samplesStep 12: Interactive context and completion examplesReferences
Appendix V — Answers to the Questions
Chapter 1, What are Transformers?Chapter 2, Getting Started with the Architecture of the Transformer ModelChapter 3, Fine-Tuning BERT ModelsChapter 4, Pretraining a RoBERTa Model from ScratchChapter 5, Downstream NLP Tasks with TransformersChapter 6, Machine Translation with the TransformerChapter 7, The Rise of Suprahuman Transformers with GPT-3 EnginesChapter 8, Applying Transformers to Legal and Financial Documents for AI Text SummarizationChapter 9, Matching Tokenizers and DatasetsChapter 10, Semantic Role Labeling with BERT-Based TransformersChapter 11, Let Your Data Do the Talking: Story, Questions, and AnswersChapter 12, Detecting Customer Emotions to Make PredictionsChapter 13, Analyzing Fake News with TransformersChapter 14, Interpreting Black Box Transformer ModelsChapter 15, From NLP to Task-Agnostic Transformer ModelsChapter 16, The Emergence of Transformer-Driven CopilotsChapter 17, The Consolidation of Suprahuman Transformers with OpenAI’s ChatGPT and GPT-4
Other Books You May Enjoy
Index

Overview

If you are eager to delve into the field of Natural Language Processing using modern and cutting-edge transformer technology such as GPT-3, GPT-4, and Hugging Face, this book serves as your guide. "Transformers for Natural Language Processing" offers hands-on exercises and deep insights into transformer models, helping you enhance your NLP expertise using Python, TensorFlow, and PyTorch.

What this Book will help me do

Acquire the ability to fine-tune advanced transformer models like GPT-4 and BERT for your NLP tasks.
Explore various transformer platforms and learn best practices for prompt engineering.
Gain experience in training NLP models from scratch using Hugging Face's tools and frameworks.
Perform practical NLP tasks like sentiment analysis, machine translation, and text summarization.
Understand the potential of transformers in image processing and creative AI applications.

Author(s)

Denis Rothman, the accomplished author of this book, has extensive experience in AI and Natural Language Processing. With a strong background in developing transformer-based solutions, Denis leverages his expertise to provide insightful and detailed guidance to his readers. His philosophy emphasizes learning by doing, equipping readers with the skills to explore complex AI models confidently.

Who is it for?

This book is designed for AI practitioners, data scientists, and NLP enthusiasts who have a foundation in Python and deep learning. If you're looking to elevate your skills in transformer models and explore their applications, this is the book for you. Familiarity with basic NLP concepts will help maximize the learning experience. Whether you're a professional seeking to sharpen your skills or a developer keen to explore advanced models like GPT-4, you'll find this book a valuable companion.

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Transformers for Natural Language Processing

Publisher Resources

ISBN: 9781803247335

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills