book

Mastering Transformers. - Second Edition

Name: Mastering Transformers. - Second Edition
ISBN: 9781837633784

by Savaş Yıldırım, Meysam Asgari- Chenaghlu

June 2024

Intermediate to advanced

462 pages

10h 56m

English

Packt Publishing

Read now

Unlock full access

Mastering Transformers
ContributorsAbout the authorsAbout the reviewers
Preface
Who this book is forWhat this book coversTo get the most out of this bookDownload the example code filesConventions usedGet in touchShare Your Thoughts
Part 1: Recent Developments in the Field, Installations, and Hello World Applications
Chapter 1: From Bag-of-Words to the Transformers
Evolution of NLP approachesRecalling traditional NLP approachesLanguage modeling and generationLeveraging DLConsidering the word order with RNN modelsLSTMs and gated recurrent unitsContextual word embeddings and TLOverview of the Transformer architectureAttention mechanismMulti-head attention mechanismsUsing TL with TransformersMultimodal learningSummaryReferences
Chapter 2: A Hands-On Introduction to the Subject
Technical requirementsInstalling transformer with AnacondaInstallation on LinuxInstallation on WindowsInstallation on macOSInstalling TensorFlow, PyTorch, and TransformerInstalling and using Google ColabWorking with language models and tokenizersWorking with community-provided modelsWorking with multimodal transformersWorking with benchmarks and datasetsImportant benchmarksGLUE benchmarkSuperGLUE benchmarkXTREME benchmarkXGLUE benchmarkSQuAD benchmarkAccessing the datasets with an application programming interfaceData manipulation using the datasets librarySorting, indexing, and shufflingCaching and reusabilityDataset filter and map functionProcessing data with the map functionWorking with local filesPreparing a dataset for model trainingBenchmarking for speed and memorySummary
Part 2: Transformer Models: From Autoencoders to Autoregressive Models
Chapter 3: Autoencoding Language Models
Technical requirementsBERT – one of the autoencoding language modelsBERT language model pretraining tasksA deeper look into the BERT language modelAutoencoding language model training for any languageSharing models with the communityOther autoencoding modelsIntroducing ALBERTRoBERTaELECTRADeBERTaWorking with tokenization algorithmsBPEWordPiece tokenizationSentence piece tokenizationThe tokenizers librarySummary
Chapter 4: From Generative Models to Large Language Models
Technical requirementsAn introduction to GLMsWorking with GLMsGPT model familyTransformer-XLXLNetWorking with text-to-text modelsMulti-task learning with T5Zero-Shot Text Generalization with T0Another Denoising-Based Seq2Seq Model – BARTGLM trainingNLG using AR modelsSummaryReferences
Chapter 5: Fine-Tuning Language Models for Text Classification
Technical requirementsIntroduction to text classificationFine-tuning a BERT model for single-sentence binary classificationTraining a classification model with native PyTorchFine-tuning BERT for multi-class classification with custom datasetsFine-tuning the BERT model for sentence-pair regressionMultilabel text classificationUtilizing run_glue.py to fine-tune the modelsSummaryReferences
Chapter 6: Fine-Tuning Language Models for Token Classification
Technical requirementsIntroduction to token classificationUnderstanding NERUnderstanding POS taggingUnderstanding QAFine-tuning language models for NERQuestion answering using token classificationQuestion answering for many tasksSummary

Chapter 7: Text Representation
Technical requirementsIntroduction to sentence embeddingsCross-encoder versus bi-encoderBenchmarking sentence similarity modelsUsing BART for zero-shot learningSemantic similarity experiment with FLAIRAverage word embeddingsRNN-based document embeddingsTransformer-based BERT embeddingsSBERT embeddingsText clustering with Sentence-BERTTopic modeling with BERTopicSemantic search with SBERTInstruction fine-tuned embedding modelsSummaryFurther reading
Chapter 8: Boosting Model Performance
Technical requirementsImproving performance with data augmentationCharacter-level augmentationWord-level augmentationSentence-level augmentationBoosting IMDB text classification with augmentationAdapting the model to the domainOptimizing the parameters with HPOSummary
Chapter 9: Parameter Efficient Fine-Tuning
Technical requirementsIntroduction to PEFTUnderstanding Types of PEFTAdditive methodsSelective methodsLow-rank fine-tuningHands-on PEFT experimentsFine-tuning a BERT checkpoint with adapter tuningEfficiently fine-tune FLAN-T5 for an NLI task with LoraTuning with QLoRASummaryReferences
Part 3: Advanced Topics
Chapter 10: Large Language Models
Technical requirementsWhy large language models?Importance of reward functionThe instruction-following ability of LLMsFine-tuning large language modelsSummary
Chapter 11: Explainable AI (XAI) in NLP
Technical requirementsInterpreting attention headsVisualizing attention heads with exBERTMultiscale visualization of attention heads with BertVizUnderstanding the inner parts of BERT with probing classifiersExplain the model decisionInterpret Transformers’ decision with LIMEInterpret Transformers’ decision with SHAPSummary
Chapter 12: Working with Efficient Transformers
Technical requirementsIntroduction to efficient, light, and fast transformersImplementation for model size reductionWorking with DistilBERT for knowledge distillationPruning transformersQuantizationWorking with efficient self-attentionSparse attention with fixed patternsLearnable patternsLow-rank factorization, kernel methods, and other approachesEasier quantization using bitsandbytesSummaryReferences
Chapter 13: Cross-Lingual and Multilingual Language Modeling
Technical requirementsTranslation language modeling and cross-lingual knowledge sharingXLM and mBERTmBERTXLMCross-lingual similarity tasksCross-lingual text similarityVisualizing cross-lingual textual similarityCross-lingual classificationCross-lingual zero-shot learningMassive multilingual translationFine-tuning the performance of multilingual modelsSummaryReferences
Chapter 14: Serving Transformer Models
Technical requirementsFastAPI Transformer model servingDockerizing APIsFaster Transformer model serving using TFXLoad testing using LocustFaster inference using ONNXSageMaker inferenceSummaryFurther reading
Chapter 15: Model Tracking and Monitoring
Technical requirementsTracking model metricsTracking model training with TensorBoardTracking model training live with W&BSummaryFurther reading
Part 4: Transformers beyond NLP
Chapter 16: Vision Transformers
Technical requirementsVision transformersImage classification using transformersSemantic segmentation and object detection using transformersVisual prompt modelsSummary
Chapter 17: Multimodal Generative Transformers
Technical requirementsMultimodal learningGenerative multimodal AIStable Diffusion for text-to-image generationStable Diffusion in actionMusic generation using MusicGenText-to-speech generation using transformersSummary
Chapter 18: Revisiting Transformers Architecture for Time Series
Technical requirementsUnderstanding time series conceptsTransformers and time series modelingSummary
Index
Why subscribe?
Other Books You May EnjoyPackt is searching for authors like youShare Your ThoughtsDownload a free PDF copy of this book

Content preview from Mastering Transformers. - Second Edition

3 Autoencoding Language Models

In the previous chapter, we looked at and studied how a typical transformer model can be used by Hugging Face’s Transformers. All of the topics in this book so far have included instructions on how to use pre-trained or pre-built models and less information has been given about specific models and their training.

In this chapter, we will learn how we can train autoencoding language models on any given language from scratch. This training will include pretraining and task-specific training of the models. First, we will start with learning about the BERT model and how it works. Then, we will train the language model using a simple and small corpus. Afterward, we will look at how the model can be used inside any ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Publisher Resources

ISBN: 9781837633784

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Mastering Transformers. - Second Edition

by Savaş Yıldırım, Meysam Asgari- Chenaghlu

3

Autoencoding Language Models

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.