book

Getting Started with Google BERT

Name: Getting Started with Google BERT
Author: Sudharsan Ravichandiran
ISBN: 9781838821593

by Sudharsan Ravichandiran

January 2021

Beginner to intermediate

352 pages

10h 17m

English

Packt Publishing

Read now

Unlock full access

Title Page
Copyright and Credits
Getting Started with Google BERT
Dedication
About Packt
Why subscribe?
Contributors
About the authorAbout the reviewersPackt is searching for authors like you
Preface
Who this book is forWhat this book coversTo get the most out of this bookDownload the example code filesDownload the color imagesConventions usedGet in touchReviews
Section 1 - Starting Off with BERT
A Primer on Transformers
Introduction to the transformer Understanding the encoder of the transformer Self-attention mechanism Understanding the self-attention mechanism Step 1Step 2Step 3Step 4Multi-head attention mechanism Learning position with positional encoding Feedforward networkAdd and norm component Putting all the encoder components together Understanding the decoder of a transformerMasked multi-head attention Multi-head attention Feedforward networkAdd and norm component Linear and softmax layersPutting all the decoder components together Putting the encoder and decoder together Training the transformerSummaryQuestionsFurther reading
Understanding the BERT Model
Basic idea of BERT Working of BERT Configurations of BERT BERT-baseBERT-largeOther configurations of BERTPre-training the BERT modelInput data representationToken embedding Segment embeddingPosition embedding Final representation WordPiece tokenizer Pre-training strategies Language modelingAuto-regressive language modeling Auto-encoding language modelingMasked language modelingWhole word masking Next sentence prediction Pre-training procedure Subword tokenization algorithms Byte pair encoding Tokenizing with BPE Byte-level byte pair encoding WordPieceSummaryQuestionsFurther reading
Getting Hands-On with BERT
Exploring the pre-trained BERT modelExtracting embeddings from pre-trained BERT Hugging Face transformers Generating BERT embeddingsPreprocessing the input Getting the embedding Extracting embeddings from all encoder layers of BERTExtracting the embeddings Preprocessing the inputGetting the embeddings Fine-tuning BERT for downstream tasksText classification Fine-tuning BERT for sentiment analysis Importing the dependencies Loading the model and datasetPreprocessing the datasetTraining the model Natural language inference Question-answeringPerforming question-answering with fine-tuned BERT Preprocessing the inputGetting the answerNamed entity recognition Summary QuestionsFurther reading

Section 2 - Exploring BERT Variants
BERT Variants I - ALBERT, RoBERTa, ELECTRA, and SpanBERT
A Lite version of BERT Cross-layer parameter sharing Factorized embedding parameterizationTraining the ALBERT modelSentence order predictionComparing ALBERT with BERT Extracting embeddings with ALBERTRobustly Optimized BERT pre-training ApproachUsing dynamic masking instead of static masking Removing the NSP taskTraining with more data pointsTraining with a large batch size Using BBPE as a tokenizer Exploring the RoBERTa tokenizer Understanding ELECTRA Understanding the replaced token detection task Exploring the generator and discriminator of ELECTRA Training the ELECTRA modelExploring efficient training methodsPredicting span with SpanBERTUnderstanding the architecture of SpanBERTExploring SpanBERT Performing Q and As with pre-trained SpanBERT SummaryQuestionsFurther reading
BERT Variants II - Based on Knowledge Distillation
Introducing knowledge distillation Training the student network DistilBERT – the distilled version of BERT Teacher-student architecture The teacher BERTThe student BERTTraining the student BERT (DistilBERT) Introducing TinyBERT Teacher-student architecture Understanding the teacher BERT Understanding the student BERT Distillation in TinyBERT Transformer layer distillation Attention-based distillationHidden state-based distillation Embedding layer distillation Prediction layer distillationThe final loss function Training the student BERT (TinyBERT)General distillation Task-specific distillation The data augmentation method Transferring knowledge from BERT to neural networksTeacher-student architecture The teacher BERT The student network Training the student network The data augmentation methodUnderstanding the masking methodUnderstanding the POS-guided word replacement method Understanding the n-gram sampling methodThe data augmentation procedureSummaryQuestionsFurther reading
Section 3 - Applications of BERT
Exploring BERTSUM for Text Summarization
Text summarization Extractive summarizationAbstractive summarization Fine-tuning BERT for text summarization Extractive summarization using BERT BERTSUM with a classifier BERTSUM with a transformer and LSTM BERTSUM with an inter-sentence transformer BERTSUM with LSTM Abstractive summarization using BERT Understanding ROUGE evaluation metricsUnderstanding the ROUGE-N metric ROUGE-1 ROUGE-2 Understanding ROUGE-L The performance of the BERTSUM model Training the BERTSUM model Summary QuestionsFurther reading
Applying BERT to Other Languages
Understanding multilingual BERT Evaluating M-BERT on the NLI task Zero-shot TRANSLATE-TEST TRANSLATE-TRAINTRANSLATE-TRAIN-ALLHow multilingual is multilingual BERT? Effect of vocabulary overlapGeneralization across scripts Generalization across typological features Effect of language similarityEffect of code switching and transliterationCode switching Transliteration M-BERT on code switching and transliteration The cross-lingual language modelPre-training strategies Causal language modeling Masked language modeling Translation language modeling Pre-training the XLM modelEvaluation of XLMUnderstanding XLM-RLanguage-specific BERT FlauBERT for French Getting a representation of a French sentence with FlauBERT French Language Understanding EvaluationBETO for Spanish Predicting masked words using BETO BERTje for DutchNext sentence prediction with BERTjeGerman BERT Chinese BERT Japanese BERT FinBERT for FinnishUmBERTo for Italian BERTimbau for Portuguese RuBERT for Russian SummaryQuestionsFurther reading
Exploring Sentence and Domain-Specific BERT
Learning about sentence representation with Sentence-BERT Computing sentence representation Understanding Sentence-BERT Sentence-BERT with a Siamese network Sentence-BERT for a sentence pair classification taskSentence-BERT for a sentence pair regression taskSentence-BERT with a triplet networkExploring the sentence-transformers library Computing sentence representation using Sentence-BERT Computing sentence similarity Loading custom modelsFinding a similar sentence with Sentence-BERT Learning multilingual embeddings through knowledge distillation Teacher-student architectureUsing the multilingual model Domain-specific BERT ClinicalBERT Pre-training ClinicalBERT Fine-tuning ClinicalBERT Extracting clinical word similarity BioBERT Pre-training the BioBERT modelFine-tuning the BioBERT model BioBERT for NER tasks BioBERT for question answering Summary QuestionsFurther reading
Working with VideoBERT, BART, and More
Learning language and video representations with VideoBERT Pre-training a VideoBERT model Cloze task Linguistic-visual alignment The final pre-training objective Data source and preprocessing Applications of VideoBERT Predicting the next visual tokensText-to-video generation Video captioning Understanding BART Architecture of BART Noising techniques Token maskingToken deletionToken infilling Sentence shuffling Document rotationComparing different pre-training objectives Performing text summarization with BART Exploring BERT libraries Understanding ktrainSentiment analysis using ktrainBuilding a document answering model Document summarization bert-as-service Installing the library Computing sentence representationComputing contextual word representation Summary Questions Further reading
Assessments
Chapter 1, A Primer on TransformersChapter 2, Understanding the BERT ModelChapter 3, Getting Hands-On with BERTChapter 4, BERT Variants I – ALBERT, RoBERTa, ELECTRA, SpanBERTChapter 5, BERT Variants II – Based on Knowledge DistillationChapter 6, Exploring BERTSUM for Text SummarizationChapter 7, Applying BERT to Other LanguagesChapter 8, Exploring Sentence- and Domain-Specific BERTChapter 9, Working with VideoBERT, BART, and More
Other Books You May Enjoy
Leave a review - let other readers know what you think

Content preview from Getting Started with Google BERT

Working with VideoBERT, BART, and More

Congratulations! We have made it to the final chapter. We have come a long way. We started off with understanding transformers, then we learned about BERT and several different variants of BERT in detail, from ALBERT to SentenceBERT. In this chapter, we will learn about two interesting models named VideoBERT and BART. We will also explore two popular BERT libraries known as ktrain and bert-as-service. We will start off the chapter by learning how VideoBERT works. We will look at how the VideoBERT model is pre-trained to learn the representation of language and video in a joint manner. Then, we will look into some of the applications of VideoBERT.

Moving on, we will learn what BART is and how it differs ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Claude Code Masterclass: Code faster with Agentic AI

Publisher Resources

ISBN: 9781838821593

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Getting Started with Google BERT

by Sudharsan Ravichandiran

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.