book

Deep Learning with TensorFlow and Keras – 3rd edition - Third Edition

Name: Deep Learning with TensorFlow and Keras – 3rd edition - Third Edition
ISBN: 9781803232911

by Dr. Amita Kapoor, Antonio Gulli, Sujit Pal

October 2022

Intermediate to advanced

698 pages

19h 43m

English

Packt Publishing

Read now

Unlock full access

Preface
Who this book is forWhat this book coversGet in touchReferences
Neural Network Foundations with TF
What is TensorFlow (TF)?What is Keras?Introduction to neural networksPerceptronOur first example of TensorFlow codeMulti-layer perceptron: our first example of a networkProblems in training the perceptron and solutionActivation function: sigmoidActivation function: tanhActivation function: ReLUTwo additional activation functions: ELU and Leaky ReLUActivation functionsIn short: what are neural networks after all?A real example: recognizing handwritten digitsOne hot-encoding (OHE)Defining a simple neural net in TensorFlowRunning a simple TensorFlow net and establishing a baselineImproving the simple net in TensorFlow with hidden layersFurther improving the simple net in TensorFlow with dropoutTesting different optimizers in TensorFlowIncreasing the number of epochsControlling the optimizer learning rateIncreasing the number of internal hidden neuronsIncreasing the size of batch computationSummarizing experiments run to recognizing handwritten digitsRegularizationAdopting regularization to avoid overfittingUnderstanding batch normalizationPlaying with Google Colab: CPUs, GPUs, and TPUsSentiment analysisHyperparameter tuning and AutoMLPredicting outputA practical overview of backpropagationWhat have we learned so far?Toward a deep learning approachSummaryReferences
Regression and Classification
What is regression?Prediction using linear regressionSimple linear regressionMultiple linear regressionMultivariate linear regressionNeural networks for linear regressionSimple linear regression using TensorFlow KerasMultiple and multivariate linear regression using the TensorFlow Keras APIClassification tasks and decision boundariesLogistic regressionLogistic regression on the MNIST datasetSummaryReferences
Convolutional Neural Networks
Deep convolutional neural networksLocal receptive fieldsShared weights and biasA mathematical exampleConvNets in TensorFlowPooling layersMax poolingAverage poolingConvNets summaryAn example of DCNN: LeNetLeNet code in TFUnderstanding the power of deep learningRecognizing CIFAR-10 images with deep learningImproving the CIFAR-10 performance with a deeper networkImproving the CIFAR-10 performance with data augmentationPredicting with CIFAR-10Very deep convolutional networks for large-scale image recognitionRecognizing cats with a VGG16 networkUtilizing the tf.Keras built-in VGG16 net moduleRecycling pre-built deep learning models for extracting featuresDeep Inception V3 for transfer learningOther CNN architecturesAlexNetResidual networksHighwayNets and DenseNetsXceptionStyle transferContent distanceStyle distanceSummaryReferences
Word Embeddings
Word embedding ‒ origins and fundamentalsDistributed representationsStatic embeddingsWord2VecGloVeCreating your own embeddings using GensimExploring the embedding space with GensimUsing word embeddings for spam detectionGetting the dataMaking the data ready for useBuilding the embedding matrixDefining the spam classifierTraining and evaluating the modelRunning the spam detectorNeural embeddings – not just for wordsItem2Vecnode2vecCharacter and subword embeddingsDynamic embeddingsSentence and paragraph embeddingsLanguage model-based embeddingsUsing BERT as a feature extractorSummaryReferences
Recurrent Neural Networks
The basic RNN cellBackpropagation through time (BPTT)Vanishing and exploding gradientsRNN cell variantsLong short-term memory (LSTM)Gated recurrent unit (GRU)Peephole LSTMRNN variantsBidirectional RNNsStateful RNNsRNN topologiesExample ‒ One-to-many – Learning to generate textExample ‒ Many-to-one – Sentiment analysisExample ‒ Many-to-many – POS taggingEncoder-decoder architecture – seq2seqExample ‒ seq2seq without attention for machine translationAttention mechanismExample ‒ seq2seq with attention for machine translationSummaryReferences
Transformers
ArchitectureKey intuitionsPositional encodingAttentionSelf-attentionMulti-head (self-)attentionHow to compute attentionEncoder-decoder architectureResidual and normalization layersAn overview of the transformer architectureTrainingTransformers’ architecturesCategories of transformersDecoder or autoregressiveEncoder or autoencodingSeq2seqMultimodalRetrievalAttentionFull versus sparseLSH attentionLocal attentionPretrainingEncoder pretrainingDecoder pretrainingEncoder-decoder pretrainingA taxonomy for pretraining tasksAn overview of popular and well-known modelsBERTGPT-2GPT-3ReformerBigBirdTransformer-XLXLNetRoBERTaALBERTStructBERTT5 and MUMELECTRADeBERTaThe Evolved Transformer and MEENALaMDA Switch TransformerRETROPathways and PaLMImplementationTransformer reference implementation: An example of translationHugging FaceGenerating text Autoselecting a model and autotokenizationNamed entity recognitionSummarizationFine-tuningTFHubEvaluationQualityGLUESuperGLUESQuADRACENLP-progressSizeLarger doesn’t always mean betterCost of servingOptimizationQuantizationWeight pruningDistillationCommon pitfalls: dos and don’tsDosDon’tsThe future of transformersSummary
Unsupervised Learning
Principal component analysisPCA on the MNIST datasetTensorFlow Embedding APIK-means clusteringK-means in TensorFlowVariations in k-meansSelf-organizing mapsColour mapping using a SOMRestricted Boltzmann machinesReconstructing images using an RBMDeep belief networksSummaryReferences
Autoencoders
Introduction to autoencodersVanilla autoencodersTensorFlow Keras layers ‒ defining custom layersReconstructing handwritten digits using an autoencoderSparse autoencoderDenoising autoencodersClearing images using a denoising autoencoderStacked autoencoderConvolutional autoencoder for removing noise from imagesA TensorFlow Keras autoencoder example ‒ sentence vectorsVariational autoencodersSummaryReferences
Generative Models
What is a GAN?MNIST using GAN in TensorFlowDeep convolutional GAN (DCGAN)DCGAN for MNIST digitsSome interesting GAN architecturesSRGANCycleGANInfoGANCool applications of GANsCycleGAN in TensorFlowFlow-based models for data generationDiffusion models for data generationSummaryReferences

Self-Supervised Learning
Previous workSelf-supervised learningSelf-predictionAutoregressive generationPixelRNNImage GPT (IPT)GPT-3XLNetWaveNetWaveRNNMasked generationBERTStacked denoising autoencoderContext autoencoderColorizationInnate relationship predictionRelative positionSolving jigsaw puzzlesRotationHybrid self-predictionVQ-VAEJukeboxDALL-EVQ-GANContrastive learningTraining objectivesContrastive lossTriplet lossN-pair lossLifted structural lossNCE lossInfoNCE lossSoft nearest neighbors lossInstance transformationSimCLRBarlow TwinsBYOLFeature clusteringDeepClusterSwAVInterCLRMultiview codingAMDIMCMCMultimodal modelsCLIPCodeSearchNetData2VecPretext tasksSummaryReferences
Reinforcement Learning
An introduction to RLRL lingoDeep reinforcement learning algorithmsHow does the agent choose its actions, especially when untrained?How does the agent maintain a balance between exploration and exploitation?How to deal with the highly correlated input state spaceHow to deal with the problem of moving targetsReinforcement success in recent yearsSimulation environments for RLAn introduction to OpenAI GymRandom agent playing BreakoutWrappers in GymDeep Q-networksDQN for CartPoleDQN to play a game of AtariDQN variantsDouble DQNDueling DQNRainbowDeep deterministic policy gradientSummaryReferences
Probabilistic TensorFlow
TensorFlow ProbabilityTensorFlow Probability distributionsUsing TFP distributionsCoin Flip ExampleNormal distributionBayesian networksHandling uncertainty in predictions using TensorFlow ProbabilityAleatory uncertaintyEpistemic uncertaintyCreating a synthetic datasetBuilding a regression model using TensorFlowProbabilistic neural networks for aleatory uncertaintyAccounting for the epistemic uncertaintySummaryReferences
An Introduction to AutoML
What is AutoML?Achieving AutoMLAutomatic data preparationAutomatic feature engineeringAutomatic model generationAutoKerasGoogle Cloud AutoML and Vertex AIUsing the Google Cloud AutoML Tables solutionUsing the Google Cloud AutoML Text solutionUsing the Google Cloud AutoML Video solutionCostSummaryReferences
The Math Behind Deep Learning
HistorySome mathematical toolsVectorsDerivatives and gradients everywhereGradient descentChain ruleA few differentiation rulesMatrix operationsActivation functionsDerivative of the sigmoidDerivative of tanhDerivative of ReLUBackpropagationForward stepBackstepCase 1: From hidden layer to output layerCase 2: From hidden layer to hidden layerCross entropy and its derivativeBatch gradient descent, stochastic gradient descent, and mini-batchBatch gradient descentStochastic gradient descentMini-batch gradient descentThinking about backpropagation and ConvNetsThinking about backpropagation and RNNsA note on TensorFlow and automatic differentiationSummaryReferences
Tensor Processing Unit
C/G/T processing unitsCPUs and GPUsTPUsFour generations of TPUs, plus Edge TPUFirst generation TPUSecond generation TPUThird generation TPUFourth generation TPUsEdge TPUTPU performanceHow to use TPUs with ColabChecking whether TPUs are availableKeras MNIST TPU end-to-end trainingUsing pretrained TPU modelsSummaryReferences
Other Useful Deep Learning Libraries
Hugging FaceOpenAIOpenAI GPT-3 APIOpenAI DALL-E 2OpenAI CodexPyTorchONNXH2O.aiH2O AutoML AutoML using H2OH2O model explainabilityPartial dependence plotsVariable importance heatmapModel correlationSummary
Graph Neural Networks
Graph basicsGraph machine learningGraph convolutions – the intuition behind GNNsCommon graph layersGraph convolution networkGraph attention networkGraphSAGE (sample and aggregate)Graph isomorphism networkCommon graph applicationsNode classificationGraph classificationLink predictionGraph customizationsCustom layers and message passingCustom graph datasetSingle graphs in datasetsSet of multiple graphs in datasetsFuture directionsHeterogeneous graphsTemporal GraphsSummaryReferences
Machine Learning Best Practices
The need for best practicesData best practicesFeature selectionFeatures and dataAugmenting textual dataModel best practicesBaseline modelsPretrained models, model APIs, and AutoMLModel evaluation and validationModel improvementsSummaryReferences
TensorFlow 2 Ecosystem
TensorFlow HubUsing pretrained models for inferenceTensorFlow DatasetsLoad a TFDS datasetBuilding data pipelines using TFDSTensorFlow LiteQuantizationFlatBuffersMobile converterMobile optimized interpreterSupported platformsArchitectureUsing TensorFlow LiteA generic example of an applicationUsing GPUs and acceleratorsAn example of an applicationPretrained models in TensorFlow LiteImage classificationObject detectionPose estimationSmart replySegmentationStyle transferText classificationLarge language modelsA note about using mobile GPUsAn overview of federated learning at the edgeTensorFlow FL APIsTensorFlow.jsVanilla TensorFlow.jsConverting modelsPretrained modelsNode.jsSummaryReferences
Advanced Convolutional Neural Networks
Composing CNNs for complex tasksClassification and localizationSemantic segmentationObject detectionInstance segmentationApplication zoos with tf.Keras and TensorFlow HubKeras ApplicationsTensorFlow HubAnswering questions about images (visual Q&A)Creating a DeepDream networkInspecting what a network has learnedVideoClassifying videos with pretrained nets in six different waysText documentsUsing a CNN for sentiment analysisAudio and musicDilated ConvNets, WaveNet, and NSynthA summary of convolution operationsBasic CNNsDilated convolutionTransposed convolutionSeparable convolutionDepthwise convolutionDepthwise separable convolutionCapsule networksWhat is the problem with CNNs?What is new with capsule networks?SummaryReferences
Other Books You May Enjoy
Index

Content preview from Deep Learning with TensorFlow and Keras – 3rd edition - Third Edition

6 Transformers

The transformer-based architectures have become almost universal in Natural Language Processing (NLP) (and beyond) when it comes to solving a wide variety of tasks, such as:

Neural machine translation
Text summarization
Text generation
Named entity recognition
Question answering
Text classification
Text similarity
Offensive message/profanity detection
Query understanding
Language modeling
Next-sentence prediction
Reading comprehension
Sentiment analysis
Paraphrasing

and a lot more.

In less than four years, when the Attention Is All You Need paper was published by Google Research in 2017, transformers managed to take the NLP community by storm, breaking any record achieved over the previous thirty years.

Transformer-based models ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Deep Learning with TensorFlow, Keras, and PyTorch

Publisher Resources

ISBN: 9781803232911

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Deep Learning with TensorFlow and Keras – 3rd edition - Third Edition

by Dr. Amita Kapoor, Antonio Gulli, Sujit Pal

6

Transformers

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.