book

Deep Learning with TensorFlow 2 and Keras - Second Edition

Name: Deep Learning with TensorFlow 2 and Keras - Second Edition
ISBN: 9781838823412

by Antonio Gulli, Dr. Amita Kapoor, Sujit Pal

December 2019

Intermediate to advanced

646 pages

15h 11m

English

Packt Publishing

Read now

Unlock full access

Preface
MissionMachine learning, artificial intelligence, and the deep learning Cambrian explosionWho this book is forWhat this book coversWhat you need for this bookDownloading the example codeDownload the color imagesConventionsGet in touchReviewsReferences
Neural Network Foundations with TensorFlow 2.0
What is TensorFlow (TF)?What is Keras?What are the most important changes in TensorFlow 2.0?Introduction to neural networksPerceptronA first example of TensorFlow 2.0 codeMulti-layer perceptron – our first example of a networkProblems in training the perceptron and their solutionsActivation function – sigmoidActivation function – tanhActivation function – ReLUTwo additional activation functions – ELU and LeakyReLUActivation functionsIn short – what are neural networks after all?A real example – recognizing handwritten digitsOne-hot encoding (OHE)Defining a simple neural network in TensorFlow 2.0Running a simple TensorFlow 2.0 net and establishing a baselineImproving the simple net in TensorFlow 2.0 with hidden layersFurther improving the simple net in TensorFlow with DropoutTesting different optimizers in TensorFlow 2.0Increasing the number of epochsControlling the optimizer learning rateIncreasing the number of internal hidden neuronsIncreasing the size of batch computationSummarizing experiments run for recognizing handwritten chartsRegularizationAdopting regularization to avoid overfittingUnderstanding BatchNormalizationPlaying with Google Colab – CPUs, GPUs, and TPUsSentiment analysisHyperparameter tuning and AutoMLPredicting outputA practical overview of backpropagation What have we learned so far?Towards a deep learning approachReferences
TensorFlow 1.x and 2.x
Understanding TensorFlow 1.xTensorFlow 1.x computational graph program structureComputational graphsWorking with constants, variables, and placeholdersExamples of operationsConstantsSequencesRandom tensorsVariablesAn example of TensorFlow 1.x in TensorFlow 2.xUnderstanding TensorFlow 2.xEager executionAutoGraphKeras APIs – three programming modelsSequential APIFunctional APIModel subclassingCallbacksSaving a model and weightsTraining from tf.data.datasetstf.keras or Estimators?Ragged tensorsCustom trainingDistributed training in TensorFlow 2.xMultiple GPUsMultiWorkerMirroredStrategyTPUStrategyParameterServerStrategyChanges in namespacesConverting from 1.x to 2.xUsing TensorFlow 2.x effectivelyThe TensorFlow 2.x ecosystemLanguage bindingsKeras or tf.keras?Summary
Regression
What is regression?Prediction using linear regressionSimple linear regressionMultiple linear regressionMultivariate linear regressionTensorFlow EstimatorsFeature columnsInput functionsMNIST using TensorFlow Estimator APIPredicting house price using linear regressionClassification tasks and decision boundariesLogistic regressionLogistic regression on the MNIST datasetSummaryReferences
Convolutional Neural Networks
Deep Convolutional Neural Network (DCNN)Local receptive fieldsShared weights and biasA mathematical exampleConvNets in TensorFlow 2.xPooling layersMax poolingAverage poolingConvNets summaryAn example of DCNN ‒ LeNetLeNet code in TensorFlow 2.0Understanding the power of deep learningRecognizing CIFAR-10 images with deep learning Improving the CIFAR-10 performance with a deeper networkImproving the CIFAR-10 performance with data augmentationPredicting with CIFAR-10Very deep convolutional networks for large-scale image recognitionRecognizing cats with a VGG16 NetUtilizing tf.keras built-in VGG16 Net moduleRecycling prebuilt deep learning models for extracting featuresSummaryReferences
Advanced Convolutional Neural Networks
Computer visionComposing CNNs for complex tasksClassification and localizationSemantic segmentationObject detectionInstance segmentationClassifying Fashion-MNIST with a tf.keras - estimator modelRun Fashion-MNIST the tf.keras - estimator model on GPUsDeep Inception-v3 Net used for transfer learningTransfer learning for classifying horses and humansApplication Zoos with tf.keras and TensorFlow HubKeras applicationsTensorFlow HubOther CNN architecturesAlexNetResidual networksHighwayNets and DenseNetsXceptionAnswering questions about images (VQA)Style transferContent distanceStyle distanceCreating a DeepDream networkInspecting what a network has learned VideoClassifying videos with pretrained nets in six different waysTextual documentsUsing a CNN for sentiment analysisAudio and musicDilated ConvNets, WaveNet, and NSynthA summary of convolution operationsBasic convolutional neural networks (CNN or ConvNet)Dilated convolutionTransposed convolutionSeparable convolutionDepthwise convolutionDepthwise separable convolutionCapsule networksSo what is the problem with CNNs?So what is new with Capsule networks?SummaryReferences
Generative Adversarial Networks
What is a GAN?MNIST using GAN in TensorFlowDeep convolutional GAN (DCGAN)DCGAN for MNIST digitsSome interesting GAN architecturesSRGANCycleGANInfoGANCool applications of GANsCycleGAN in TensorFlow 2.0SummaryReferences
Word Embeddings
Word embedding ‒ origins and fundamentalsDistributed representationsStatic embeddingsWord2VecGloVeCreating your own embedding using gensimExploring the embedding space with gensimUsing word embeddings for spam detectionGetting the dataMaking the data ready for useBuilding the embedding matrixDefine the spam classifierTrain and evaluate the modelRunning the spam detectorNeural embeddings – not just for wordsItem2Vecnode2vecCharacter and subword embeddingsDynamic embeddingsSentence and paragraph embeddingsLanguage model-based embeddingsUsing BERT as a feature extractorFine-tuning BERTClassifying with BERT ‒ command lineUsing BERT as part of your own networkSummaryReferences
Recurrent Neural Networks
The basic RNN cellBackpropagation through time (BPTT)Vanishing and exploding gradientsRNN cell variantsLong short-term memory (LSTM)Gated recurrent unit (GRU)Peephole LSTMRNN variantsBidirectional RNNsStateful RNNsRNN topologiesExample ‒ One-to-Many – learning to generate textExample ‒ Many-to-One – Sentiment AnalysisExample ‒ Many-to-Many – POS taggingEncoder-Decoder architecture – seq2seqExample ‒ seq2seq without attention for machine translationAttention mechanismExample ‒ seq2seq with attention for machine translationTransformer architectureSummaryReferences
Autoencoders
Introduction to autoencodersVanilla autoencodersTensorFlow Keras layers ‒ defining custom layersReconstructing handwritten digits using an autoencoderSparse autoencoderDenoising autoencodersClearing images using a Denoising autoencoderStacked autoencoderConvolutional autoencoder for removing noise from imagesKeras autoencoder example ‒ sentence vectorsSummaryReferences

Unsupervised Learning
Principal component analysisPCA on the MNIST datasetTensorFlow Embedding APIK-means clusteringK-means in TensorFlow 2.0Variations in k-meansSelf-organizing mapsColour mapping using SOMRestricted Boltzmann machinesReconstructing images using RBMDeep belief networksVariational AutoencodersSummaryReferences
Reinforcement Learning
IntroductionRL lingoDeep reinforcement learning algorithmsReinforcement success in recent yearsIntroduction to OpenAI GymRandom agent playing BreakoutDeep Q-NetworksDQN for CartPoleDQN to play a game of AtariDQN variantsDouble DQNDueling DQNRainbowDeep deterministic policy gradientSummaryReferences
TensorFlow and Cloud
Deep learning on cloudMicrosoft AzureAmazon Web Services (AWS)Google Cloud Platform (GCP)IBM CloudVirtual machines on cloudEC2 on AmazonCompute Instance on GCPVirtual machine on Microsoft AzureJupyter Notebooks on cloudSageMakerGoogle ColaboratoryMicrosoft Azure NotebooksTensorFlow Extended for productionTFX PipelinesTFX pipeline componentsTFX librariesTensorFlow EnterpriseSummaryReferences
TensorFlow for Mobile and IoT and TensorFlow.js
TensorFlow MobileTensorFlow LiteQuantizationFlatBuffersMobile converterMobile optimized interpreterSupported platformsArchitectureUsing TensorFlow LiteA generic example of applicationUsing GPUs and acceleratorsAn example of applicationPretrained models in TensorFlow LiteImage classificationObject detectionPose estimationSmart replySegmentationStyle transferText classificationQuestion and answeringA note about using mobile GPUsAn overview of federated learning at the edgeTensorFlow FL APIsTensorFlow.jsVanilla TensorFlow.jsConverting modelsPretrained modelsNode.jsSummaryReferences
An introduction to AutoML
What is AutoML?Achieving AutoMLAutomatic data preparationAutomatic feature engineeringAutomatic model generationAutoKerasGoogle Cloud AutoMLUsing Cloud AutoML ‒ Tables solutionUsing Cloud AutoML ‒ Vision solutionUsing Cloud AutoML ‒ Text Classification solutionUsing Cloud AutoML ‒ Translation solutionUsing Cloud AutoML ‒ Video Intelligence Classification solutionCostBringing Google AutoML to KaggleSummaryReferences
The Math Behind Deep Learning
HistorySome mathematical toolsDerivatives and gradients everywhereGradient descentChain ruleA few differentiation rulesMatrix operationsActivation functionsDerivative of the sigmoidDerivative of tanhDerivative of ReLUBackpropagationForward stepBackstepCase 1 – From hidden layer to output layerCase 2 ‒ From hidden layer to hidden layerLimit of backpropagationCross entropy and its derivativeBatch gradient descent, stochastic gradient descent, and mini-batchBatch Gradient Descent (BGD)Stochastic Gradient Descent (SGD)Mini-Batch Gradient Descent (MBGD)Thinking about backpropagation and convnetsThinking about backpropagation and RNNsA note on TensorFlow and automatic differentiationSummaryReferences
Tensor Processing Unit
C/G/T processing unitsCPUs and GPUsTPUsThree generations of TPUs and Edge TPUFirst-generation TPUSecond-generation TPUThird-generation TPUEdge TPUTPU performanceHow to use TPUs with ColabChecking whether TPUs are availableLoading data with tf.dataBuilding a model and loading it into the TPUUsing pretrained TPU modelsUsing TensorFlow 2.1 and nightly buildSummaryReferences
Other Books You May Enjoy
Index

Content preview from Deep Learning with TensorFlow 2 and Keras - Second Edition

7 Word Embeddings

In the last few chapters, we talked about convolutional networks and GANs, which have been very successful against image data. Over the next few chapters, we will switch tracks to focus on strategies and networks to handle text data.

In this chapter, we will first look at the idea behind word embeddings, and then cover the two earliest implementations – Word2Vec and GloVe. We will learn how to build word embeddings from scratch using gensim on our own corpus, and navigate the embedding space we created.

We will also learn how to use third party embeddings as a starting point for our own NLP tasks, such as spam detection, that is, learning to automatically detect unsolicited and unwanted emails. We will then learn about various ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Advanced Deep Learning with TensorFlow 2 and Keras - Second Edition

Publisher Resources

ISBN: 9781838823412

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Deep Learning with TensorFlow 2 and Keras - Second Edition

by Antonio Gulli, Dr. Amita Kapoor, Sujit Pal

7

Word Embeddings

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.