book

Modern Computer Vision with PyTorch - Second Edition

by V Kishore Ayyadevara, Yeshwanth Reddy

June 2024

Intermediate to advanced

746 pages

17h 59m

English

Packt Publishing

Read now

Unlock full access

Who this book is forWhat this book coversTo get the most out of this bookGet in touch
Comparing AI and traditional machine learningLearning about the ANN building blocksImplementing feedforward propagationCalculating the hidden layer unit valuesApplying the activation functionCalculating the output layer valuesCalculating loss valuesCalculating loss during continuous variable predictionCalculating loss during categorical variable predictionFeedforward propagation in codeActivation functions in codeLoss functions in codeImplementing backpropagationGradient descent in codeImplementing backpropagation using the chain rulePutting feedforward propagation and backpropagation togetherUnderstanding the impact of the learning rateLearning rate of 0.01Learning rate of 0.1Learning rate of 1Summarizing the training process of a neural networkSummaryQuestions
Installing PyTorchPyTorch tensorsInitializing a tensorOperations on tensorsAuto gradients of tensor objectsAdvantages of PyTorch’s tensors over NumPy’s ndarraysBuilding a neural network using PyTorchDataset, DataLoader, and batch sizePredicting on new data pointsImplementing a custom loss functionFetching the values of intermediate layersUsing a sequential method to build a neural networkSaving and loading a PyTorch modelUsing state_dictSavingLoadingSummaryQuestions
Representing an imageConverting images into structured arrays and scalarsCreating a structured array for colored imagesWhy leverage neural networks for image analysis?Preparing our data for image classificationTraining a neural networkScaling a dataset to improve model accuracyUnderstanding the impact of varying the batch sizeBatch size of 32Batch size of 10,000Understanding the impact of varying the loss optimizerBuilding a deeper neural networkUnderstanding the impact of batch normalizationVery small input values without batch normalizationVery small input values with batch normalizationThe concept of overfittingImpact of adding dropoutImpact of regularizationL1 regularizationL2 regularizationSummaryQuestions
The problem with traditional deep neural networksBuilding blocks of a CNNConvolutionFiltersStrides and paddingStridesPaddingPoolingPutting them all togetherHow convolution and pooling help in image translationImplementing a CNNClassifying images using deep CNNsVisualizing the outcome of feature learningBuilding a CNN for classifying real-world imagesImpact on the number of images used for trainingSummaryQuestions
Introducing transfer learningUnderstanding the VGG16 architectureImplementing VGG16Understanding the ResNet architectureImplementing ResNet18Implementing facial keypoint detection2D and 3D facial keypoint detectionImplementing age estimation and gender classificationIntroducing the torch_snippets librarySummaryQuestions
Generating CAMsUnderstanding the impact of data augmentation and batch normalizationCoding up road sign detectionPractical aspects to take care of during model implementationImbalanced dataThe size of the object within an imageThe difference between training and validation dataThe number of nodes in the flatten layerImage sizeOpenCV utilitiesSummaryQuestions
Introducing object detectionCreating a bounding-box ground truth for trainingUnderstanding region proposalsLeveraging SelectiveSearch to generate region proposalsImplementing SelectiveSearch to generate region proposalsUnderstanding IoUNon-max suppressionMean average precisionTraining R-CNN-based custom object detectorsWorking details of R-CNNImplementing R-CNN for object detection on a custom datasetDownloading the datasetPreparing the datasetFetching region proposals and the ground truth of offsetCreating the training dataR-CNN network architecturePredicting on a new imageTraining Fast R-CNN-based custom object detectorsWorking details of Fast R-CNNImplementing Fast R-CNN for object detection on a custom datasetSummaryQuestions

Components of modern object detection algorithmsAnchor boxesRegion proposal networkClassification and regressionTraining Faster R-CNN on a custom datasetWorking details of YOLOTraining YOLO on a custom datasetInstalling DarknetSetting up the dataset formatConfiguring the architectureTraining and testing the modelWorking details of SSDComponents in SSD codeSSD300MultiBoxLossTraining SSD on a custom datasetSummaryQuestions
Exploring the U-Net architecturePerforming upscalingImplementing semantic segmentation using U-NetExploring the Mask R-CNN architectureRoI AlignMask headImplementing instance segmentation using Mask R-CNNPredicting multiple instances of multiple classesSummaryQuestions
Multi-object instance segmentationFetching and preparing dataTraining the model for instance segmentationMaking inferences on a new imageHuman pose detectionCrowd countingImplementing crowd countingImage colorization3D object detection with point cloudsTheoryInput encodingOutput encodingTraining the YOLO model for 3D object detectionData formatData inspectionTrainingTestingAction recognition from videoIdentifying an action in a given videoTraining a recognizer on a custom datasetSummaryQuestions
Understanding autoencodersHow autoencoders workImplementing vanilla autoencodersImplementing convolutional autoencodersGrouping similar images using t-SNEUnderstanding variational autoencodersThe need for VAEsHow VAEs workKL divergenceBuilding a VAEPerforming an adversarial attack on imagesUnderstanding neural style transferHow neural style transfer worksPerforming neural style transferUnderstanding deepfakesHow deepfakes workGenerating a deepfakeSummaryQuestions
Introducing GANsUsing GANs to generate handwritten digitsUsing DCGANs to generate face imagesImplementing conditional GANsSummaryQuestions
Leveraging the Pix2Pix GANLeveraging CycleGANHow CycleGAN worksImplementing CycleGANLeveraging StyleGAN on custom imagesThe evolution of StyleGANImplementing StyleGANIntroducing SRGANArchitectureCoding SRGANSummaryQuestions
Learning the basics of reinforcement learningCalculating the state valueCalculating the state-action valueImplementing Q-learningDefining the Q-valueUnderstanding the Gym environmentBuilding a Q-tableLeveraging exploration-exploitationImplementing deep Q-learningUnderstanding the CartPole environmentPerforming CartPole balancingImplementing deep Q-learning with the fixed targets modelUnderstanding the use caseCoding up an agent to play PongImplementing an agent to perform autonomous drivingSetting up the CARLA environmentInstalling the CARLA binariesInstalling the CARLA Gym environmentTraining a self-driving agentCreating model.pyCreating actor.pyTraining a DQN with fixed targetsSummaryQuestions
Introducing transformersBasics of transformersEncoder blockDecoder blockHow ViTs workImplementing ViTsTranscribing handwritten imagesHandwriting transcription workflowHandwriting transcription in codeDocument layout analysisUnderstanding LayoutLMImplementing LayoutLMv3Visual question answeringIntroducing BLIP2Representation learningGenerative learningImplementing BLIP2SummaryQuestions
Introducing CLIPHow CLIP worksBuilding a CLIP model from scratchLeveraging OpenAI CLIPIntroducing SAMHow SAM worksImplementing SAMHow FastSAM worksAll-instance segmentationPrompt-guided selectionImplementing FastSAMIntroducing diffusion modelsHow diffusion models workDiffusion model architectureImplementing a diffusion model from scratchConditional image generationUnderstanding Stable DiffusionBuilding blocks of the Stable Diffusion modelCrossAttnDownBlock2DUNetMidBlock2DcrossAttnCrossAttnUpBlock2DDownBlock2D, UpBlock2DTransformer2DModelResnetBlock2DImplementing Stable DiffusionSummaryQuestions
In-paintingModel training workflowIn-painting using Stable DiffusionControlNetArchitectureImplementing ControlNetSDXL TurboArchitectureImplementing SDXL TurboDepthNetWorkflowImplementing DepthNetText to videoWorkflowImplementing text to videoSummaryQuestions
Understanding the basics of an APICreating an API and making predictions on a local serverInstalling the API module and dependenciesServing an image classifiersdd.pyserver.pyRunning the serverContainerizing the applicationBuilding a Docker imageCreating the requirements.txt fileCreating a DockerfileBuilding a Docker image and creating a Docker containerShipping and running the Docker container on the cloudConfiguring AWSCreating a Docker repository on AWS ECR and pushing the imagePulling the image and building the Docker containerIdentifying data driftUsing vector storesSummaryQuestions

Content preview from Modern Computer Vision with PyTorch - Second Edition

15 Combining Computer Vision and NLP Techniques

In the previous chapter, we learned about applications that combine reinforcement learning and computer vision. In this chapter, we will switch gears and learn about how a convolutional neural network (CNN) can be used in conjunction with algorithms in the broad family of transformers, which are heavily used (as of the time of writing this book) in natural language processing (NLP) to develop solutions that leverage both computer vision and NLP.

To understand combining CNNs and transformers, we will first learn how vision transformers (ViTs) work and how they help in performing image classification. After that, we will learn about leveraging transformers to perform the transcription of handwritten ...