book

Deep Learning with C++

Name: Deep Learning with C++
ISBN: 9781835880029

by Bill Chen, Vikash Gupta

April 2026

Intermediate

610 pages

14h 54m

English

Packt Publishing

Read now

Unlock full access

Preface
Who this book is forWhat this book coversTo get the most out of this bookGet in touchFree benefits with your bookHow to UnlockShare Your Thoughts
Foundations of Deep Learning in C++
Introduction to Deep Learning with C++ and Environment Setup
Your purchase includes a free PDF copy + exclusive extrasTechnical requirementsIntroduction to deep learning conceptsNeural networks – the building blocks of deep learningTypes of learning paradigmsSupervised learningUnsupervised learningSemi-supervised learning and reinforcement learningAdvantages of using C++ for deep learningSetting up a C++ environment for deep learningUsing the free Google ColabSetting up the C++ development environment on-premisesKey performance benefits of C++ in deep learningImplementing a simple neural network in C++SummaryQuestionsFurther readingAnswersGet this book’s PDF version and more
Data Preparation and Preprocessing in C++
Technical requirementsUnderstanding the importance of data preprocessing in DLThe role of preprocessingStructured versus unstructured data – comprehensive techniques and examplesStructured dataDetailed examplesLibrary-powered preprocessing (same techniques, less code)Unstructured dataText dataImage dataAudio/video dataExploring normalization and augmentation techniquesNormalization strategiesAugmentation techniquesIntegrating the PyTorch Dataset API for data pipelinesManaging large-scale datasetsMemory mappingBatch processingSummaryQuestionsFurther readingAnswersGet this book’s PDF version and more
CUDA for GPU Acceleration in Deep Learning with C++
GPU-accelerated versus CPU-only applicationsInstalling and setting up CUDAInstalling the NVIDIA driverDownloading and installing the CUDA ToolkitInstalling additional dependenciesUsing cloud-based CUDA environmentsOptimizing C++ code with CUDACUDA programming modelCPU baseline: array addition in C++Converting to CUDA codeLaunching the CUDA kernelFull CUDA implementationProfiling CUDA performanceCUDA thread hierarchySpeeding up with multiple threadsScaling up with multiple blocksHow to debug in CUDAChecking CUDA API callsChecking kernel launch errorsChecking asynchronous execution errorsCUDA error-handling functionSummaryQuestionsFurther readingAnswersGet this book’s PDF version and more
Building and Training Neural Networks in C++
Building a Basic Neural Network in C++
Understanding the basicsBuilding the forward pass and losses (e.g., BCE/cross-entropy for classification)Implementing neurons and activations in C++ with LibTorchDeriving/backpropagating gradients and updating weights with Stochastic Gradient DescentSummaryQuestionsFurther readingAnswersGet this book’s PDF version and more
Multilayer Perceptron’s in C++
Technical requirementsUnderstanding MLP architectureImplementing MLP in C++Layer classMultilayerPerceptron classImplementing MLP using CUDAImplementing MLP using LibTorchChoosing your MLP implementation approach: A strategic guideRaw C++ with EigenCUDA: maximum performance at significant costLibTorch: practical default for production systemsThe hybrid approachExploring deep learning training conceptsActivation functionsSigmoid familyReLU familyOther advanced functionsOptimization techniquesGradient descentRMS PropMomentumADAM optimizerAdaGradAdaDeltaSummaryFurther readingGet this book’s PDF version and more
Convolutional Neural Networks in C++
History of CNNsMathematical foundations of convolutionImplementation in C++Forward pass implementationIntegration with larger networksMatrix-based implementation with EigenGPUS-accelerated implementation with CUDAConvolutionLayer using PyTorch C++ APIBuilding a C++-based neural network for image classificationImage segmentation with U-NetImage processing for deep learningRotationTranslationCroppingScalingZoomingFlippingPaddingResamplingTerminologies associated with CNNPoolingStrideFeature mapsSummaryFurther readingGet this book’s PDF version and more
Recurrent Neural Networks and Long Short-Term Memory Networks in C++
Understanding time series and sequential dataRNN architectures and mathematicsTypes of RNNsMathematical foundationsMemory dynamics and information flowC++ implementation for RNNsEigen-based approachBPTTMathematical foundation for BPTTGradient flow through timeParameter gradient accumulationPractical implementation considerationsC++ implementation for BPTTVector-based implementationEigen-based implementationLibtorch-based implementationLSTM networksThe cell state: LSTM’s memory highwayLSTM mathematical formulationInformation flow dynamicsC++ implementationVector-based implementationMatrix-based implementationLibtorch-based implementationLSTM backpropagationMathematical framework for LSTM backpropagationGradient flow through gatesParameter gradient accumulationPractical implementation considerationsText processingEnhanced string operations in modern C++File I/O operations for text dataBasic string manipulation and processingLanguage-specific considerationsConvert to lowercaseRemove punctuation and special charactersSplit sentences into wordsStop words in sequential text processingWord embeddings: Bridging discrete tokens and continuous representationsWord2Vec: Learning semantic representationsMathematical formulationC++ implementation with LibTorchIntegration with LSTM networksText processing pipeline for deep learningApplications of RNN on language tasksBuilding a sequence-to-sequence translator with LibTorchVocabulary construction and token managementThe encoder-decoder forward passTraining with teacher forcingInference through autoregressive decodingSummaryFurther readingGet this book’s PDF version and more

Generative Networks, Autoencoders, and Large Language Models in C++
Introduction to generative modelsAutoencoders: foundation of generative learningBasic autoencoder architectureLoss functions and reconstruction errorC++ implementation using LibTorchForward pass implementationTraining loopVAEsThe reparameterization trickC++ implementation of VAEGANs – fundamentalsGAN loss functionsDiscriminator lossGenerator lossGAN training processAlternative loss functionsC++ implementationGenerator network architectureDiscriminator network architectureTraining a GANAutoregressive generation and causal language modelingTemperature and sampling strategiesTemperature scalingSampling strategyGreedy decodingBeam searchTop-K samplingTF-IDFN-GramsEvaluation metrics for generative modelsCategories of evaluation metricsCommon statistical metricsModel-based metricsBERTScoreSummaryFurther readingGet this book’s PDF version and more
Transformers and Large Language Model Fine-Tuning in C++
Introduction to transformer architectureThe attention mechanism revolutionSelf-Attention mechanismMathematical foundations of self-attentionC++ implementation of self-attentionMulti-Head attentionC++ implementation of multi-head attentionPositional encodingLearned position embeddingsSinusoidal position embeddingRoPEMathematical mechanism: pairing even and odd dimensionsWhy the dot product depends on relative positionAlternative modern approachesImplementation considerationsComplete transformer implementationEncoder stack implementationProcessing flowSelf-AttentionAdd and normalizeFeed-Forward networkStacking the layersDecoder stack implementationProcessing flowMulti-head cross-attentionFeed-Forward network and layer stackingPutting it all togetherTraining versus inferencePre-trained language modelsBERT: bidirectional context understandingGPT: autoregressive text generationArchitectural differences between BERT and GPTModel parallelizationDDPUnderstanding DDP implementation: code walkthroughDDPTrainer classFully Sharded Data ParallelUnderstanding FSDP: code walkthroughModel compression techniquesKnowledge distillationThe training process: dual objectivesPruningPipeline overviewQuantizationRange selection strategiesPost-Training quantization (PTQ)Quantization-Aware training (QAT)C++ implementation of quantization for neural networksSummaryFurther readingGet this book’s PDF version and more
Deploying, Monitoring, and Explaining Deep Learning Systems in Production
Deploying and Optimizing Models for Inference
Exporting models for C++ inferenceA compact model in the C++ FrontendProducing a TorchScript artifact (from C++)Loading and running TorchScript in C++Parity: does the artifact match the native graph?Consuming ONNX from C++ for portabilityWhat to keep consistentDeploying models to real targetsPackaging the binary (the one you ship)The heart of serving: a Micro-Batcher with Concurrency ControlConcurrency controls (keeping p95/p99 honest)Cloud, On-Prem, Edge: what actually changes?Optimizing models for speed and costQuantization: fewer bits, more throughputFP16 mixed precision (TorchScript, CUDA)INT8 (ONNX Runtime, Q/DQ graphs)Pruning: remove structure, not just valuesDistillation: deploy the studentRuntime tuning: feeding the hardware efficientlyLayout, threads, algorithms (LibTorch)Runtime tuning with ONNX RuntimeTie it together: a simple, repeatable benchmarkPractical playbookServing models in real timeContract at the BoundaryThe Scheduling Layer: Micro-Batching with Concurrency ControlHTTP and gRPC as the Service EdgeHTTP (REST and Server-Sent Events)gRPC (Unary and Server-Streaming)Tail-Latency DisciplineCloud, On-Prem, and EdgeA Minimal End-to-End ShapeOperating and evolving production systemsMake Behavior VisibleDetect Drift Before Users DoRelease Safely: Shadow, Canary, Roll BackClose the Loop: Retraining on PurposeReliability, Cost, and GovernanceA Stable System is a Measured SystemSummaryQuestionsFurther readingAnswersGet this book’s PDF version and more
Debugging and Retraining Deployed Models
Identifying and handling driftObserving drift in practiceBuilding always-on drift monitorsChoosing the right responseHandling delayed labelsOperationalizing drift monitoringDebugging with evidenceStructured logsIn-process metricsTracesMaking failures reproducibleIsolating the class of bugNumerical bugsData contract bugsConcurrency and synchronization bugsSeparating queue time from compute timeUsing an evidence-first runbookRetraining on purposeDeciding when to retrainMaking retraining deterministicEvaluating models under serving constraintsPromoting safelyClosing the loop after promotionTwo concrete scenariosSummaryQuestionsFurther readingAnswersGet this book’s PDF version and more
Monitoring Deployed Models
Importance of model monitoringFrom aspiration to contract: defining useful SLIs and SLOsInterpreting performance, quality, and resource signalsTurning monitoring into practiceKey metrics to trackOnline accuracy & calibrationHow to instrument (C++)Reading the numbersLatency & throughputHow to instrument (C++)Reading the numbersCPU, GPU, and memory utilizationHow to instrument (C++)Reading the numbersHow metrics drive action (tying to SLOs)C++ instrumentation: Prometheus-style metrics, structured logs, and traces/spansFormat, design principles, and a minimal registryDesign principlesMetric primitives (metrics.hpp)LabelsRegistryExporting /metrics (tiny HTTP handler)Structured logsTraces and spansPutting the pieces togetherOnline quality in the real worldDelayed labels: join later, not in the hot pathLeading indicators: quality signals that don’t need labelsCohorting: make problems small before they get bigFrom signals to actionWiring online quality into telemetry you already exposeFrom signals to actionFrom SLI to policyDrift budgetsHooking alerts to actionsTriggering slower loops with guardrailsTest your policies before production doesSummaryQuestionsFurther readingAnswersGet this book’s PDF version and more
Explainability and Transparency in Deep Learning Models
Understanding the importance of explainability in deep learningIdentifying when explanations are requiredUnderstanding what different stakeholders needConnecting explainability to trust, safety, and regulationApplying practical design principlesOverview of core techniques (LIME, SHAP, Grad‑CAM)LIME – local, model-agnostic explanations for a single predictionShap: Additive attributions with game-theoretic guaranteesGrad-CAM showing where a CNN focuses for a class decisionChoosing and combining methods under real constraintsImplementing LIME and SHAP in C++From prediction function to explainerLIME: local surrogates require believable perturbationsSHAP: KernelSHAP as a practical, model-agnostic estimatorKeeping explanations inside your SLOsWhat to show (and what not to)Troubleshooting faithfulnessUsing Grad-CAM for CNNsBuilding a minimal CNN that exposes the target convolutional featuresUsing OpenCV helpers for image-to-tensor conversion and overlayComputing the Grad-CAM map from activations and gradientsRunning an end-to-end demoApplying Grad-CAM on a real networkEthical considerationsCommunicating uncertainty without false precisionAvoiding misleading attributionDocumenting known failure modesHuman override and audit trailsPutting it into practiceSummaryQuestionsFurther readingAnswersGet this book’s PDF version and more
Unlock Your Exclusive Benefits
Unlock this Book’s Free Benefits in 3 Easy StepsStep 1Step 2Step 3Need Help
Other Books You May Enjoy
Share Your Thoughts
Index

Content preview from Deep Learning with C++

10 Deploying and Optimizing Models for Inference

You’ve trained a model that performs brilliantly offline. Now comes the hard part: shipping it—and keeping it fast, reliable, and cost-effective as usage grows. This chapter is about inference in the real world: turning a checkpoint into a predictable C++ service that meets strict latency/throughput Service Level Objectives (SLOs), scales cleanly, and stays healthy over time.

By “inference,” we mean running a trained model to produce predictions; “serving” is exposing that computation behind an API or stream while managing versions, scaling, and observability. We’ll cover the common modes you’ll encounter: online/real-time (low-latency requests), streaming (e.g., token output for LLMs), batch ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Deep Learning with Python, Second Edition

Publisher Resources

ISBN: 9781835880029

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Deep Learning with C++

by Bill Chen, Vikash Gupta

10

Deploying and Optimizing Models for Inference

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.