book

AI Engineering Interviews

by Mina Ghashami, Ali Torkamani

December 2026

Beginner to intermediate

484 pages

12h 41m

English

O'Reilly Media, Inc.

Read now

Unlock full access

I. Practical AI Engineering
1. Hello AI Engineering!
GenAI: The Core of AI EngineeringLanguage Modeling: The Core of GenAIThe Era of Statistical Language ModelingThe Era of Neural Language ModelingThe Era of Large Language Models (LLMs)Large Language ModelsIndustry Landscape and CommunitiesOpen-Weight vs Closed ModelsArchitecturesEncoder-Only ModelsDecoder-Only ModelsEncoder-Decoder ModelsBuilding Large Language ModelsStage 1: PretrainingStage 2: Fine-TuningStage 3: AlignmentStage 4: Inference and DeploymentEvaluation as a Continuous ProcessCurrent State of PerformancePerformance Among Pre-trained ModelsPerformance Among Instruction-Tuned ModelsWhat the Leaderboard Doesn’t Tell YouEmerging TrendsCareer Paths in Generative AIRoles, Responsibilities, and Required SkillsInterview Types and PreparationSummaryInterview Questions and AnswersThread: Why AI?Thread: Career Tracks and Role SelectionThread: Model Landscape and SelectionThread: LLM Lifecycle and System Thinking
2. Pretrained Models
Why Pretraining MattersPretraining DatasetsHow Pretraining WorksHow Text Becomes TokensScaling Laws and ComputeMixture of ExpertsEvolution of Pretrained Model FamiliesChoosing the Right Pretrained ModelTask RequirementsDeployment ConstraintsLicensing and OpennessA Practical Decision FrameworkReading the LeaderboardsSummaryInterview Questions and AnswersThread: The Pretrained Model LandscapeThread: Scaling Laws and Training DecisionsThread: Mixture of ExpertsThread: The Frontier and How to Track ItThread: TokenizationThread: Synthetic Data, Bias, and Practical Deployment
3. Prompt Engineering
What Is Prompt Engineering?From Prompt Engineering to Context EngineeringAnatomy of the Context WindowSystem promptUser messageFew-shot examplesRetrieved contextTool definitionsConversation historyOutput schemaFundamentals of Effective PromptingElements of an effective promptControlling Output VariabilityHandling edge casesPrompting TechniquesIn-Context LearningZero-shot promptingFew-shot promptingRole, style, and emotion promptingRephrase and RespondThought GenerationChain-of-thought promptingStep-Back promptingProgram-of-ThoughtsAutomatic Chain-of-ThoughtDecompositionLeast-to-Most promptingSkeleton-of-ThoughtPrompt chainingEnsemblingSelf-consistencyTree-of-thoughtGenerated knowledgeSelf-Criticism and VerificationSelf-refinementChain-of-VerificationSelf-CalibrationPrompting with External SystemsRAG as context engineeringFrom prompts to agents: ReAct and beyondChallenges and Best PracticesPrompt sensitivityPrompt injection and jailbreakingTesting and iterationAutomated prompt optimizationSummaryInterview Questions and AnswersThread: Foundations and Context EngineeringThread: In-Context Learning TechniquesThread: Reasoning and Thought GenerationThread: Decomposition and Self-CriticismThread: Building Context and Multi-Step PipelinesThread: Production, Security, and Best Practices
4. RAG and Agents
Retrieval-Augmented Generation (RAG)Adaptive RAG PatternsAgentic RAGAgents and Tool UseStructured Workflow PatternsAgent LoopToolsModel Context Protocol (MCP)Core Design ChoicesProduction Reality CheckEvaluationRAG Evaluation TriadAgent EvaluationSummaryInterview Questions and AnswersThread: RAG FundamentalsThread: RAG Evaluation and DebuggingThread: From Static RAG to Agentic RAGThread: Agents, Tools, and the Agent LoopThread: Agent Architecture and OrchestrationThread: Production Operations
5. Evaluation
Evaluation AxesAutomated MetricsPerplexityReference-Based Text MetricsStandard Capability BenchmarksGeneral Knowledge BenchmarksMath and Reasoning BenchmarksCode Generation BenchmarksAggregated LeaderboardsHallucination EvaluationTypes of HallucinationDetection MethodsLLM-as-Judge EvaluationKnown BiasesBest PracticesHuman EvaluationChatbot Arena and Elo RatingsStructured Human EvaluationRed-Teaming and Adversarial EvaluationBenchmark ContaminationDynamic BenchmarksEvaluation in PracticeGolden Test SetCI/CD IntegrationShadow DeploymentsProduction MonitoringSummaryInterview Questions and AnswersThread: What to MeasureThread: Benchmarks and Their Shelf LifeThread: When Models LieThread: Humans in the LoopThread: Breaking the ModelThread: Evaluation in ProductionThread: The Bigger Picture
II. Foundations of Modern Generative AI
6. Machine Learning Foundation
From Neurons to Deep NetworksUniversal ApproximationLayer Types in TransformersActivation FunctionsLearning ParadigmsLoss FunctionsOptimization and Training StabilityRegularization and GeneralizationSummaryInterview Questions and AnswersThread: Neural Network ArchitectureThread: Learning ParadigmsThread: Loss FunctionsThread: OptimizationThread: Training Practicalities
7. Transformer Architecture
Transformer Architecture TypesTransformer BlockAttention MechanismSelf-AttentionMasked Self-AttentionCross AttentionMulti-Head AttentionFeed-Forward Network (FFN)Residual Connection and Layer NormalizationPositional EncodingAbsolute Positional EncodingRelative Positional EncodingRotary Positional Embeddings (RoPE)Attention with Linear Biases (ALiBi)Comparing Positional Encoding MethodsSummaryInterview Questions and AnswersThread: Why Transformers?Thread: Attention from First PrinciplesThread: Inside a Transformer BlockThread: Positional EncodingThread: Architecture Types and Why Decoder-Only WonThread: The LLaMA Recipe and Model ComparisonsThread: Emerging Architectures
8. Data Preparation
Data CollectionData CleaningSynthetic Data GenerationCorpus SamplingTokenizationData SchedulingSummaryInterview Questions and AnswersThread: Building a Pretraining CorpusThread: Cleaning and Filtering at ScaleThread: Data Scheduling and CurriculumThread: Tokenization

9. Training of Generative AI Models
Training Core ComponentsDataset and DataLoaderTokenizer and Data CollatorForward Pass, Loss, and BackpropagationTraining OptimizationMixed Precision TrainingActivation CheckpointingGradient AccumulationParallelism in TrainingMixture of Experts (MoE)Architecture and RoutingTraining ChallengesScaling and Systems ConsiderationsDesign Trade-offsKnowledge DistillationSummaryInterview Questions and AnswersThread: The Training Data PipelineThread: Mixed Precision TrainingThread: Memory and Compute OptimizationThread: Training Stability, Monitoring, and ScalingThread: Distributed Training and ParallelismThread: Mixture of ExpertsThread: Knowledge Distillation
10. Fine-Tuning
When to Fine-Tune vs Not?Prompt Engineering: The First StepRAG: When You Need Current or Private InformationFine-Tuning: When You Need Deep CustomizationCombining ApproachesTypes of Fine-TuningWhere the Data Comes FromQuality and GuidelinesParameter-Efficient Fine-Tuning (PEFT)AdaptersPrompt TuningPrefix TuningLoRA (Low-Rank Adaptation)QLoRA (Quantized LoRA)SFT Training BasicsFine-Tuning Knobs That Actually Change OutcomesEvaluation of SFTSummaryInterview Questions and AnswersThread: When and Why to Fine-TuneThread: PEFT Methods ComparedThread: LoRA Deep DiveThread: SFT Training and DataThread: Failure Modes, Evaluation, and Production
11. Alignment
Reinforcement Learning from Human Feedback (RLHF)Stage 1: Policy TrainingStage 2: Reward Model TrainingStage 3: Optimize Policy Against the Reward ModelDirect Preference OptimizationDPO VariantsConstitutional AI and RLAIFAlignment ChallengesReward Hacking and Specification GamingThe Alignment TaxInner vs. Outer AlignmentSafety and Adversarial RobustnessJailbreak AttacksDefense StrategiesScalable OversightEvaluationSummaryInterview Questions and AnswersThread: Alignment and the RLHF PipelineThread: Simplifying the PipelineThread: Scaling Feedback Without HumansThread: When Alignment Goes WrongThread: Safety Under Adversarial PressureThread: The Bigger Picture
12. Reasoning Models
Types of ReasoningApproaches to ReasoningPrompted General-Purpose LLMsTrained Reasoning ModelsNeuro-symbolic ModelsTraining Reasoning ModelsThinking TokensReinforcement Learning with Verifiable Rewards (RLVR)Process Reward Models vs. Outcome Reward ModelsGroup Relative Policy Optimization (GRPO)Emergence of Self-VerificationMulti-Stage Training PipelineReasoning DistillationInference-Time Compute ScalingEvaluation and BenchmarksChallenges and LimitationsError Propagation in Long ChainsOverthinking and UnderthinkingUnfaithful Chain-of-ThoughtLogical InconsistencyReasoning or Pattern Matching?SummaryInterview Questions and AnswersThread: Understanding Reasoning ModelsThread: Training Reasoning ModelsThread: Inference-Time ReasoningThread: Making Reasoning AccessibleThread: The Reasoning LandscapeThread: Challenges, Safety, and Open Questions
13. LLM Inference
Inference FundamentalsPrefill and DecodeKey MetricsDecoding StrategiesAttention and Memory OptimizationArchitecture-Level KV Cache ReductionRuntime KV Cache OptimizationEfficient Attention Computation (FlashAttention)Model CompressionQuantizationPruningFP8 InferenceKnowledge DistillationDecoding AccelerationSpeculative DecodingDraft-Free and Self-Speculative VariantsServing and System-Level OptimizationsGraph Compilation with torch.compileMulti-GPU ParallelismContinuous BatchingPrefill/Decode DisaggregationRequest Scheduling Under LoadMoE Inference ChallengesModel Routing and CascadingInference FrameworksSummaryInterview Questions and AnswersThread: LLM Inference FundamentalsThread: The KV CacheThread: Model CompressionThread: Serving at ScaleThread: Decoding and SpeculationThread: Deployment
14. Long Context Modeling
Why Long Sequences Are HardPositional Encodings for Long-Context GeneralizationEfficient Long-Context TransformersExact Attention With Efficient ComputationApproximate AttentionState Space Models (SSMs)Engineering Long Context in PracticeContext Extension TechniquesContinued pretraining at longer sequence lengthsPositional interpolationPositional scalingExternal memory and recurrenceRetrieval-based extension (RAG as context extension)Inference and Evaluation of Long ContextInference at Long ContextEvaluating Long Context ReliabilitySummaryInterview Questions and AnswersThread: The Long-Context ChallengeThread: Positional Encoding for Long SequencesThread: Efficient Attention MechanismsThread: KV Cache and Memory OptimizationThread: State Space Models and Alternatives to AttentionThread: Context Extension TechniquesThread: Ring Attention and Distributed Long ContextThread: Practical Challenges, Evaluation, and Benchmarks

Content preview from AI Engineering Interviews

Chapter 4. RAG and Agents

In Chapter 3, we looked at prompt engineering as one way to equip LLMs with the information they need. In this chapter, we introduce retrieval-augmented generation (RAG) and show how it evolves into agentic RAG and, ultimately, tool-using agents.

The shortcoming with prompt engineering is that in many real systems, better prompts are not enough. The model may need facts that were never in its training data, details that change frequently, or private information that cannot be included in pretraining. RAG offers a practical and cost-effective alternative; instead of forcing the model to “know” everything, the system retrieves relevant information at runtime and injects it into the model’s context before generation. Figure 4-1 shows the comparison between RAG and prompt engineering.

Diagram comparing Prompt Engineering and Retrieval-Augmented Generation (RAG) workflows, highlighting RAG's advantage of accessing external data to enhance responses and reduce hallucinations.

RAG changed how LLM applications are built. Rather than treating the model as a self-contained knowledge base, modern systems increasingly treat it as a reasoning and synthesis engine that operates over external data. We’ll build up to that perspective in three steps:

First, we cover vanilla RAG (also known as classical RAG or basic RAG). In its simplest form, a RAG system retrieves a set of top-k documents relevant to each user query, augments the user query with them, and ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Publisher Resources

ISBN: 9798341623521Errata Page

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

AI Engineering Interviews

by Mina Ghashami, Ali Torkamani

Chapter 4. RAG and Agents

Figure 4-1. Comparison of prompt engineering and RAG. Both are common ways to adapt an LLM system.

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.