book

AWS Certified Data Engineer Associate Study Guide

by Sakti Mishra, Dylan Qu, Anusha Challa

August 2025

Intermediate to advanced

476 pages

12h 52m

English

O'Reilly Media, Inc.

Book available

Read now

Unlock full access

What This Book Isn’tWhat This Book Is AboutWho Should Read This BookHow This Book Is OrganizedAccessing the Book’s Images OnlineConventions Used in This BookO’Reilly Online LearningHow to Contact UsAcknowledgments
Who Is a Data Engineer?Becoming an AWS Data Engineer AssociateExam TopicsExam FormatRegistering for the ExamExam-Style QuestionsThink Like an AWS Solutions Architect: Translating a Real-World Problem-Solving Framework into CertificationThe Solutions Architect’s Problem-Solving FrameworkReal-World Example: Designing a Serverless Stream Analytics Platform to Detect FraudHow This Thought Process Applies to Certification QuestionsStudy PlanConclusion
Databases and Types of DatabasesWhat Is a Database?What Is a Database Management System?Types of DatabasesHierarchical DatabasesRelational DatabasesNoSQL DatabasesOLTP Versus OLAPOverview of Big DataDistributed Processing Frameworks for Big DataMapReduceSparkFlinkHivePrestoTrinoWhat Is a Data Lake?What Is a Data Warehouse?Data Warehouse Versus Data LakeETL Versus ELTDifferent Ways to Process DataBatch Processing PipelineReal-Time Stream ProcessingEvent-Driven ProcessingHigh-Level Architecture Overview of Data Processing PipelinesWorking with Code RepositoriesWhat Is a Code Repository?How to Work with Code RepositoriesCI/CDCloud Computing and AWSWhat Is Cloud Computing?An Overview of Amazon Web ServicesGetting Started with AWSHow to Set Up an AWS AccountConfigure Access with AWS IAMCreate an IAM User for AuthenticationAdd Permissions to Authorize the UserWhat Is an IAM Policy?What Is an IAM Role?Best Practices to Follow with AWS IAMConclusionResources
AWS Analytics ServicesAmazon Kinesis Data StreamsAmazon Data FirehoseAmazon Managed Service for Apache FlinkAmazon Managed Streaming for Apache KafkaReference Architecture: Streaming Analytics Pattern with Apache Flink and MSKAWS GlueAWS Glue DataBrewAmazon AthenaAmazon EMRAmazon RedshiftAmazon QuickSightReference Architecture: Lakehouse with Glue, Redshift, and AthenaAmazon OpenSearch ServiceAmazon DataZoneAWS Lake FormationAuxiliary Services for AnalyticsApplication IntegrationCompute and ContainersDatabaseStorageMachine LearningMigration and TransferNetworking and Content DeliverySecurity, Identity, and ComplianceManagement GovernanceDeveloper ToolsCloud Financial ManagementAWS Well-Architected ToolConclusionAdditional Resources
Data IngestionReal-Time Streaming Data IngestionKinesis Data Streams Versus Amazon MSKSample Streaming Ingestion Use CasesIngesting Data Using Zero-ETL IntegrationsIngesting Data from Databases with CDC Using AWS Data Migration ServiceSupported Sources for AWS DMSSupported Targets for AWS DMSSample Use CasesBest Practices for Data IngestionBest Practices for Streaming IngestionBest Practices for Choosing Data Stream Capacity ModeBest Practices for ShardingBest Practices for Consuming Data from KDSBest Practices for Amazon MSKBest Practices for Amazon Data FirehoseBest Practices for AWS DMS Replication Instances and TasksBest Practices for AWS DMS Tasks with Amazon Redshift TargetData TransformationBatch Data TransformationStreaming Data TransformationData Transformation Using AWS GlueGlue ConnectorsGlue BookmarksData Processing UnitsWorker TypeGlue JobsData Sources and DestinationsBest Practices for AWS GlueData Transformation Using Amazon EMRStorageDeployment OptionsInstance TypesBest Practices for Amazon EMRAWS Glue Versus Amazon EMR OptionsSQL-Based Data Transformation Using Amazon RedshiftAmazon Redshift ComputeAmazon Redshift StorageSQL Data TransformationsAmazon Managed Service for Apache FlinkAmazon Data Firehose for TransformationAWS Lambda for TransformationChoosing the Right Streaming Transformation ServiceChoosing the Right Batch Transformation ServiceData Preparation for Nontechnical PersonasFill Missing ValuesIdentify Duplicate RecordsFormatting FunctionsIntegrating Data from Multiple SourcesNesting and Unnesting Data StructuresProtecting Sensitive DataOther Data Preparation TransformationsOrchestrating Data PipelinesAWS Step FunctionsManaged Workflows for Apache AirflowSample Use CaseAWS Glue WorkflowsSample Use CaseAmazon Redshift SchedulerAmazon EventBridgeSample Use CaseChoosing the Right Orchestration ServiceConclusionPractice QuestionsAdditional Resources
Choosing a Data StoreAWS Core Storage ServicesAWS Cloud DatabasesData Storage Formats for Data LakesRow-Based File FormatsColumn-Based File FormatsTable FormatsBuilding a Data Strategy with Multiple Data StoresData Cataloging SystemsComponents of Metadata and Data CatalogsPopulating an AWS Glue Data CatalogData Catalog Best PracticesEnriching Data Catalogs with Data ClassificationManaging the Lifecycle of DataSelecting Storage Solutions for Hot and Cold DataExample: Building a Petabyte-Scale Log Analytics Solution on AWSStorage Tier Decisions for Different Access PatternsDefining Data Retention Policy and Archiving StrategiesPerforming COPY and UNLOAD Operations to Move Data Between Amazon S3 and Amazon RedshiftOptimizing Data Management with Amazon S3Overview of S3 Storage ClassesChoosing the Right Storage ClassS3 Intelligent-TieringManaging the Data Lifecycle with Amazon S3 LifecycleMonitoring the Amazon S3 Data LifecycleExpiring Snapshots from Open Table FormatsArchiving Data from Amazon DynamoDB to Amazon S3Ensuring S3 Data Resiliency with S3 VersioningEnabling Versioning on an S3 BucketS3 Versioning and Object Lifecycle ManagementDesigning Data Models and SchemaIntroduction to Data ModelingData Modeling Strategies for Amazon RedshiftData Modeling Strategies for Amazon DynamoDBData Modeling Strategies for Data LakesAmazon S3 Data Lake Best PracticesConclusionPractice QuestionsAdditional Resources
Amazon QuickSightData SourcesDatasetsRefreshing SPICE DatasetsVisualizationsPresentation FormatsQuickSight GenBI Capabilities (QuickSight Q)SQL Analytics Using Amazon AthenaChoice of Querying EngineWorkgroupsCapacity ReservationsAthena Federated SQLUse CasesDDL CapabilitiesBest Practices When Using Amazon AthenaSQL Analytics Using Amazon RedshiftSQL FunctionsSemi-Structured Data AnalysisGeospatial Data AnalysisQuery Data from Data LakeAnalyzing Data from Operational Data Stores Using Amazon RedshiftRedshift ML and Generative AIUser-Defined FunctionsAnalyzing Data Using NotebooksAWS Glue Interactive SessionsAmazon EMR NotebooksData Pipeline ResiliencyMonitoringAlertingEvent-Driven Pipeline Maintenance with EventBridgeEnsuring Data Quality and Reliability: Deequ and DQDLAutomated Data Quality Checks and Error HandlingTroubleshooting and Performance TuningCI/CD PipelinesVersion Control and CollaborationInfrastructure as CodeDisaster Recovery and High AvailabilityCost Optimization for Data PipelinesLeveraging Serverless ServicesAutoscalingTiered StorageColumnar FormatsMonitor and Control Data Transfer CostsFollow Cost Optimization Best PracticesConclusionPractice QuestionsAdditional Resources
Network SecurityAmazon VPC OverviewSecurity Groups OverviewBest Practices for Configuring Security Groups for Your WorkloadsConfiguring a VPC and Security Group for an Amazon EMR ClusterManaged Services Versus Unmanaged ServicesVPC Endpoints OverviewUser Authentication and AuthorizationAuthenticating Users with IAM CredentialsIAM Role-Based Authentication and AuthorizationService-Linked RolesManaged Versus Self-Managed PoliciesEnable Single Sign-on with AWS IAM Identity CenterData Security and PrivacySecure Data in Amazon S3Manage Database CredentialsData Encryption and Decryption and Managing the Encryption KeysManaging Encryption Keys with AWS KMSEnabling Encryption in AWS Analytics ServicesSensitive Data Detection and RedactionFine-Grained Access Control with AWS Lake FormationDatabase Security in Amazon RedshiftFine-Grained Access Control in Amazon QuickSightData GovernanceMetadata Management and Technical CatalogData SharingData QualityData ProfilingData Lifecycle ManagementData LineageLogging and AuditingAnalyzing Logs Using AWS ServicesConclusionPractice QuestionsAdditional Resources
Data Processing PipelineImplementing a Batch Processing PipelineUse Case and Architecture OverviewOverview of Input DatasetStep-by-Step Implementation GuideBest Practices and Optimization TechniquesImplementing a Real-Time Streaming PipelineUse Case and Architecture OverviewStep-by-Step Implementation GuideConclusionResources

Amazon SageMaker Unified StudioAmazon SageMaker CatalogAmazon SageMaker LakehouseAmazon SageMaker AIAmazon S3 TablesAmazon S3 MetadataImproving the Developer Experience with Generative AIGenerative AI–Powered Code Generation with Amazon Q DeveloperAutomated Script Upgrade in AWS GlueGenAI-Powered Troubleshooting for Spark in AWS GlueConclusionResources
Chapter 4Chapter 5Chapter 6Chapter 7Chapter 9

Content preview from AWS Certified Data Engineer Associate Study Guide

Chapter 10. What’s New in AWS for Data Engineers

Chapters 1 to 9 provided details on established AWS services and concepts that have been widely available since late 2024. AWS has since introduced several new features, with some currently in the preview phase and others having recently achieved general availability status.

This chapter aims to provide a high-level overview of some of these new features, which will help you get familiar with the new capabilities and answer questions related to these features should they appear in the certification exam.

The following are the new announcements that change how developers integrate AWS data analytics services:

Amazon SageMaker Unified Studio
Amazon SageMaker Catalog
Amazon SageMaker Lakehouse
Improving the developer experience with generative AI
Amazon S3 Tables and S3 Metadata

Let’s get an overview of each of these capabilities.

Amazon SageMaker Unified Studio

As explained in previous chapters, Amazon Web Services offer a wide range of data analytics services (e.g., Amazon EMR, AWS Glue, Amazon Athena, Amazon Redshift, and more) that customers can integrate to build an end-to-end data pipeline. These services help address analytical needs, but they require effort to assemble or require in-depth knowledge to integrate with each other through their respective service interfaces. To address the ease-of-use concern, during re:Invent 2024, AWS announced Amazon SageMaker Unified Studio.

Amazon SageMaker Unified Studio provides an integrated ...