book

Data Engineering with AWS Cookbook

Name: Data Engineering with AWS Cookbook
ISBN: 9781805127284

by Trâm Ngọc Phạm, Gonzalo Herreros González, Viquar Khan, Huda Nofal

November 2024

Intermediate to advanced

528 pages

12h 11m

English

Packt Publishing

Read now

Unlock full access

Data Engineering with AWS Cookbook
Contributors
About the authors
About the reviewers
Preface
Who this book is forWhat this book coversTo get the most out of this bookDownload the example code filesConventions usedSectionsGetting readyHow to do it…How it works…There’s more…See alsoGet in touchShare Your ThoughtsDownload a free PDF copy of this book
Chapter 1: Managing Data Lake Storage
Technical requirementsControlling access to S3 bucketsGetting readyHow to do it…How it works…There’s more…See alsoStorage types in S3 for optimized storage costsGetting readyHow to do it…How it works…There’s more…See alsoEnforcing encryption on S3 bucketsGetting readyHow to do it…How it works…There’s more…See alsoSetting up retention policies for your objectsGetting readyHow to do it…How it works…There’s more…See alsoVersioning your dataGetting readyHow to do it…How it works…There’s more…See alsoReplicating your dataGetting readyHow to do it…How it works…There’s more…See alsoMonitoring your S3 bucketGetting readyHow to do it…How it works…There’s more…See also
Chapter 2: Sharing Your Data Across Environments and Accounts
Technical requirementsCreating read-only replicas for RDSGetting readyHow to do it…How it works…There’s more…See alsoRedshift live data sharing among your clustersGetting readyHow to do it…How it works…There’s more…Synchronizing Glue Data Catalog to a different accountGetting readyHow to do it…How it works…There’s more…See alsoEnforcing fine-grained permissions on S3 data sharing using Lake FormationGetting readyHow to do it…How it works…There’s more…See alsoSharing your S3 data temporarily using a presigned URLGetting readyHow to do it…How it works…There’s more…See alsoReal-time sharing of S3 dataGetting readyHow to do it…How it works…There’s more…See alsoSharing read-only access to your CloudWatch data with another AWS accountGetting readyHow to do it…How it works…There’s more…See also
Chapter 3: Ingesting and Transforming Your Data with AWS Glue
Technical requirementsCreating ETL jobs visually using AWS Glue StudioGetting readyHow to do it…How it works…There’s more…See alsoParameterizing jobs to make them more flexible and reusableGetting readyHow to do it…How it works…There’s more…See alsoHandling job failures and reruns for partial resultsGetting readyHow to do it…How it works…There’s more…See alsoProcessing data incrementally using bookmarks and bounded executionGetting readyHow to do it…How it works…There’s more…See alsoHandling a high quantity of small files in your jobGetting readyHow to do it…How it works…There’s more…See alsoReusing libraries in your Glue jobGetting readyHow to do it...How it works…There’s more...See alsoUsing data lake formats to store your dataGetting readyHow to do it…How it works…There’s more…See alsoOptimizing your catalog data retrieval using pushdown filters and indexesGetting readyHow to do it…How it works…There’s more…See alsoRunning pandas code using AWS Glue for RayGetting readyHow to do it…How it works…There’s more…See also
Chapter 4: A Deep Dive into AWS Orchestration Frameworks
Technical requirementsDefining a simple workflow using AWS Glue workflowsGetting readyHow to do it…See alsoSetting up event-driven orchestration with Amazon EventBridgeGetting readyHow to do it…Creating a data workflow using AWS Step FunctionsHow to do it…See alsoManaging data pipelines with MWAAHow to do it…See alsoMonitoring your pipeline’s healthHow to do it…Setting up a pipeline using AWS Glue to ingest data from a JDBC database into a catalog tableHow to do it…
Chapter 5: Running Big Data Workloads with Amazon EMR
Technical requirementsRunning jobs using AWS EMR serverlessGetting readyHow to do it…How it works…There’s more…See alsoRunning your AWS EMR cluster on EKSGetting readyHow to do it...How it works...There’s more...See alsoUsing the AWS Glue catalog from another accountGetting readyHow to do it...How it works...There’s more...See alsoMaking your cluster highly availableGetting readyHow to do it...How it works...There’s more...See alsoScaling your cluster based on workloadGetting readyHow to do it...How it works...There’s more...See alsoCustomizing the cluster nodes easily using bootstrap actionsGetting readyHow to do it...How it works...There’s more...See alsoTuning Apache Spark resource usageGetting readyHow to do it...How it works...There’s more...See alsoCode development on EMR using WorkspacesHow to do it...There’s more...See alsoMonitoring your clusterGetting readyHow to do it...There’s moreSee alsoProtecting your cluster from security vulnerabilitiesGetting readyHow to do it...How it works...There’s more...See also
Chapter 6: Governing Your Platform
Technical requirementsApplying a data quality check on Glue tablesGetting readyHow to do it…How it works…There’s more…See alsoAutomating the discovery and reporting of sensitive data on your S3 bucketsGetting readyHow to do it…How it works…There’s more…See alsoEstablishing a tagging strategy for AWS resourcesGetting readyHow to do it…How it works…There’s more…See alsoBuilding your distributed data community with Amazon DataZone following data mesh principlesGetting readyHow to do it…How it works…There’s more…See alsoHandling security-sensitive data (PII and PHI)Getting readyHow to do it…How it works…There’s more…See alsoEnsuring S3 compliance with AWS ConfigGetting readyHow to do it…How it works…There’s more…See also

Chapter 7: Data Quality Management
Technical requirementsCreating data quality for ETL jobs in AWS Glue Studio notebooksGetting readyHow to do it…How it works…There’s more…See alsoUnit testing your data quality using DeequGetting readyHow to do it…How it works…There’s more…See alsoSchema management for ETL pipelinesGetting readyHow to do it…How it works…There’s more…See alsoBuilding unit test functions for ETL pipelinesHow to do it…How it works…There’s more…See alsoBuilding data cleaning and profiling jobs with DataBrewGetting readyHow to do it…How it works…There’s more…See also
Chapter 8: DevOps – Defining IaC and Building CI/CD Pipelines
Technical requirementsSetting up a code deployment pipeline using CDK and AWS CodePipelineGetting readyHow to do it…How it works…There’s more…See alsoSetting up a CDK pipeline to deploy on multiple accounts and regionsGetting readyHow to do it…How it works…There’s more…See alsoRunning code in a CloudFormation deploymentGetting readyHow to do it…How it works…There’s more…See alsoProtecting resources from accidental deletionGetting readyHow to do it…How it works…There’s more…See alsoDeploying a data pipeline using TerraformGetting readyHow to do it…How it works…There’s more…See alsoReverse-engineering IaCGetting readyHow to do it…How it works…There’s more…See alsoIntegrating AWS Glue and Git version controlGetting readyHow to do it…How it works…There’s more…See also
Chapter 9: Monitoring Data Lake Cloud Infrastructure
Technical requirementsAdditional informationAutomatically setting CloudWatch log group retention to reduce costGetting readyHow to do it…How it works…There’s more…See alsoCreating custom dashboards to monitor Data Lake servicesGetting readyHow to do it…How it works…There’s more…See alsoSetting up System Manager to remediate non-compliance with AWS Config rulesGetting readyHow to do it…There’s more…See alsoUsing AWS config to automate non-compliance S3 server access logging policyGetting readyHow to do it…How it works…There’s more…See alsoTracking AWS Data Lake cost per analytics workloadGetting readyHow to do it…How it works…There’s more…See also
Chapter 10: Building a Serving Layer with AWS Analytics Services
Technical requirementsUsing Redshift workload management (WLM) to manage workload priorityGetting readyHow to do it…How it works…There’s more…See alsoQuerying large historical data with Redshift SpectrumGetting readyHow to do it…How it works…There’s more…See alsoCreating a VPC endpoint to a Redshift clusterGetting readyHow to do it…How it works…There’s more…See alsoAccessing a Redshift cluster using JDBC to query dataGetting readyHow to do it…There’s more…See alsoUsing AWS SDK for pandas, the Redshift Data API, and Lambda to execute SQL statementsGetting readyHow to do it…How it works…There’s more…See alsoUsing the AWS SDK for Python to manage Amazon QuickSightGetting readyHow to do it…How it works…There’s more…See also
Chapter 11: Migrating to AWS – Steps, Strategies, and Best Practices for Modernizing Your Analytics and Big Data Workloads
Technical requirementsReviewing the steps and processes for migrating an on-premises platform to AWSGetting readyHow to do it…Choosing your AWS analytics stack – the re-platforming approachGetting readyHow to do it…See alsoPicking the correct migration approach for your workloadGetting readyHow to do it…Planning for prototyping and testingGetting readyHow to do it…Converting ETL processes with big data frameworksGetting readyHow to do it…Defining and executing your migration process with HadoopGetting readyHow to do it…Migrating the existing Hadoop security authentication and authorization processesGetting readyHow to do it…
Chapter 12: Harnessing the Power of AWS for Seamless Data Warehouse Migration
Technical requirementsCreating SCT migration assessment report with AWS SCTGetting readyHow to do it…See alsoExtracting data with AWS DMSGetting readyHow to do it…See alsoLive example – migrating an Oracle database from a local laptop to AWS RDS using AWS SCTGetting readyHow to do it…Leveraging AWS Snow Family for large-scale data migrationThe Snow Family Large Data Migration Manager as a serviceGetting readyHow to do it…See also
Chapter 13: Strategizing Hadoop Migrations – Cost, Data, and Workflow Modernization with AWS
Technical requirementsCalculating total cost of ownership (TCO) using AWS TCO calculatorsGetting readyHow to do it…See alsoConducting a Hadoop migration assessment using the TCO simulatorHadoop to Amazon EMR TCO simulatorGetting readyHow to do it…See alsoSelecting how to store your dataGetting readyHow to do it…See alsoMigrating on-premises HDFS data using AWS DataSyncGetting readyHow to do it...See alsoMigrating the Hive Metastore to AWSGetting readyHow to do it…Migrating and running Apache Oozie workflows on Amazon EMRGetting readyHow to do it…Migrating an Oozie database to the Amazon RDS MySQLGetting readyHow to do it…See alsoSetting up networking – establishing a secure connection to your EMR clusterGetting readyHow to do it…See alsoPerforming a seamless HBase migration to AWSGetting readyHow to do it…Migrating HBase to DynamoDB on AWSGetting readyHow to do it…See also
Index
Why subscribe?
Other Books You May EnjoyPackt is searching for authors like youShare Your ThoughtsDownload a free PDF copy of this book

Content preview from Data Engineering with AWS Cookbook

7 Data Quality Management

Unreliable data can lead to incorrect insights, misguided business decisions, and a significant loss of resources. As organizations treat data as a product and rely more on data freshness, data engineers and analysts must implement robust data quality control mechanisms to ensure the data’s accuracy, completeness, consistency, and reliability to maintain high data quality standards.

In this chapter, we will explore various methods and tools available on AWS for maintaining data quality. We’ll provide step-by-step recipes to help you implement these tools effectively in your data engineering workflows. The recipes will guide you through practical examples, starting with data quality control using AWS DataBrew, Deequ, ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Data Engineering with AWS - Second Edition

Publisher Resources

ISBN: 9781805127284

Cloud Computing