book

Serverless ETL and Analytics with AWS Glue

by Vishal Pathak, Subramanya Vajiraya, Noritaka Sekiyama, Tomohiro Tanaka, Albert Quiroga, Ishan Gaur

August 2022

Intermediate to advanced

434 pages

10h 34m

English

Packt Publishing

Read now

Unlock full access

ContributorsAbout the authorsAbout the reviewers
Who this book is forWhat this book coversTo get the most out of this bookDownload the example code filesDownload the color imagesConventions usedGet in touchShare Your Thoughts
Types of data processing – OLTP and OLAPData warehouses and data martsData lakesData lakehouseData meshDistributed computing for big dataApache SparkApache Spark on the AWS cloudAWS Glue Querying data using AWSSummary
Data integration Integrating data with AWS GlueData discoveryData ingestionData preparationData replicationFeatures of AWS GlueAWS Glue Data CatalogGlue connections AWS Glue crawlersCustom classifiersAWS Glue Schema RegistryAWS Glue ETL jobsGlue development endpoints AWS Glue interactive sessions TriggersSummary
Technical requirementsData ingestion from file/object storesData ingestion from Amazon S3Data ingestion from HDFS data storesData ingestion from JDBC data storesAWS Glue custom JDBC connectorsData ingestion from streaming data sourcesAWS Glue Schema RegistryData ingestion from SaaS data storesSummary
Technical requirementsIntroduction to data preparation Data preparation using AWS GlueVisual data preparation using AWS Glue DataBrewSource code-based approach to data preparation using AWS GlueSelecting the right service/toolSummary
Technical requirementsWhy do we need to pay attention to data layout?Key techniques to optimally storing data Selecting a file formatCompressing your dataSplittable or unsplittable filesPartitioningBucketingOptimizing the number of files and each file sizeWhat is compaction?Compaction with AWS Glue ETL Spark jobsAutomatic Compaction with AWS Lake Formation accelerationOptimizing your storage with Amazon S3Selecting suitable S3 storage classes for your dataUsing S3 Lifecycle for managing object lifecyclesSummaryFurther reading
Technical requirementsNormalizing dataCasting data types and map column namesInferring schemasComputing schemas on the flyEnforcing schemasFlattening nested schemasNormalizing scaleHandling missing values and outliersNormalizing date and time valuesHandling error recordsDeduplicating recordsDenormalizing tablesSecuring data contentMasking valuesHashing valuesManaging data qualityAWS Glue DataBrew data quality rulesDeeQuSummary

Technical requirementsPopulating metadataGlue Data Catalog APIDDL statementsGlue crawlersCrawler configurationMaintaining metadataGlue crawlersUpdating Data Catalog tables from ETL jobsPartition managementPartition indexesVersioning and rollbackTable versioningLake Formation-governed tablesLineageGlue DataBrewSummary
Technical requirementsAccess controlIAM permissionsGlue dependencies on other AWS servicesS3 bucket policiesS3 object ownershipLake Formation permissionsEncryptionEncryption at restEncryption in transitNetworkGlue network architectureGlue connectionsNetwork configuration requirements and limitationsConnecting to resources on the public internetConnecting to resources in your on-premise networkSummary
Technical requirementsOverview of data sharing strategiesSingle tenantHub and spokeData meshSharing data with multiple AWS accounts using S3 bucket policies and Glue catalog policiesScenario 1 – sharing data from one account with another using S3 bucket policies and Glue catalog policiesPrerequisite – S3Prerequisite – GlueConfiguring S3 bucket policies and Glue Catalog resource policiesSharing data with multiple AWS accounts using AWS Lake Formation permissionsLake Formation permission modelLake Formation cross-account sharingLake Formation named resource-based access controlLake Formation tag-based access controlScenario 2 – sharing data from one account with another using Lake Formation Tag-based access controlPrerequisite – S3Prerequisite – GluePrerequisite – Lake Formation and IAMStep 1 – configuring Glue catalog policiesStep 2 – configuring Lake Formation permissions (producer)Step 3 – configuring Lake Formation permissions (consumer)Summary
Technical requirementsWhat are data pipelines?Why do we need data pipelines?How do we build and manage data pipelines?Selecting the appropriate data processing services for your analysisAWS BatchAmazon ECSAWS LambdaAWS Glue ETL jobsAmazon EMROrchestrating your pipelines with workflow toolsUsing AWS Glue workflowsUsing AWS Step FunctionsUsing Amazon Managed Workflows for Apache Airflow utomating how you provision your pipelines with provisioning toolsProvisioning resources with AWS CloudFormationProvisioning AWS Glue workflows and resources with AWS Glue BlueprintsDeveloping and maintaining your data pipelinesDeveloping AWS Glue ETL jobs locallyDeploying AWS Glue ETL jobs Deploying workflows and pipelines using provisioning tools such as IaC SummaryFurther reading
Defining an SLA for a data platformMonitoring the SLA of a data platformMonitoring the components of a data platformMonitoring state changesMonitoring delayMonitoring performanceMonitoring common failuresMonitoring log messagesAnalyzing usageSummary
Tuning AWS Glue workloadsTuning AWS Glue crawlersTuning the performance of AWS Glue Spark ETL jobsTroubleshooting and debugging common issues in AWS Glue ETLETL job failuresSummary
Creating Marketplace connectionsCreating the Glue Hudi connectionCreating a Delta Lake connectionCreating an OpenSearch connectionCreating the CloudFormation stackPrerequisites for creating the CloudFormation stackThe benefit of ad hoc analysis and how a data lake enables it Amazon AthenaAmazon Redshift SpectrumCreating and updating Hudi tables using GlueCreating and updating Delta Lake tables using GlueInserting data into Lake Formation governed tablesConsuming streaming data using GlueCreating chapter-data-analysis-msk-connectionLoading and consuming data from MSK using GlueGlue streaming job as a consumer of a Kafka topicHudi DeltaStreamer streaming job as a consumer of a Kafka topicCreating and consuming CDC data through streaming jobs on GlueGlue’s integration with OpenSearchCleaning upSummary
Technical requirementsGlue ML transformationsCreating an ML transformTraining an ML transformUsing an ML transformSageMaker integrationDeveloping ML pipelines with GlueSummary
Technical requirementsRunning a highly selective query on a big fact table using AWS GlueHands-on tutorialDealing with Join performance issues with big fact and small dimension tables in ETL workloadsSolving Join problems involving big fact and big dimension tables using AWS GlueHands-on tutorialSolutionReducing time on read operations using AWS Glue groupingSolving S3 eventual consistency problems using AWS GlueUsing glueparquetS3-optimized output committerSummary
Other Books You May EnjoyPackt is searching for authors like youShare Your Thoughts

Content preview from Serverless ETL and Analytics with AWS Glue

Section 3 – Tuning, Monitoring, Data Lake Common Scenarios, and Interesting Edge Cases

Here, you will learn various ways to monitor and troubleshoot an AWS Glue job. You will also learn about different ways to consume data after it is processed by AWS Glue and apply the concepts introduced in this book to real-world data transformation scenarios.

This section includes the following chapters: