book

Simplify Big Data Analytics with Amazon EMR

by Sakti Mishra

March 2022

Beginner to intermediate

430 pages

9h 24m

English

Packt Publishing

Read now

Unlock full access

ContributorsAbout the authorAbout the reviewers
Who this book is forWhat this book coversTo get the most out of this bookDownload the example code filesCode in ActionDownload the color imagesConventions usedGet in touchShare Your Thoughts
What is Amazon EMR?What is big data?Hadoop – processing framework to handle big dataOverview of Amazon EMR – managed and scalable Hadoop cluster in AWSA brief history of the major big data releasesBenefits of Amazon EMR Decoupling compute and storagePersistent versus transient clustersIntegration with other AWS servicesAmazon S3 with EMR File System (EMRFS)Amazon Kinesis Data Streams (KDS)Amazon Managed Streaming for Kafka (MSK)AWS Glue Data CatalogAmazon Relational Database Service (RDS)Amazon DynamoDBAmazon RedshiftAWS Lake FormationAWS Identity and Access Management (IAM)AWS Key Management Service (KMS)Lake House architecture overviewEMR release historyComparing Amazon EMR with AWS Glue and AWS Glue DataBrewAWS GlueAWS Glue DataBrewChoosing the right service for your use caseSummaryTest your knowledgeFurther reading
EMR architecture deep diveDistributed storage layerYARN – cluster resource managerDistributed processing frameworksHadoop applicationsUnderstanding clusters and nodesUniform instance groupsInstance fleetUsing S3 versus HDFS for cluster storageHDFS as cluster-persistent storageAmazon S3 as a persistent data storeUnderstanding the cluster life cycleOptions to submit work to the clusterSubmitting jobs to the cluster as EMR stepsBuilding Hadoop jobs with dependencies in a specific EMR release versionEMR deployment options Amazon EMR on Amazon EC2Amazon EMR on Amazon EKSAmazon EMR on AWS OutpostsEMR pricing for different deployment optionsMonitoring and controlling your costs with AWS Budgets and Cost ExplorerSummaryTest your knowledgeFurther reading
Reference architecture for batch ETL workloadsUse case overviewReference architecture walkthroughBest practices to follow during implementationReference architecture for clickstream analyticsUse case overviewReference architecture walkthroughBest practices to follow during implementationReference architecture for interactive analytics and MLUse case overviewReference architecture walkthroughBest practices to follow during implementationReference architecture for real-time streaming analyticsUse case overviewReference architecture walkthroughBest practices to follow during implementationReference architecture for genomics data analyticsUse case overviewReference architecture walkthroughBest practices to follow during implementationReference architecture for log analyticsUse case overviewReference architecture walkthroughBest practices to follow during implementationSummaryTest your knowledgeFurther reading
Technical requirementsUnderstanding popular big data applications in EMRHivePrestoSparkHBaseHueGangliaMachine learning frameworks available in EMRTensorFlowMXNetNotebook options available in EMREMR Notebooks JupyterHub EMR Studio ZeppelinSummaryTest your knowledgeFurther reading
Technical requirementsSetting up and configuring clusters with the EMR console's quick create optionAdvanced configuration for cluster hardware and software Understanding the Software Configuration sectionUnderstanding Steps Understanding the Hardware Configuration sectionUnderstanding general configurationsUnderstanding Security Options Working with AMIs and controlling cluster termination Working with AMIsControlling the EMR cluster termination processTroubleshooting and logging in your EMR cluster Tools available to debug your EMR cluster Viewing and restarting cluster application processesTroubleshooting a failed clusterTroubleshooting a slow clusterLogging in your EMR cluster SummaryTest your knowledgeFurther reading
Technical requirementsMonitoring your EMR clusterMonitoring clusters and applications with web user interfaces Monitoring cluster metrics with CloudWatch monitoring EMR API audit logging with AWS CloudTrail Scaling cluster resources Managed scaling in EMR Autoscaling in EMR with a custom policy for instance groupsManually resizing your EMR clusterComparing managed scaling with autoscaling Cluster cloning and high availability with multiple master nodesHigh availability with multiple master nodes Cloning an existing EMR cluster SummaryTest your knowledgeFurther reading

Technical requirementsUnderstanding the basics of security Creating security configurations Specifying a security configuration for your cluster AWS IAM integration with Amazon EMR Configuring an IAM service role for your EMR cluster Configuring IAM roles for EMRFS Integrating IAM roles in applications that invoke AWS services directly Allowing users and groups to create and modify roles Identity-based policies and best practices Understanding authentication to cluster nodes Understanding data protection in EMR Encrypting data at rest for EMRFS on Amazon S3 data Encrypting data in transit for EMRFS on Amazon S3 data Role of security groups and interface VPC endpoints Controlling cluster network traffic with security groups Connecting to Amazon EMR on an EC2 cluster using an interface VPC endpoint Connecting to Amazon EMR on an EKS cluster using an interface VPC endpoint Summary Test your knowledgeFurther reading
Technical requirementsUnderstanding data catalog and access management options Using AWS Glue Data Catalog Integrating AWS Glue Data Catalog with Amazon EMR Permission management on top of a data catalogUnderstanding Amazon EMR integration with AWS Lake Formation Integrating Lake Formation with Amazon EMR Launching an EMR cluster with Lake Formation Setting up EMR notebooks to work with Lake Formation Understanding Amazon EMR integration with Apache Ranger Setting up Apache Ranger in EMR Understanding Apache Ranger plugins SummaryTest your knowledgeFurther reading
Technical requirementsUse case and architecture overviewArchitecture overview Implementation stepsCreating Amazon S3 buckets Creating the AWS Lambda function Configuring an S3 file arrival event to trigger the Lambda function Triggering the EMR job Validating the output using Amazon Athena Defining a virtual Glue Data Catalog table on top of Amazon S3 data Querying output data using Amazon Athena standard SQL Spark ETL and Lambda function code walk-throughUnderstanding the AWS Lambda function code Understanding the PySpark script integrated into the EMR step SummaryTest your knowledgeFurther reading
Technical requirementsUse case and architecture overviewArchitecture overview Implementation stepsCreating Amazon S3 buckets Creating the Amazon Kinesis data stream Creating and configuring the Kinesis Data Generator toolCreating an Amazon EMR cluster and configuring a Spark Streaming job Validating output using Amazon Athena Defining a virtual Glue Catalog table on top of Amazon S3 data Querying output data using a standard SQL query in Amazon AthenaSpark Streaming code walk-throughSummaryTest your knowledgeFurther reading
Technical requirementsApache Hudi overviewPopular use casesRegistering Hudi data with your Hive or Glue Data Catalog metastore Creating an EMR cluster and an EMR notebookCreating an EMR cluster Creating an EMR notebookCreating an Amazon S3 bucketInteractive development with Spark and HudiCreating a PySpark notebook for development Integrating Hudi with our PySpark notebook Executing Spark and Hudi scripts in your notebook SummaryTest your knowledgeFurther reading
Technical requirementsOverview of AWS Step FunctionsIntegrating AWS Step Functions to orchestrate EMR jobsOverview of Apache Airflow and MWAAIntegrating Airflow to trigger EMR jobsSummaryTest your knowledgeFurther reading
Understanding migration approachesLift and shift Re-architecting Hybrid architectureMigrating data and metadata catalogsMigrating data Migrating metadata catalogsMigrating ETL jobs and Oozie workflows Migrating Oozie workflowsTesting and validation Validating metadata quality Validating data quality Best practices for migration SummaryTest your knowledgeFurther reading
Best practices around EMR cluster configurationsChoosing the correct cluster type (transient versus long-running) Best practices around sizing your clusterOptimization techniques for data processing and storageBest practices for cluster persistent storage Best practices while processing data using EMR Security best practicesConfiguring edge nodes outside of the cluster to limit connectivity Integrating logging, monitoring, and audit controls into your clusterBlocking public access to your EMR cluster Protecting your data at rest and in transitCost-optimization techniquesCost savings with compute resources Cost savings with storage Integrating AWS Budgets and Cost Explorer AWS Trusted AdvisorCost allocation tags Limitations of Amazon EMR and possible workaroundsSummaryTest your knowledgeFurther reading
Other Books You May EnjoyPackt is searching for authors like youShare Your Thoughts

Content preview from Simplify Big Data Analytics with Amazon EMR

Chapter 9: Implementing Batch ETL Pipeline with Amazon EMR and Apache Spark

In Chapter 2, Exploring the Architecture and Deployment Options, you learned about different EMR use cases such as batch Extract, Transform, and Load (ETL), real-time streaming with EMR and Spark streaming, data preparation for machine learning (ML) models, interactive analytics, and more.

In this chapter, we will dive deep into a use case – Batch ETL with Amazon EMR and Apache Spark, where we will look at the implementation steps that you can follow to replicate the setup in your AWS account.

We will cover the following topics, which will help you understand the use case, its application architecture, and how a transient EMR cluster with Spark can be integrated for ...