book

Data Engineering with Google Cloud Platform - Second Edition

Name: Data Engineering with Google Cloud Platform - Second Edition
Author: Adi Wijaya
ISBN: 9781835080115

by Adi Wijaya

April 2024

Beginner to intermediate

476 pages

12h 22m

English

Packt Publishing

Read now

Unlock full access

Data Engineering with Google Cloud Platform
ForewordContributorsAbout the authorAbout the reviewers
Preface
Who this book is forWhat this book coversTo get the most out of this bookDownload the example code filesConventions usedGet in touchShare Your ThoughtsDownload a free PDF copy of this book
Part 1: Getting Started with Data Engineering with GCP
Chapter 1: Fundamentals of Data Engineering
Understanding the data life cycleUnderstanding the need for a data warehouseStart with knowing the roles of a data engineerA data engineer versus a data scientistThe focus of data engineersGoing through the foundational concepts for data engineeringETL concept in data engineeringThe difference between ETL and ELTWhat is not big data?A quick look at how big data technologies store dataA quick look at how to process multiple files using MapReduceSummaryExerciseFurther Reading
Chapter 2: Big Data Capabilities on GCP
Technical requirementsUnderstanding what the cloud isThe difference between the cloud and non-cloud eraThe on-demand nature of the cloudGetting started with GCPIntroduction to the GCP consolePracticing pinning servicesA quick overview of GCP services for data engineeringUnderstanding the GCP serverless serviceService mapping and prioritizationThe concept of quotas on GCP servicesUser account versus service accountSummary
Part 2: Build Solutions with GCP Components
Chapter 3: Building a Data Warehouse in BigQuery
Technical requirementsIntroduction to GCS and BigQueryBigQuery data locationIntroduction to the BigQuery consoleCreating a dataset in BigQuery using the consoleLoading the local CSV file into the BigQuery tableUsing public data in BigQueryData types in BigQuery compared to other databasesTimestamp data in BigQuery compared to other databasesPreparing the prerequisites before developing our data warehouseStep 1 – Accessing Cloud ShellStep 2 – Checking the current setup using the command lineStep 3 – Initializing the gcloud init commandStep 4 – Downloading example data from GitStep 5 – Uploading data to GCS from GitPracticing developing a data warehouseData warehouse in BigQuery – Requirements for scenario 1Steps and planning for handling scenario 1Data warehouse in BigQuery – Requirements for scenario 2Using the GCP console versus the code-based approachSteps and planning for handling scenario 2BigQuery’s useful featuresBigQuery console sub-menu optionsBigQuery partitioned tableSummaryExercise – Scenario 3See also
Chapter 4: Building Workflows for Batch Data Loading Using Cloud Composer
Technical requirementsIntroduction to Cloud ComposerUnderstanding the working of AirflowCloud Composer 1 vs Cloud Composer 2Provisioning Cloud Composer in a GCP projectIntroducing the Airflow web UICloud Composer bucket directoriesExercise – build data pipeline orchestration using Cloud ComposerLevel 1 DAG – creating dummy workflowsDeploying the DAG file into Cloud ComposerLevel 2 DAG – scheduling a pipeline from Cloud SQL to GCS and BigQuery datasetsLevel 3 DAG – parameterized variablesLevel 4 DAG – Guaranteeing task idempotency in Cloud ComposerLevel 5 DAG – handling DAG dependency using an Airflow datasetSummary
Chapter 5: Building a Data Lake Using Dataproc
Technical requirementsIntroduction to DataprocA brief history of the data lake and Hadoop ecosystemA deeper look into Hadoop componentsHow much Hadoop-related knowledge do you need on GCP?Introducing the Spark RDD and DataFrame conceptsIntroducing the data lake conceptHadoop and Dataproc positioning on GCPIntroduction to Dataproc ServerlessExercise – Building a data lake on a Dataproc clusterCreating a Dataproc cluster on GCPUsing GCS as an underlying Dataproc filesystemExercise – Creating and running jobs on a Dataproc clusterPreparing log data in GCS and HDFSDeveloping a Spark ETL job from HDFS to HDFSDeveloping a Spark ETL job from GCS to GCSDeveloping a Spark ETL job from GCS to BigQueryUnderstanding the concept of an ephemeral clusterPracticing using a workflow template on DataprocBuilding an ephemeral cluster using Dataproc and Cloud ComposerSubmitting a Spark ETL job from GCS to BigQuery using Dataproc ServerlessSummary
Chapter 6: Processing Streaming Data with Pub/Sub and Dataflow
Technical requirementsProcessing streaming dataIntroduction to Pub/SubIntroduction to DataflowExercise – publishing event streams to Pub/SubCreating a Pub/Sub topicCreating and running a Pub/Sub publisher using PythonCreating a Pub/Sub subscriptionExercise – using Dataflow to stream data from Pub/Sub to GCSCreating a HelloWorld application using Apache BeamCreating a Dataflow streaming job without aggregationCreating a streaming job with aggregationIntroduction to CDC and DatastreamWhat is Datastream?Exercise – Datastream ETL streaming to BigQueryStep 1 – create a CloudSQL MySQL tableStep 2 – create a GCS bucketStep 3 – create a GCS notification to the Pub/Sub topic and subscriptionStep 4 – create a BigQuery datasetStep 5 – configure a Datastream jobStep 6 – run a Dataflow job from the Dataflow templateStep 7 – insert a value in MySQL and check the result in BigQuerySummary

Chapter 7: Visualizing Data to Make Data-Driven Decisions with Looker Studio
Technical requirementsUnlocking the power of your data with Looker StudioDon’t confuse Looker Studio with LookerFrom data to metrics in minutes with an Illustrative use caseUnderstanding what BigQuery INFORMATION_SCHEMA isExercise – accessing the BigQuery INFORMATION_SCHEMA table using Looker StudioExercise – creating a Looker Studio report using data from a bike-sharing data warehouseUnderstanding how Looker Studio can impact the cost of BigQueryWhat kind of table could be 1 TB in size?How can a table be accessed 10,000 times in a month?Creating Materialized Views and understanding how BI Engine worksUnderstanding BI EngineSummary
Chapter 8: Building Machine Learning Solutions on GCP
Technical requirementsA quick look at MLExercise – practicing ML code using PythonPreparing the ML dataset by using a table from the BigQuery public datasetTraining the ML model using Random Forest in PythonCreating a batch prediction using the training dataset’s outputThe MLOps landscape in GCPUnderstanding the basic principles of MLOpsIntroducing GCP services related to MLOpsExercise – leveraging pre-built GCP models as a serviceUploading the image to a GCS bucketCreating a detect text function in PythonExercise – using GCP in AutoML to train an ML modelExercise – deploying a dummy workflow with Vertex AI PipelinesCreating a dedicated regional GCS bucketDeveloping the pipeline on PythonMonitoring the pipeline on the Vertex AI Pipelines consoleExercise – deploying a scikit-learn model pipeline with Vertex AICreating the first pipeline, which will result in an ML model file in GCSRunning the first pipeline in Vertex AI PipelinesCreating the second pipeline, which will use the model file from the prediction results as a CSV file in GCSRunning the second pipeline in Vertex AI PipelinesSummary
Part 3: Key Strategies for Architecting Top-Notch Solutions
Chapter 9: User and Project Management in GCP
Technical requirementsUnderstanding IAM in GCPPlanning a GCP project structureUnderstanding the GCP organization, folder, and project hierarchyDeciding how many projects we should have in a GCP organizationControlling user access to our data warehouseUse-case scenario – planning BigQuery ACLs on an eCommerce organizationPracticing the concept of IaC using TerraformExercise – creating and running basic Terraform scriptsSelf-exercise – managing a GCP project and resources using TerraformSummary
Chapter 10: Data Governance in GCP
Technical requirementsIntroduction to data governanceA deeper understanding of data usabilityExercise – implementing metadata tagging using DataplexA deeper understanding of data securityExample – BigQuery data maskingExercise – finding PII using SDPA deeper understanding of being accountableClear traceabilityClear data ownershipData lineageClear data quality processExercise – practicing data quality using DataformSummary
Chapter 11: Cost Strategy in GCP
Technical requirementsEstimating the cost of your end-to-end data solution in GCPComparing BigQuery on-demand and editionsAn example – an estimating data engineering use caseTips to optimize BigQuery using partitioned and clustered tablesPartitioned tablesClustered tablesAn exercise – optimizing BigQuery on-demand costSummary
Chapter 12: CI/CD on GCP for Data Engineers
Technical requirementsAn introduction to CI/CDUnderstanding the data engineer’s relationship with CI/CD practicesUnderstanding CI/CD components with GCP servicesExercise – implementing CI using Cloud BuildCreating a GitHub repository using a Cloud Source RepositoryDeveloping the code and Cloud Build scriptsCreating a Cloud Build triggerPushing the code to the GitHub repositoryExercise – deploying Cloud Composer jobs using Cloud BuildPreparing the CI/CD environmentPreparing the cloudbuild.yaml configuration filePushing the DAG to our GitHub repositoryChecking the CI/CD result in the GCS bucket and Cloud ComposerCI/CD best practices in data engineeringSummaryFurther reading
Chapter 13: Boosting Your Confidence as a Data Engineer
Overviewing the Google Cloud certificationExam preparation tipsExtra GCP service materialsQuiz – reviewing all the concepts you’ve learned aboutQuestionsAnswersThe past, present, and future of data engineeringBoosting your confidence and final thoughtsSummary
Index
Why subscribe?
Other Books You May EnjoyPackt is searching for authors like youShare Your ThoughtsDownload a free PDF copy of this book

Content preview from Data Engineering with Google Cloud Platform - Second Edition

5 Building a Data Lake Using Dataproc

A data lake shares similarities with a data warehouse, yet its fundamental distinction lies in the nature of stored content. Unlike a data warehouse, a data lake is designed to manage extensive raw data, agnostic to its eventual value or purpose. This pivotal divergence reshapes approaches to data storage and retrieval within a data lake, setting it apart from the principles that we learned in Chapter 3, Building a Data Warehouse in BigQuery.

This chapter helps you understand how to build a data lake using Dataproc, which is a managed Hadoop cluster in Google Cloud Platform (GCP). But, more importantly, it helps you understand the key benefit of using a data lake in the cloud, which is allowing the use of ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Publisher Resources

ISBN: 9781835080115

Cloud Computing