book

Data Engineering with Google Cloud Platform

by Adi Wijaya

March 2022

Beginner to intermediate

440 pages

9h 43m

English

Packt Publishing

Read now

Unlock full access

ContributorsAbout the authorAbout the reviewer
Who this book is forWhat this book coversTo get the most out of this bookDownload the example code filesDownload the color imagesConventions usedGet in touchShare Your Thoughts
Understanding the data life cycleUnderstanding the need for a data warehouseKnowing the roles of a data engineer before startingData engineer versus data scientistThe focus of data engineersFoundational concepts for data engineeringETL concept in data engineering The difference between ETL and ELTWhat is NOT big data?A quick look at how big data technologies store dataA quick look at how to process multiple files using MapReduceSummaryExerciseSee also
Technical requirementsUnderstanding what the cloud isThe difference between the cloud and non-cloud eraThe on-demand nature of the cloud Getting started with Google Cloud PlatformIntroduction to the GCP console Practicing pinning servicesCreating your first GCP projectUsing GCP Cloud ShellA quick overview of GCP services for data engineeringUnderstanding the GCP serverless serviceService mapping and prioritizationThe concept of quotas on GCP servicesUser account versus service accountSummary
Technical requirementsIntroduction to Google Cloud Storage and BigQueryBigQuery data locationIntroduction to the BigQuery consoleCreating a dataset in BigQuery using the consoleLoading a local CSV file into the BigQuery tableUsing public data in BigQueryData types in BigQuery compared to other databasesTimestamp data in BigQuery compared to other databasesPreparing the prerequisites before developing our data warehouseStep 1: Access your Cloud shellStep 2: Check the current setup using the command line Step 3: The gcloud init commandStep 4: Download example data from GitStep 5: Upload data to GCS from GitPracticing developing a data warehouseData warehouse in BigQuery – Requirements for scenario 1Steps and planning for handling scenario 1Data warehouse in BigQuery – Requirements for scenario 2Steps and planning for handling scenario 2SummaryExercise – Scenario 3See also
Technical requirementsIntroduction to Cloud ComposerUnderstanding the working of Airflow Provisioning Cloud Composer in a GCP projectExercise: Build data pipeline orchestration using Cloud ComposerLevel 1 DAG – Creating dummy workflowsLevel 2 DAG – Scheduling a pipeline from Cloud SQL to GCS and BigQuery datasetsLevel 3 DAG – Parameterized variablesLevel 4 DAG – Guaranteeing task idempotency in Cloud ComposerLevel 5 DAG – Handling late data using a sensorSummary
Technical requirementsIntroduction to DataprocA brief history of the data lake and Hadoop ecosystemA deeper look into Hadoop componentsHow much Hadoop-related knowledge do you need on GCP? Introducing the Spark RDD and the DataFrame conceptIntroducing the data lake conceptHadoop and Dataproc positioning on GCPExercise – Building a data lake on a Dataproc clusterCreating a Dataproc cluster on GCPUsing Cloud Storage as an underlying Dataproc file system Exercise: Creating and running jobs on a Dataproc clusterPreparing log data in GCS and HDFSDeveloping Spark ETL from HDFS to HDFSDeveloping Spark ETL from GCS to GCSDeveloping Spark ETL from GCS to BigQueryUnderstanding the concept of the ephemeral clusterPracticing using a workflow template on DataprocBuilding an ephemeral cluster using Dataproc and Cloud ComposerSummary
Technical requirementsProcessing streaming dataStreaming data for data engineersIntroduction to Pub/SubIntroduction to DataflowExercise – Publishing event streams to cloud Pub/SubCreating a Pub/Sub topicCreating and running a Pub/Sub publisher using PythonCreating a Pub/Sub subscriptionExercise – Using Cloud Dataflow to stream data from Pub/Sub to GCSCreating a HelloWorld application using Apache BeamCreating a Dataflow streaming job without aggregationCreating a streaming job with aggregationSummary

Technical requirementsUnlocking the power of your data with Data StudioFrom data to metrics in minutes with an illustrative use caseUnderstanding what BigQuery INFORMATION_SCHEMA isExercise – Exploring the BigQuery INFORMATION_SCHEMA table using Data StudioExercise – Creating a Data Studio report using data from a bike-sharing data warehouseUnderstanding how Data Studio can impact the cost of BigQueryWhat kind of table could be 1 TB in size? How can a table be accessed 10,000 times in a month? How to create materialized views and understanding how BI Engine worksUnderstanding BI EngineSummary
Technical requirementsA quick look at machine learningExercise – practicing ML code using PythonPreparing the ML dataset by using a table from the BigQuery public datasetTraining the ML model using Random Forest in PythonCreating Batch Prediction using the training dataset's outputThe MLOps landscape in GCPUnderstanding the basic principles of MLOpsIntroducing GCP services related to MLOpsExercise – leveraging pre-built GCP models as a service Uploading the image to a GCS bucketCreating a detect text function in PythonExercise – using GCP in AutoML to train an ML modelExercise – deploying a dummy workflow with Vertex AI PipelineCreating a dedicated regional GCS bucketDeveloping the pipeline on PythonMonitoring the pipeline on the Vertex AI Pipeline consoleExercise – deploying a scikit-learn model pipeline with Vertex AICreating the first pipeline, which will result in an ML model file in GCSRunning the first pipeline in Vertex AI PipelineCreating the second pipeline, which will use the model file from the prediction results as a CSV file in GCSRunning the second pipeline in Vertex AI PipelineSummary
Technical requirements Understanding IAM in GCP Planning a GCP project structureUnderstanding the GCP organization, folder, and project hierarchyDeciding how many projects we should have in a GCP organizationControlling user access to our data warehouseUse-case scenario – planninga BigQuery ACL on an e-commerce organizationColumn-level security in BigQueryPracticing the concept of IaC using TerraformExercise – creating and running basic Terraform scriptsSelf-exercise – managing a GCP project and resources using TerraformSummary
Technical requirementsEstimating the cost of your end-to-end data solution in GCPComparing BigQuery on-demand and flat-rate Example – estimating data engineering use caseTips for optimizing BigQuery using partitioned and clustered tables Partitioned tablesClustered tablesExercise – optimizing BigQuery on-demand costSummary
Technical requirements Introduction to CI/CDUnderstanding the data engineer's relationship with CI/CD practicesUnderstanding CI/CD components with GCP servicesExercise – implementing continuous integration using Cloud BuildCreating a GitHub repository using Cloud Source RepositoryDeveloping the code and Cloud Build scriptsCreating the Cloud Build TriggerPushing the code to the GitHub repositoryExercise – deploying Cloud Composer jobs using Cloud Build Preparing the CI/CD environmentPreparing the cloudbuild.yaml configuration filePushing the DAG to our GitHub repositoryChecking the CI/CD result in the GCS bucket and Cloud ComposerSummaryFurther reading
Overviewing the Google Cloud certificationExam preparation tipsExtra GCP services materialQuiz – reviewing all the concepts you've learned aboutQuestionsAnswersThe past, present, and future of Data EngineeringBoosting your confidence and final thoughtsSummary
Other Books You May EnjoyPackt is searching for authors like youShare Your Thoughts

Content preview from Data Engineering with Google Cloud Platform

Chapter 3: Building a Data Warehouse in BigQuery

The power of a data warehouse is delivered when organizations combine multiple sources of information into a single place that becomes the single source of truth. The utopia of data analytics will be when every single business aspect in an organization relies on data. That condition will be met when all business decision makers know how to access data, trust the data, and can make decisions based on it.

Unfortunately, most of the time, utopia is far removed from reality. There are many challenges along the way. Based on my experience, there are three main challenges – technology bottlenecks, data consistency, and the ability to serve multiple business purposes.

The preceding challenges are natural ...