book

Google Cloud Cookbook

by Rui Santos Costa, Drew Hodun

October 2021

Beginner

282 pages

5h 38m

English

O'Reilly Media, Inc.

Audiobook available

Read now

Unlock full access

Preface
Who Should Read This BookWhy I Wrote This BookNavigating This BookConventions Used in This BookUsing Code ExamplesO’Reilly Online LearningHow to Contact UsAcknowledgments
1. Introduction
2. Cloud Functions
2.1. Creating a Public HTTP Google Cloud Function2.2. Authenticating an HTTP Google Cloud Function2.3. Accessing Environment Variables at Runtime2.4. Sending Emails from Cloud Functions with SendGrid2.5. Deploying Cloud Functions with a GitLab CI/CD Pipeline2.6. Responding to SMS Messages with Twilio and Cloud Functions2.7. Unit Testing with GitLab and Cloud Functions2.8. Building an API Gateway to Gather Telemetry Data
3. Google Cloud Run
3.1. Deploying a Prebuilt Hello World Container3.2. Building Your Own Hello World Container3.3. Using Cloud Run with a Custom Domain3.4. Triggering a Cloud Run from Cloud Pub/Sub3.5. Deploying a Web Application to Cloud Run3.6. Rolling Back a Cloud Run Service Deployment3.7. Deploying Cloud Run Services in a Gradual Rollout3.8. Cloud Run Configuration Parameters
4. Google App Engine
4.1. Deploying a Hello World to App Engine (Standard)4.2. Deploying a Hello World to App Engine (Flexible)4.3. Securing Your Application with Identity-Aware Proxy4.4. Mapping Custom Domains with App Engine4.5. Using the Google Cloud Translation Machine Learning APIs with App Engine4.6. Building User Interfaces for Viewing Charts and Graphs4.7. Debugging an Instance4.8. Using CI/CD
5. Google Cloud Compute Engine
5.1. Creating a Windows Virtual Machine5.2. Creating a Linux Virtual Machine and Installing NGINX5.3. Connecting to Your Windows Virtual Machines with Identity-Aware Proxy TCP Forwarding5.4. Securing Your Virtual Machine Logins with Two-Step Verification5.5. Running Startup Scripts5.6. Creating a Group of NGINX Web Servers with a Managed Instance Group5.7. Deploying Containers to Managed Instance Groups5.8. Transferring Files to Your Virtual Machine5.9. Using VM Manager for Patch Management5.10. Backing Up Your Virtual Machine
6. Google Cloud Kubernetes Engine
6.1. Creating a Zonal Cluster6.2. Creating a Regional Cluster6.3. Resizing a Cluster6.4. Automatically Routing Traffic to the Nearest Cluster with Multi-Cluster Ingress6.5. Deploying a Spring Boot Java Application6.6. Deploying a Java Application to Kubernetes, Using Skaffold6.7. Using GKE Autopilot for Running an Application You Don’t Have to Manage
7. Working with Data
7.1. Speeding Up Cloud Storage Bulk Transfers by Multiprocessing7.2. Speeding Up GCS Transfers for Large Files with Parallel Composite Uploads7.3. Mounting GCS as a Filesystem7.4. Automatically Archiving and Deleting GCS Objects7.5. Creating and Restoring from Persistent Disk Snapshots in GCE7.6. Using Interleaved Tables in Your Cloud Spanner Database7.7. Locking Down Firestore Database So a User Can Edit Only Their Data
8. BigQuery and Data Warehousing
8.1. Using Cloud Console to Run a BigQuery Query8.2. Loading Data to BigQuery from CSV8.3. Building a Pivot Table in BigQuery8.4. Adding Partitioned and Clustered Columns to an Existing Table8.5. Adding Clustering to a Table That Can’t or Shouldn’t Be Partitioned8.6. Selecting the Top-1 Result8.7. Merging Tables in BigQuery Without Duplicates8.8. Deduplicating Rows in BigQuery with Timestamps8.9. Undeleting a Table in BigQuery8.10. Streaming JSON or Avro Data into BigQuery with a Dataflow Template
9. Data Processing Tools
9.1. Cleaning Data Using the Data Fusion GUI9.2. Running a Simple Python Dataflow Pipeline9.3. Building a Streaming Pipeline in Dataflow SQL9.4. Querying BigQuery from a Dataproc Job9.5. Adding Event Timestamps to Pub/Sub9.6. Inferring and Using Schemas in Dataflow9.7. Mini-batching and Streaming Dataflow Data to BigQuery Using Filters9.8. Triggering a Dataflow Job Automatically from a GCS Upload

10. AI/ML
10.1. Creating a Vertex AI Notebook10.2. Training a Python ML Model Serverlessly10.3. Making Serverless Predictions with a Python Model10.4. Creating a Custom Notebook Environment10.5. Extracting Data from BigQuery to Pandas for Model Training10.6. Training a Model in SQL with BQML
11. Google Cloud Security and Access
11.1. Creating a Service Account11.2. Creating Custom Roles to Access a Cloud Storage Bucket11.3. Authenticating an Application Running on Kubernetes Engine11.4. Retrieving the Authenticated User’s Identity11.5. Authenticating a Java Application Using a Service Account11.6. Building Reports Using the Cloud Asset API11.7. Allowing a List of IP Addresses to Access Your Application
12. Google Cloud Networking
12.1. Creating a Custom Mode VPC Network12.2. Creating a Static External IP Address12.3. Create a Firewall Rule12.4. Serving Content for Users in a Specific Region12.5. Configuring VPC Network Peering12.6. Creating VPN Gateways with Cloud Routers12.7. Deployments of Networks Using Terraform12.8. Limiting Access to Only Authorized Networks with VPC Service Controls
Index

Content preview from Google Cloud Cookbook

Chapter 9. Data Processing Tools

Google Cloud offers a variety of scalable data processing tools. Dataflow and Dataproc are the most commonly used (outside of BigQuery, covered in another chapter). These tools allow you to run open source Apache Spark or Apache Beam pipelines in a serverless or near-serverless environment. Cloud Dataflow, in particular, is an excellent environment for running large-scale, mission-critical, streaming pipelines for real-time analytics, data ingestion, and business logic. There are also low and no-code data processing toolsets, such as Cloud Data Fusion. These recipes are examples of some of the most common tasks you’ll perform as you implement solutions on these tools and include a few more advanced Dataflow pipeline patterns.

All code samples for this chapter are in this book’s GitHub repository. You can follow along and copy the code for each recipe by going to the folder with that recipe’s number.

9.1 Cleaning Data Using the Data Fusion GUI

Problem

You want to clean and join data sets in a repeatable pipeline in a low or no-code, GUI-driven tool.

Solution

Cloud Data Fusion allows users to interact with data from sources such as GCS and BigQuery and author-repeatable pipelines in a GUI, and execute them scalably and on a schedule, using Dataproc under the hood.

In this example, we will ingest some data from CSV to BigQuery, applying transformations and filters along the way.

From the Cloud Console, navigate to the Data Fusion page. You may ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Publisher Resources

ISBN: 9781492092889Errata Page

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business