Building Machine Learning Pipelines

Appendix B. Setting Up a Kubernetes Cluster on Google Cloud

This appendix provides a brief overview of how to create a Kubernetes cluster on Google Cloud that can run our example project. If Kubernetes is new to you, take a look at Appendix A and our suggested reading at the end of Chapter 9. While the exact commands we will cover only apply to Google Cloud, the overall setup process is the same with other managed Kubernetes services like AWS EKS or Microsoft Azure’s AKS.

Before You Get Started

For the following installation steps, we assume you have an account with Google Cloud. If you don’t have an account, you can create one. Furthermore, we assume that you have installed Kubernetes kubectl (client version 1.18.2 or higher) on your local computer and that you can also execute Google Cloud’s SDK gcloud (version 289.0.0 or higher).

Watch Your Cloud Infrastructure Costs

Operating Kubernetes clusters can accumulate significant infrastructure costs. Therefore, we highly recommend to watch your infrastructure costs by setting billing alerts and budgets. Details can be found in the Google Cloud documentation. We also recommend turning off idling compute instances because they accrue costs even if they are idling and no pipeline task is being computed.

Steps on how to install a kubectl client for your operating system can be found as part of the Kubernetes documentation. The Google Cloud documentation provides step-by-step details on how to install their client for your operating ...