Building Machine Learning Pipelines

Chapter 2. Introduction to TensorFlow Extended

In the previous chapter, we introduced the concept of machine learning pipelines and discussed the components that make up a pipeline. In this chapter, we introduce TensorFlow Extended (TFX). The TFX library supplies all the components we will need for our machine learning pipelines. We define our pipeline tasks using TFX, and they can then be executed with a pipeline orchestrator such as Airflow or Kubeflow Pipelines. Figure 2-1 gives an overview of the pipeline steps and how the different tools fit together.

Figure 2-1. TFX as part of ML pipelines

In this chapter, we will guide you through the installation of TFX, explaining basic concepts and terminology that will set the stage for the following chapters. In those chapters, we take an in-depth look at the individual components that make up our pipelines. We also introduce Apache Beam in this chapter. Beam is an open source tool for defining and executing data-processing jobs. It has two uses in TFX pipelines: first, it runs under the hood of many TFX components to carry out processing steps like data validation or data preprocessing. Second, it can be used as a pipeline orchestrator, as we discussed in Chapter 1. We introduce Beam here because it will help you understand TFX components, and it is essential if you wish to write custom components, as we discuss in Chapter 10.