Building Machine Learning Pipelines

Appendix C. Tips for Operating Kubeflow Pipelines

When you operate your TFX pipelines with Kubeflow Pipelines, you might want to customize the underlying container images of your TFX components. Custom TFX images are required if your components rely on additional Python dependencies outside of the TensorFlow and TFX packages. In the case of our demo pipeline, we have an additional Python dependency, the TensorFlow Hub library, for accessing our language model.

In the second half of this appendix, we want to show you how to transfer data to and from your local computer and your persistent volume. The persistent volume setup is beneficial if you can access your data via a cloud storage provider (e.g., with an on-premise Kubernetes cluster). The presented steps will guide you through the process of copying data to and from your cluster.

Custom TFX Images

In our example project, we use a language model provided by TensorFlow Hub. We use the tensorflow_hub library to load the language model efficiently. This particular library isn’t part of the original TFX image; therefore, we need to build a custom TFX image with the required library. This is also the case if you plan to use custom components like the ones we discussed in Chapter 10.

Fortunately, as we discussed in Appendix A, Docker images can be built without much trouble. The following Dockerfile shows our custom image setup:

FROM tensorflow/tfx:0.22.0

RUN python3.6 -m pip install "tensorflow-hub" 
RUN ... 

ENTRYPOINT ["python3.6", ...