Machine Learning Production Systems

Chapter 19. Advanced TFX

In the preceding chapter, we showed you how to orchestrate your ML pipelines using standard TFX components. In this chapter, we’ll introduce advanced concepts of ML pipelines and show you how to extend your portfolio of components by quickly writing your own custom components. We will also show you different ways of writing your own components and explain when to use which option.

Advanced Pipeline Practices

In this section, we will discuss additional concepts to advance your pipeline setups. So far, all the pipeline concepts we’ve discussed comprised linear graphs with one entry and one exit point. In the preceding chapter, we discussed the fundamentals of directed acyclic graphs (DAGs). As long as our pipeline graph is directed and doesn’t create any circular connections, we can be creative with our setup. In the following subsections, we will highlight a few concepts to increase the productivity of pipelines.

Warning

Some of the concepts in this chapter are part of the v1 TFX API, but they’re still in an experimental stage. That means the specific API is still subject to change, though that is highly unlikely at this stage.

Configure Your Components

Sometimes you’ll need to set up a component of the same type twice. For example, you might do this if you want to evaluate your model twice. TFX will complain that it already is set up with an Evaluator component. In those cases, we highly recommend giving your components custom identifiers. You can ...