Data Analysis with Python and PySpark

14 Building custom ML transformers and estimators

This chapter covers

• Creating your own transformers using Params for parameterization
• Creating your own estimators using the companion model approach
• Integrating custom transformers and estimators in an ML Pipeline

In this chapter, we cover how to create and use custom transformers and estimators. While the ecosystem of transformers and estimators provided by PySpark covers a lot of frequent use cases and each version brings new ones to the table, sometimes you just need to go off trail and create your own. The alternative is to cut your pipeline in half and insert a data transformation function into the mix. This basically nullifies all the advantages (portability, self-documentation) of the ...