Data Analysis with Python and PySpark

13 Robust machine learning with ML Pipelines

This chapter covers

• Using transformers and estimators to transform data into ML features
• Assembling features into a vector through an ML pipeline
• Training a simple ML model
• Evaluating a model using relevant performance metrics
• Optimizing a model using cross-validation
• Interpreting a model’s decision-making process through feature weights

In the previous chapter, we set the stage for machine learning: from a raw data set, we tamed the data and crafted features based on our exploration and analysis of the data. Looking back at the data transformation steps from chapter 12, we performed the following work, resulting in a data frame named food_features:

1. Read a CSV file containing dish names and multiple ...