Distributed Machine Learning with PySpark: Migrating Effortlessly from Pandas and Scikit-Learn

© The Author(s), under exclusive license to APress Media, LLC, part of Springer Nature 2023

A. TestasDistributed Machine Learning with PySparkhttps://doi.org/10.1007/978-1-4842-9751-3_17

17. Pipelines with Scikit-Learn and PySpark

Abdelaziz Testas¹  

(1)

Fremont, CA, USA

 

In this chapter, we explore the topic of pipeline techniques in both Scikit-Learn and PySpark. By harnessing the power of pipelines, data scientists can automate and standardize the steps involved in the modeling workflow. This enables the building of robust and scalable models, enhances model interpretability, and facilitates the integration of additional preprocessing steps and feature engineering techniques.

To illustrate how pipelines can streamline the modeling process and improve ...