Learn PySpark: Build Python-based Machine Learning and Deep Learning Models

© Pramod Singh 2019

P. SinghLearn PySparkhttps://doi.org/10.1007/978-1-4842-4961-1_2

2. Data Processing

Pramod Singh¹ 

(1)

Bangalore, Karnataka, India

 

This chapter covers different steps to preprocess and handle data in PySpark. Preprocessing techniques can certainly vary from case to case, and many different methods can be used to massage the data into desired form. The idea of this chapter is to expose some of the common techniques for dealing with big data in Spark. In this chapter, we are going to go over different steps involved in preprocessing data, such as handling missing values, merging datasets, applying functions, aggregations, and sorting. One major part of data preprocessing is the transformation of numerical columns into categorical ...