Mastering Machine Learning with Spark 2.x

Useless columns

The first step is to remove columns that contain unique values per line. Typical examples of this are user IDs or transaction IDs. In our case, we will identify them manually based on data description:

import com.packtpub.mmlwspark.utils.Tabulizer.tableval idColumns = Seq("id", "member_id")println(s"Columns with Ids: ${table(idColumns, 4, None)}")

The output is as follows:

The next step is to identify useless columns, such as the following:

• Constant columns
• Bad columns (containing only missing values)

The following code will help us do so:

val constantColumns = loanDataHf.names().indices .filter(idx => loanDataHf.vec(idx).isConst ...