Mastering Machine Learning with Spark 2.x

Missing data

The last step in our data-exploration journey is to explore missing values. We already observed that some columns contain a value that represents a missing value; however, in this section, we will focus on pure missing values. First, we need to collect them:

val naColumns = loanDataHf.names().indices   .filter(idx => loanDataHf.vec(idx).naCnt() >0)   .map(idx =>          (loanDataHf.name(idx),            loanDataHf.vec(idx).naCnt(),f"${100*loanDataHf.vec(idx).naCnt()/loanDataHf.numRows().toFloat}%2.1f%%")   ).sortBy(-_._2)println(s"Columns with NAs (#${naColumns.length}):${table(naColumns)}")

The list contains 111 columns with the number of missing values varying from 0.2 percent to 86 percent:

There are plenty of columns with five missing values, which ...