Mastering Machine Learning with Spark 2.x

Interest RateModel

The second model predicts the interest rate of accepted loans. In this case, we will use only the part of the training data that corresponds to good loans, since they have assigned a proper interest rate. However, we need to understand that the remaining bad loans could carry useful information related to the interest rate prediction.

As in the rest of the cases, we will start with the preparation of training data. We will use initial data, filter out bad loans, and drop string columns:

val intRateDfSplits = loanStatusDfSplits.map(df => {   df     .where("loan_status == 'good loan'")     .drop("emp_title", "desc", "loan_status")     .withColumn("int_rate", toNumericRateUdf(col("int_rate"))) })val trainIRHf = toHf(intRateDfSplits(0), ...