Python Machine Learning By Example - Second Edition

Feature engineering on categorical variables with Spark

We have demonstrated how to build an ad click predictor that learns from massive click logs using Spark. Thus far, we have been using one-hot encoding to employ the categorical inputs. In this section, we will be talking about two popular feature engineering techniques: feature hashing and feature interaction. One is an alternative to one-hot encoding, another is a variant of one-hot encoding. Feature engineering means generating new features based on domain knowledge or defined rules, in order to improve learning performance achieved with existing feature space.