Python Data Science Essentials - Third Edition

Feature engineering

From a visual analysis, only four fields are strings: protocol_type, service, flag, and target (which is the multiclass target label, as expected). As we will use a tree-based classifier, we want to encode the text of each level to a number for each variable. With scikit-learn, this operation can be done with a sklearn.preprocessing.LabelEncoder object. It's equivalent in Spark is the StringIndexer of the pyspark.ml.feature package. We need to encode four variables with Spark, and then we have to chain four StringIndexer objects together in a cascade: each of them will operate on a specific column of the DataFrame, outputting a DataFrame with an additional column (similar to a map operation). The mapping is automatic, ...