The San Francisco city also has an open data portal (https://datasf.org/opendata/) providing data from different departments online. In this section, we take the dataset providing about 12 years (from January 2003 to May 2015) of crime reports from across all of San Francisco's neighborhoods and train a model to predict the category of crime that occurred. There are 39 discreet crime categories, thus it's a multi-class classification problem.
We will use make use of Apache's PySpark and use its easy to use text processing features for this dataset. So the first step will be to create a Spark session:
- The first step is to import the necessary modules and create a Spark session:
from pyspark.ml.classification ...
