Hands-On Data Science and Python Machine Learning

Import statements

We're going to start by importing the SparkConf and SparkContext libraries that we need for any Spark script that we run in Python, and then we're going to import HashingTF and IDF using the following commands.

from pyspark import SparkConf, SparkContext 
from pyspark.mllib.feature import HashingTF 
from pyspark.mllib.feature import IDF 

So, this is what computes the term frequencies (TF) and inverse document frequencies (IDF) within our documents.