Hands-On Big Data Analytics with PySpark

What is parallelization?

The best way to understand Spark, or any language, is to look at the documentation. If we look at Spark's documentation, it clearly states that, for the textFile function that we used last time, it reads the text file from HDFS.

On the other hand, if we look at the definition of parallelize, we can see that this is creating an RDD by distributing a local Scala collection.

So, the main difference between using parallelize to create an RDD and using the textFile to create an RDD is where the data is sourced from.

Let's look at how this works practically. Let's go to the PySpark installation screen, from where we left off previously. So, we imported urllib, we used urllib.request to retrieve some data from the internet, ...