O'Reilly logo

Learning Real-time Processing with Spark Streaming by Sumit Gupta

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Configuring and running the Spark cluster

In this section, we will configure our Spark cluster so that we can deploy and execute our Spark application.

Spark essentially enables the distributed execution of a given piece of code. Though we will talk about Spark architecture in the next chapter, let's briefly talk about the major components which need to be configured for setting up the Spark cluster.

The following are the high-level components involved in setting up the Spark cluster:

  • Driver: It is the client program which defines SparkContext. It connects to the cluster manager and requests resources for further execution of the jobs in distributed mode.
  • Cluster manager / Spark master: Cluster manager manages and allocates the required system resources ...

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required