High Performance Spark, 2nd Edition

Chapter 6. Spark Components and Packages

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 12th chapter of the final book. Please note that the GitHub repo is available at https://github.com/high-performance-spark/high-performance-spark-examples.

If you have comments about how we might improve the content and/or examples in this book, or if you notice missing material within this chapter, please reach out to the editor at [e-mail].

park has many components designed to work together as an integrated system, and many of them are distributed as part of Spark. This differs from much of the rest of the Hadoop ecosystem, which has different projects or systems for each task. You’ve already seen how to effectively use Spark Core, SQL, Streaming, and ML components. This chapter will look at the projects outside of Spark itself, sometimes called external/community components (often called packages). Having a largely integrated system gives Spark two advantages: it simplifies deployment/cluster management, upgrades, and application development by having fewer dependencies and systems to keep track of.

While Spark is comparatively integrated, there are still times when bringing in outside components is well worth the increased complexity. In this chapter, we’ll help ...