Learning Spark, 2nd Edition

Foreword

Apache Spark has evolved significantly since I first started the project at UC Berkeley in 2009. After moving to the Apache Software Foundation, the open source project has had over 1,400 contributors from hundreds of companies, and the global Spark meetup group has grown to over half a million members. Spark’s user base has also become highly diverse, encompassing Python, R, SQL, and JVM developers, with use cases ranging from data science to business intelligence to data engineering. I have been working closely with the Apache Spark community to help continue its development, and I am thrilled to see the progress thus far.

The release of Spark 3.0 marks an important milestone for the project and has sparked the need for updated learning material. The idea of a second edition of Learning Spark has come up many times—and it was overdue. Even though I coauthored both Learning Spark and Spark: The Definitive Guide (both O’Reilly), it was time for me to let the next generation of Spark contributors pick up the narrative. I’m delighted that four experienced practitioners and developers, who have been working closely with Apache Spark from its early days, have teamed up to write this second edition of the book, incorporating the most recent APIs and best practices for Spark developers in a clear and informative guide.

The authors’ approach to this edition is highly conducive to hands-on learning. The key concepts in Spark and distributed big data processing have been distilled ...