Data Science on the Google Cloud Platform, 2nd Edition

Chapter 6. Bayesian Classifier with Apache Spark on Cloud Dataproc

Having become accustomed to running queries in BigQuery where there were no clusters to manage, I’m dreading going back to configuring and managing Hadoop clusters. But I did promise you a tour of data science on the cloud, and in many companies, Hadoop plays an important role in that.

In this chapter, we tackle the next stage of our data science problem, by creating a Bayes model to predict the likely arrival delay of a flight. We will do this through an integrated workflow that involves BigQuery and Spark SQL.

All of the code snippets in this chapter are available in the folder 06_dataproc of the book’s GitHub repository. See the README.md file in that directory for instructions on how to do the steps described in this chapter.

MapReduce and the Hadoop Ecosystem

MapReduce was described in a paper by Jeff Dean and Sanjay Ghemawat as a way to process large datasets on a cluster of machines. They showed that many real-world tasks can be decomposed into a sequence of two types of functions: map functions that process key-value pairs to generate intermediate key-value pairs, and reduce functions that merge all the intermediate values associated with the same key. A flexible and general-purpose framework can run programs that are written following this MapReduce model on a cluster of commodity machines. Such a MapReduce framework will take care of many of the details that make writing distributed system applications ...