R in a Nutshell, 2nd Edition

Chapter 26. R and Hadoop

One of the best techniques for speeding up large computing problems is to break them into lots of little pieces, solve the pieces separately, and then put the pieces back together. This is called parallel computing, because it enables you to solve problems in parallel. For example, suppose that you had a lot of laundry: enough to fill 10 washing machines. Suppose each wash took 45 minutes, and each drying took 45 minutes. If you had only one washing machine and dryer, it would take 495 minutes to finish all the laundry. However, if you had 10 washing machines and 10 dryers, you could finish the laundry in 90 minutes.

In Chapters 20 and 21, we showed some cutting-edge techniques for statistical modeling. Many of these problems are very computationally intensive and could take a long time to finish. Luckily, many of them are very parallelizable. For example, we will show several algorithms that build models by fitting a large number of tree models to the underlying data (such as boosting, bagging, and random forests). Each of these algorithms could be run in parallel if more processors were available.

There are many packages available for parallel and distributed computing with R. In this chapter, I’ve focused on Hadoop-based solutions. Hadoop isn’t the best solution for all problems, but it’s popular, open source, and runs well in the cloud.