book

Parallel R

by Q. Ethan McCallum, Stephen Weston

October 2011

Intermediate to advanced

126 pages

3h 10m

English

O'Reilly Media, Inc.

Read now

Unlock full access

Conventions Used in This BookUsing Code ExamplesSafari® Books OnlineHow to Contact UsAcknowledgmentsQ. Ethan McCallumStephen Weston
Why R?Why Not R?The Solution: Parallel ExecutionA Road Map for This BookWhat We’ll CoverLooking Forward…What We’ll Assume You Already KnowIn a Hurry?snowmulticoreparallelR+HadoopRHIPESegueSummary
Quick LookHow It WorksSetting UpWorking with ItCreating Clusters with makeClusterParallel K-MeansInitializing WorkersLoad Balancing with clusterApplyLBTask Chunking with parLapplyVectorizing with clusterSplitLoad Balancing ReduxFunctions and EnvironmentsRandom Number Generationsnow ConfigurationInstalling RmpiExecuting snow Programs on a Cluster with RmpiExecuting snow Programs with a Batch Queueing SystemTroubleshooting snow ProgramsWhen It Works……And When It Doesn’tThe Wrap-up
Quick LookHow It WorksSetting UpWorking with ItThe mclapply FunctionThe mc.cores OptionThe mc.set.seed OptionLoad Balancing with mclapplyThe pvec FunctionThe parallel and collect FunctionsUsing collect OptionsParallel Random Number GenerationThe Low-Level APIWhen It Works……And When It Doesn’tThe Wrap-up
Quick LookHow It WorksSetting UpWorking with ItGetting StartedCreating Clusters with makeClusterParallel Random Number GenerationSummary of DifferencesWhen It Works……And When It Doesn’tThe Wrap-up
Hadoop at Cruising AltitudeA MapReduce PrimerThinking in MapReduce: Some Pseudocode ExamplesCalculate Average Call Length for Each DateNumber of Calls by Each User, on Each DateRun a Special Algorithm on Each RecordBinary and Whole-File Data: SequenceFilesNo Cluster? No Problem! Look to the Clouds…The Wrap-up
Quick LookHow It WorksSetting UpWorking with ItSimple Hadoop Streaming (All Text)Streaming, Redux: Indirectly Working with Binary DataThe Java API: Binary Input and OutputProcessing Related Groups (the Full Map and Reduce Phases)When It Works……And When It Doesn’tThe Wrap-up

Quick LookHow It WorksSetting UpWorking with ItPhone Call Records, ReduxTweet BrevityMore Complex Tweet AnalysisWhen It Works……And When It Doesn’tThe Wrap-up
Quick LookHow It WorksSetting UpWorking with ItModel Testing: Parameter SweepWhen It Works……And When It Doesn’tThe Wrap-up
doRedisRevoScale R and RevoConnectR (RHadoop)cloudNumbers.com

Overview

It’s tough to argue with R as a high-quality, cross-platform, open source statistical software product—unless you’re in the business of crunching Big Data. This concise book introduces you to several strategies for using R to analyze large datasets, including three chapters on using R and Hadoop together. You’ll learn the basics of Snow, Multicore, Parallel, Segue, RHIPE, and Hadoop Streaming, including how to find them, how to use them, when they work well, and when they don’t.

With these packages, you can overcome R’s single-threaded nature by spreading work across multiple CPUs, or offloading work to multiple machines to address R’s memory barrier.

Snow: works well in a traditional cluster environment
Multicore: popular for multiprocessor and multicore computers
Parallel: part of the upcoming R 2.14.0 release
R+Hadoop: provides low-level access to a popular form of cluster computing
RHIPE: uses Hadoop’s power with R’s language and interactive shell
Segue: lets you use Elastic MapReduce as a backend for lapply-style operations