book

Parallel R

by Q. Ethan McCallum, Stephen Weston

October 2011

Intermediate to advanced

126 pages

3h 10m

English

O'Reilly Media, Inc.

Read now

Unlock full access

Conventions Used in This BookUsing Code ExamplesSafari® Books OnlineHow to Contact UsAcknowledgmentsQ. Ethan McCallumStephen Weston
Why R?Why Not R?The Solution: Parallel ExecutionA Road Map for This BookWhat We’ll CoverLooking Forward…What We’ll Assume You Already KnowIn a Hurry?snowmulticoreparallelR+HadoopRHIPESegueSummary
Quick LookHow It WorksSetting UpWorking with ItCreating Clusters with makeClusterParallel K-MeansInitializing WorkersLoad Balancing with clusterApplyLBTask Chunking with parLapplyVectorizing with clusterSplitLoad Balancing ReduxFunctions and EnvironmentsRandom Number Generationsnow ConfigurationInstalling RmpiExecuting snow Programs on a Cluster with RmpiExecuting snow Programs with a Batch Queueing SystemTroubleshooting snow ProgramsWhen It Works……And When It Doesn’tThe Wrap-up
Quick LookHow It WorksSetting UpWorking with ItThe mclapply FunctionThe mc.cores OptionThe mc.set.seed OptionLoad Balancing with mclapplyThe pvec FunctionThe parallel and collect FunctionsUsing collect OptionsParallel Random Number GenerationThe Low-Level APIWhen It Works……And When It Doesn’tThe Wrap-up
Quick LookHow It WorksSetting UpWorking with ItGetting StartedCreating Clusters with makeClusterParallel Random Number GenerationSummary of DifferencesWhen It Works……And When It Doesn’tThe Wrap-up
Hadoop at Cruising AltitudeA MapReduce PrimerThinking in MapReduce: Some Pseudocode ExamplesCalculate Average Call Length for Each DateNumber of Calls by Each User, on Each DateRun a Special Algorithm on Each RecordBinary and Whole-File Data: SequenceFilesNo Cluster? No Problem! Look to the Clouds…The Wrap-up
Quick LookHow It WorksSetting UpWorking with ItSimple Hadoop Streaming (All Text)Streaming, Redux: Indirectly Working with Binary DataThe Java API: Binary Input and OutputProcessing Related Groups (the Full Map and Reduce Phases)When It Works……And When It Doesn’tThe Wrap-up

Quick LookHow It WorksSetting UpWorking with ItPhone Call Records, ReduxTweet BrevityMore Complex Tweet AnalysisWhen It Works……And When It Doesn’tThe Wrap-up
Quick LookHow It WorksSetting UpWorking with ItModel Testing: Parameter SweepWhen It Works……And When It Doesn’tThe Wrap-up
doRedisRevoScale R and RevoConnectR (RHadoop)cloudNumbers.com

Content preview from Parallel R

Chapter 9. New and Upcoming

A perfect world would let us stop time to research and write, since a technical book covers a moving target. We didn’t have such a luxury, so instead we set aside some space to pick up on some new arrivals.

This chapter mentions a few tools for which we could have provided more coverage, had we been willing to postpone the book’s release date. Think of this as a look into one possible future of R parallelism. Special thanks to our colleagues, reviewers, and friends who so kindly brought these to our attention.

doRedis

The foreach() function^[62] executes an arbitrary R expression across an input. foreach()’s strength is that it can execute in parallel with the help of a supplied parallel backend. The doRedis package provides such a backend, using the Redis datastore^[63] as a job queue.

doRedis can work locally to take advantage of multicore systems, and also farm tasks out to remote R instances (“workers”). It’s straightforward to add or remove workers at runtime—even in mid-job—to adapt to changing work conditions or speed up job processing. Similar to Hadoop, doRedis is fault-tolerant in that failed tasks are automatically resubmitted to their job queue.

doRedis supports Linux, Mac OS X, and Windows systems.