book

MapReduce Design Patterns

by Donald Miner, Adam Shook

December 2012

Intermediate to advanced

247 pages

6h 48m

English

O'Reilly Media, Inc.

Read now

Unlock full access

Intended AudiencePattern FormatThe Examples in This BookConventions Used in This BookUsing Code ExamplesSafari® Books OnlineHow to Contact UsAcknowledgments
Design PatternsMapReduce HistoryMapReduce and Hadoop RefresherHadoop Example: Word CountPig and Hive
Numerical SummarizationsPattern DescriptionIntentMotivationApplicabilityStructureConsequencesKnown usesResemblancesPerformance analysisNumerical Summarization ExamplesMinimum, maximum, and count exampleMinMaxCountTuple codeMapper codeReducer codeCombiner optimizationData flow diagramAverage exampleMapper codeReducer codeCombiner optimizationData flow diagramMedian and standard deviationMapper codeReducer codeCombiner optimizationMemory-conscious median and standard deviationMapper codeReducer codeCombiner optimizationData flow diagramInverted Index SummarizationsPattern DescriptionIntentMotivationApplicabilityStructureConsequencesPerformance analysisInverted Index ExampleWikipedia reference inverted indexMapper codeReducer codeCombiner optimizationCounting with CountersPattern DescriptionIntentMotivationApplicabilityStructureConsequencesKnown usesPerformance analysisCounting with Counters ExampleNumber of users per stateMapper codeDriver code
FilteringPattern DescriptionIntentMotivationApplicabilityStructureConsequencesKnown usesResemblancesPerformance analysisFiltering ExamplesDistributed grepMapper codeSimple Random SamplingMapper CodeBloom FilteringPattern DescriptionIntentMotivationApplicabilityStructureConsequencesKnown usesResemblancesPerformance analysisBloom Filtering ExamplesHot listBloom filter trainingMapper codeHBase Query using a Bloom filterMapper CodeTop TenPattern DescriptionIntentMotivationApplicabilityStructureConsequencesKnown usesResemblancesPerformance analysisTop Ten ExamplesTop ten users by reputationMapper codeReducer codeDistinctPattern DescriptionIntentMotivationApplicabilityStructureConsequencesKnown usesResemblancesPerformance analysisDistinct ExamplesDistinct user IDsMapper codeReducer codeCombiner optimization
Structured to HierarchicalPattern DescriptionIntentMotivationApplicabilityStructureConsequencesKnown usesResemblancesPerformance analysisStructured to Hierarchical ExamplesPost/comment building on StackOverflowDriver codeMapper codeReducer codeQuestion/answer building on StackOverflowMapper codeReducer codePartitioningPattern DescriptionIntentMotivationApplicabilityStructureConsequencesKnown usesResemblancesPerformance analysisPartitioning ExamplesPartitioning users by last access dateDriver codeMapper codePartitioner codeReducer codeBinningPattern DescriptionIntentMotivationStructureConsequencesResemblancesPerformance analysisBinning ExamplesBinning by Hadoop-related tagsDriver codeMapper codeTotal Order SortingPattern DescriptionIntentMotivationApplicabilityStructureConsequencesResemblancesPerformance analysisTotal Order Sorting ExamplesSort users by last visitDriver codeAnalyze mapper codeOrder mapper codeOrder reducer codeShufflingPattern DescriptionIntentMotivationStructureConsequencesResemblancesPerformance analysisShuffle ExamplesAnonymizing StackOverflow commentsMapper codeReducer code
A Refresher on JoinsReduce Side JoinPattern DescriptionIntentMotivationApplicabilityStructureConsequencesResemblancesPerformance analysisReduce Side Join ExampleUser and comment joinDriver codeUser mapper codeComment mapper codeReducer codeCombiner optimizationReduce Side Join with Bloom FilterReputable user and comment joinUser mapper codeComment mapper codeReplicated JoinPattern DescriptionIntentMotivationApplicabilityStructureConsequencesResemblancesPerformance analysisReplicated Join ExamplesReplicated user comment exampleMapper codeComposite JoinPattern DescriptionIntentMotivationApplicabilityStructureConsequencesPerformance analysisComposite Join ExamplesComposite user comment joinDriver codeMapper codeReducer and combinerCartesian ProductPattern DescriptionIntentMotivationApplicabilityStructureConsequencesResemblancesPerformance AnalysisCartesian Product ExamplesComment ComparisonInput format codeDriver codeRecord reader codeMapper code
Job ChainingWith the DriverJob Chaining ExamplesBasic job chainingJob one mapperJob one reducerJob two mapperDriver codeParallel job chainingMapper codeReducer codeDriver codeWith Shell ScriptingBash exampleBash scriptSample runWith JobControlJob control exampleMain methodHelper methodsChain FoldingThe ChainMapper and ChainReducer ApproachChain Folding ExampleBin users by reputationParsing mapper codeReplicated join mapper codeReducer codeBinning mapper codeDriver codeJob MergingJob Merging ExamplesAnonymous comments and distinct usersTaggedText WritableComparableMerged mapper codeMerged reducer codeDriver code
Customizing Input and Output in HadoopInputFormatRecordReaderOutputFormatRecordWriterGenerating DataPattern DescriptionIntentMotivationStructureConsequencesResemblancesPerformance analysisGenerating Data ExamplesGenerating random StackOverflow commentsDriver codeInputSplit codeInputFormat codeRecordReader codeExternal Source OutputPattern DescriptionIntentMotivationStructureConsequencesPerformance analysisExternal Source Output ExampleWriting to Redis instancesOutputFormat codeRecordWriter codeMapper CodeDriver CodeExternal Source InputPattern DescriptionIntentMotivationStructureConsequencesPerformance analysisExternal Source Input ExampleReading from Redis InstancesInputSplit codeInputFormat codeRecordReader codeDriver codePartition PruningPattern DescriptionIntentMotivationStructureConsequencesResemblancesPerformance analysisPartition Pruning ExamplesPartitioning by last access date to Redis instancesCustom WritableComparable codeOutputFormat codeRecordWriter codeMapper codeDriver codeQuerying for user reputation by last access dateInputSplit codeInputFormat codeRecordReader codeDriver code
Trends in the Nature of DataImages, Audio, and VideoStreaming DataThe Effects of YARNPatterns as a Library or ComponentHow You Can Help