Exercises in Programming Style

Chapter 30

Map Reduce

30.1 Constraints

• Input data is divided in blocks.
• A map function applies a given worker function to each block of data, potentially in parallel.
• A reduce function takes the results of the many worker functions and recombines them into a coherent output.

30.2 A Program in this Style

  1 #!/usr/bin/env python
  2 import sys, re, operator, string
  3
  4 #
  5 # Functions for map reduce
  6 #
  7 def partition(data_str, nlines):
  8 """
  9 Partitions the input data_str (a big string)
 10 into chunks of nlines.
 11 """
 12 lines = data_str.split('\n')
 13 for i in xrange(0, len(lines), nlines):
 14  yield '\n'.join(lines[i:i+nlines])
 15
 16 def ...