O'Reilly logo

Exercises in Programming Style by Cristina Videira Lopes

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Chapter 30

Map Reduce

image

30.1 Constraints

  • Input data is divided in blocks.
  • A map function applies a given worker function to each block of data, potentially in parallel.
  • A reduce function takes the results of the many worker functions and recombines them into a coherent output.

30.2 A Program in this Style

  1 #!/usr/bin/env python
  2 import sys, re, operator, string
  3
  4 #
  5 # Functions for map reduce
  6 #
  7 def partition(data_str, nlines):
  8 """
  9 Partitions the input data_str (a big string)
 10 into chunks of nlines.
 11 """
 12 lines = data_str.split('\n')
 13 for i in xrange(0, len(lines), nlines):
 14    yield '\n'.join(lines[i:i+nlines])
 15
 16 def ...

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required