Reducer implementation

At this point in our MapReduce run, the mappers have run and eventually write their intermediate output data to S3. Mappers were triggered by an invocation of SNS events on a given SNS topic. We will set up the reducers to be triggered based on an s3:ObjectCreated event. Taking a look at the serverless.yml file, we can see how I've done this:

functions:  Reducer:    handler: handler.reducer    events:      - s3:          bucket: brianz-${env:ENV}-mapreduce-results          event: s3:ObjectCreated:*          rules:            - suffix: '-done.csv'

The s3 section in the events block says: Whenever a new object is uploaded to s3 with the -done.csv suffix, invoke the hander.reducer function.

Just as the mapper was reasonably straightforward, so too is the reducer. Much ...

Get Serverless Design Patterns and Best Practices now with the O’Reilly learning platform.

O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.