Configuring a Hadoop cluster with the new AMI

Starting a Hadoop cluster with the new AMI is simple and straightforward. This recipe will list steps to start up a Hadoop cluster with the new AMI.

Getting ready

Before getting started, we assume that you have registered with AWS and have successfully created a new AMI with Hadoop properly configured.

How to do it...

Use the following steps to configure a Hadoop cluster with EC2:

  1. Run a number of instances either from the command line or from the web interface.
  2. After the instances are all in running state, run the following command to get the internal hostname of these instances:
    ec2-describe-instances | grep running | egrep -o 'ip.*?internal' | sed -e 's/.ec2.internal//g' > nodes.txt
    

    The nodes.txt file ...

Get Hadoop Operations and Cluster Management Cookbook now with O’Reilly online learning.

O’Reilly members experience live online training, plus books, videos, and digital content from 200+ publishers.