Hadoop Cluster using Whirr – BYON in AWS VPC – Part 1


In this article series we will look at the steps in creating a Whirr base instance which will be used to launch the Hadoop cluster over your custom created instances  “using AWS VPC – BYON(Bring Your Own Network)”, ie., on pre-setup machine instances in AWS VPC. In this article we will focus on how to create a Whirr base instance. 

Key challenge in getting a hadoop cluster through Whirr – BYON over Amazon AWS VPC, is that each instance in the Hadoop clusters should have a hostname which is traceable both by forward and reverse dns lookup in their network. Any AWS VPC instance is not assigned with any dns names(only ip address) and is not associated with a local dns.  So we will have to get a local dns server setup in the Whirr base instance and then setup Whirr with Open JDK. In the local dns we will configure the dns names and ip address of instances which we will use for hadoop cluster launch.

Creating Whirr base template with ability to launch BYON.

To start with launch default 12.04.1 ubuntu instance (small)

$ sudo su

Step1: Install DNS Server: bind9

$ apt-get install bind9
$ cd /etc/bind9

Step2: Forward dns lookup setup: db.<dns domain>

Make decision on your local dns name say, “ck.local”. Current machine will be the SOA & NameServer(NS) for the domain. Calling current machine (hostname) as “dc”, meaning dc.ck.local.

Make a copy of the db.local file as db.ck.local

$ cp db.local db.ck.local

Edit and update as below:

With SOA as dc.ck.local & administrator as root.ck.local (instead of root@ck.local)

I wanted to configure 3 more machine in dns other than the current dc.ck.local (10.0.1.80 in my case)

Make the others as node1.ck.local(10.0.1.220), node2.ck.local(10.0.1.221), node3.ck.local(10.0.1.222)

db_ck_local

Step3: Reverse Lookup db.<ip address range in reverse>

Now we need to create reverse lookup, copy the db.0 and name it as db.1.0.10 (where all my machines are going to be in the ip ranges of 10.0.1.* and update as below.

db_1_0_10

Step 4: Include in Name configuration

Next step is to update the bind name configuration to include all these files.

$ vi named.conf.default-zones

update as below, to include both the db.ck.local (instead of the db.local) & update a new entry for db.1.0.10

named_local

Step5: Update External DNS forwarder

Now the next update, is to configure the forwarder for the external dns names to the (default dns server for VPC 10.0.0.2)

$ vi named.conf.options

named_conf

Step6: Update the current machine hostname & nameserver configurations.

In this Ubuntu it is indirectly controlled by dhcp configuration and editing /etc/resolv.conf is the not the right idea.

$ vi /etc/dhcp/dhclient.conf

Whirr-BYON

Update the set host-name to hostname and then uncomment the “supersede domain-name” and update to the chosen domain name, then uncomment the prepend domain-name-servers and update it with the current machine as the dns server “10.0.1.80” (I have updated the vpc dns server[10.0.0.2] which is not really required)

Now make sure the bind9 is part of startup service. I did that using

$ chkconfig bind9 on

If you don’t have the chkconfig install first using apt-get install chkconfig.

$ reboot

Now you have a DNS Server ready.

Step7: Next Step is to get the OpenJDK, Whirr downloaded as below.

Whirr:

$ wget http://apache.techartifact.com/mirror/whirr/stable/whirr-0.8.1.tar.gz

$ tar –xzf whirr-0.8.1.tar.gz

OpenJDK:

Follow the instructions as in

http://www.mkyong.com/java/how-to-install-java-jdk-on-ubuntu-linux/

That is it you are now ready with the Whirr base. Create a Amazon Machine Image(AMI) out of this and should be useful to get a whirr base instance on demand. Next document will give inputs on how to launch a Hadoop cluster via Whirr – BYON service provider.

Reference:

http://aws.amazon.com/vpc

http://whirr.apache.org

2 thoughts on “Hadoop Cluster using Whirr – BYON in AWS VPC – Part 1

  1. Pingback: Hadoop Cluster using Whirr – BYON in AWS VPC – Part 2 | blog@cloud-kinetics

  2. Pingback: Revue de liens du 8 février 2013 | Blog Affini-Tech

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s