Installing Hortonwork’s Hadoop on AWS

Rohit GhatolDirector of Engineering @ Synerzip

http://www.linkedin.com/in/rohitghatol @rohitghatol http://rohitghatol.com

BY

Software Stack

EC2

Hadoop

Hadoop

Hadoop

Hadoop

Hadoop

Hadoop

Controls &

Monitors

Hadoop Services• HDFS• YARN• Hbase• Hive• Pig• Sqoop• Oozie• Zookeeper• Nagios• Ganglia

Ambari

Apache Ambari

Step 1 – Create Base AMI

Community Red hat AMI Base AMI

6

SSHDownload PEM Key

Step 2 – Password Less SSH AccessLaptop accessing EC2 Instance EC2 Instance accessing other

EC2 Instances

AccessUsingSSH

Upload PEM Key

EC2(Ambari Server)

EC2

EC2

EC2

EC2

AMI

EC2

EC2

EC2

EC2

AMI

EC2

EC2

EC2

Step 3 – Launch 6 Instances using Base Image

Launch

6

Note the Private DNS names of all these machines• ip-10-23-12-33• ip-10-23-11-23• ip-10-23-32-54• ip-10-23-44-14• ip-10-23-65-73• ip-10-23-37-47

Install Ambari Server

EC2

EC2EC2

Step 4 – Install Ambari Server on EC2

EC2

Supply the earlier recorded private DNS to Ambari• ip-10-23-12-33• ip-10-23-11-23• ip-10-23-32-54• ip-10-23-44-14• ip-10-23-65-73• ip-10-23-37-47

Supply list of private DNS

EC2 EC2 EC2

EC2 EC2 EC2

Ambari Server Installs Ambari Agent

EC2

EC2

EC2

EC2

EC2

EC2

EC2

Installsservices

Hadoop Services• HDFS• YARN• Hbase• Hive• Pig• Sqoop• Oozie• Zookeeper• Nagios• Ganglia

Step 5 – Install X service on Y machine

EC2 withAmbari

Step 6 – Launch Ambari Web Consolehttp://<<ambari-server>>:8080