Manage Hadoop Cluster with Ambari

54
Mr. Charnsilp Chinprasert Email: [email protected] Apache Ambari A Hadoop management simpler

Transcript of Manage Hadoop Cluster with Ambari

Page 1: Manage Hadoop Cluster with Ambari

Mr. Charnsilp Chinprasert Email: [email protected]

Apache Ambari A Hadoop management simpler

Page 2: Manage Hadoop Cluster with Ambari

Apache Hadoop

The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models.

Page 3: Manage Hadoop Cluster with Ambari

Hadoop Cluster

Page 4: Manage Hadoop Cluster with Ambari

Install Cluster ยังไง?

Page 5: Manage Hadoop Cluster with Ambari

แก้ไข Config ทั้ง Cluster ยังไง?

Page 6: Manage Hadoop Cluster with Ambari

Hadoop Ecosystem

Page 7: Manage Hadoop Cluster with Ambari

Install แต่ละ Service ยังไง?

Page 8: Manage Hadoop Cluster with Ambari

Config ยังไง

แต่ละ Service ไม่เหมือนกัน?

Page 9: Manage Hadoop Cluster with Ambari

จัดการแต่ละ Service ยังไง?

Page 10: Manage Hadoop Cluster with Ambari

จะรู้ได้ยังไงว่า Service ยังปกติ?

Page 11: Manage Hadoop Cluster with Ambari
Page 12: Manage Hadoop Cluster with Ambari

What is Apache Ambari?

The Apache Ambari a 100% open source platform provisioning, managing and monitoring Apache Hadoop Cluster

Page 13: Manage Hadoop Cluster with Ambari

Hadoop Ecosystem

Page 14: Manage Hadoop Cluster with Ambari

Apache Ambari

Provisioning

Monitoring Management

Page 15: Manage Hadoop Cluster with Ambari

Apache Ambari• Agent Installation• Service Installation

• Service Management• Cluster Configuration

• OverAll Dashboard• By Host• By Service• Alert Trigger

Provisioning

Monitoring Management

Page 16: Manage Hadoop Cluster with Ambari

Welcome Ambari!

Page 17: Manage Hadoop Cluster with Ambari

Install Cluster ยังไง?

Page 18: Manage Hadoop Cluster with Ambari

System Architecture

Ambari Server

Linux Server

Linux Server

Ambari AgentLinux

Server

Ambari Agent

Linux Server

Ambari Agent

Ambari Agent

Port: 8440/8441(SSL)

Ambari Web

Ambari Api

Page 19: Manage Hadoop Cluster with Ambari

Installation Agent❖ Provisioning by remote Secure Shell (SSH)

❖ OS Support

❖ RHEL (Redhat Enterprise Linux)

❖ SLES (SuSE Linux Enterprise Server)

❖ Ubuntu

❖ Debian

Ambari Server

Linux Server

Ambari Server

Linux Server

Ambari Agent

SSH SSH

ambari-server start

Page 20: Manage Hadoop Cluster with Ambari

Setup Cluster

Page 21: Manage Hadoop Cluster with Ambari

Installation Agent

• Cluster name• Stacks version

• Remote Repos• Local Repos

Page 22: Manage Hadoop Cluster with Ambari

Installation Agent• Target Hosts

• FQDN• SSH private key

Page 23: Manage Hadoop Cluster with Ambari

Installation AgentHost Checks

• มีกระบวนการตรวจสอบและconfig OS ให้พร้อมกัน Environment ของHadoop

• ทำให้ Linux ที่นำมาติดตั้งไม่จำเป็นต้อง ลงApplication ใดๆเลย • ถ้ายังไม่ได้ Install หรือ Config Ambari จะจัดการให้• ใช้ Puppet

Page 24: Manage Hadoop Cluster with Ambari

Install & Configure ServicesSelect

Services

Assign Masters

Assign Slaves & Clients

Customize Services

Review & Confirm

Page 25: Manage Hadoop Cluster with Ambari

Install & Configure ServicesSelect

Services

Assign Masters

Assign Slaves & Clients

Customize Services

Review & Confirm

Page 26: Manage Hadoop Cluster with Ambari

Install & Configure ServicesSelect

Services

Assign Masters

Assign Slaves & Clients

Customize Services

Review & Confirm

Page 27: Manage Hadoop Cluster with Ambari

Install & Configure ServicesSelect

Services

Assign Masters

Assign Slaves & Clients

Customize Services

Review & Confirm

Page 28: Manage Hadoop Cluster with Ambari

Install & Configure ServicesSelect

Services

Assign Masters

Assign Slaves & Clients

Customize Services

Review & Confirm

Page 29: Manage Hadoop Cluster with Ambari

Installing

Page 30: Manage Hadoop Cluster with Ambari

Installing

Page 31: Manage Hadoop Cluster with Ambari

Installing

Page 32: Manage Hadoop Cluster with Ambari

Welcome Ambari!

Page 33: Manage Hadoop Cluster with Ambari

จัดการแต่ละ Service ยังไง?

Page 34: Manage Hadoop Cluster with Ambari

Service ManagementHDFS Yarn

Service Management

Cluster Configuration

Move namenode ได้

Page 35: Manage Hadoop Cluster with Ambari

Move namenodeService

Management

Cluster Configuration

Page 36: Manage Hadoop Cluster with Ambari

Move namenodeService

Management

Cluster Configuration

Page 37: Manage Hadoop Cluster with Ambari

Move namenodeService

Management

Cluster Configuration

Page 38: Manage Hadoop Cluster with Ambari

Move namenodeService

Management

Cluster Configuration

Page 39: Manage Hadoop Cluster with Ambari

Move namenodeService

Management

Cluster Configuration

Page 40: Manage Hadoop Cluster with Ambari

แก้ไขConfig ทั้ง Cluster ยังไง?

Page 41: Manage Hadoop Cluster with Ambari

Cluster Configuration

Config ที่เดียว มีผลทั้งCluster

Basic View สำหรับ Configส่วนที่จำเป็นและสำคัญ!!

Service Management

Cluster Configuration

Page 42: Manage Hadoop Cluster with Ambari

Cluster Configuration

ถ้ามีแก้ไขConfig ระบบจะแนะนำConfigตัวอื่นๆที่ตวรแก้ไขตาม

Service Management

Cluster Configuration

Page 43: Manage Hadoop Cluster with Ambari

Cluster Configuration

Compare Code ระหว่าง version ได้!!!

Service Management

Cluster Configuration

Page 44: Manage Hadoop Cluster with Ambari

Configuration History LogService

Management

Cluster Configuration

Page 45: Manage Hadoop Cluster with Ambari

จะรู้ได้ไงว่าที่เราConfigไปแล้วมันดี?

Page 46: Manage Hadoop Cluster with Ambari

จะรู้ได้ยังไงว่า Service ยังปกติ?

Page 47: Manage Hadoop Cluster with Ambari

Monitoring

Page 48: Manage Hadoop Cluster with Ambari

DashboardDashboard

By Service

By Host

Alert Trigger

Page 49: Manage Hadoop Cluster with Ambari

By ServiceDashboard

By Service

By Host

Alert Trigger

Page 50: Manage Hadoop Cluster with Ambari

By HostDashboard

By Service

By Host

Alert Trigger

Page 51: Manage Hadoop Cluster with Ambari

Alert TriggerDashboard

By Service

By Host

Alert Trigger

Page 52: Manage Hadoop Cluster with Ambari

Apache Ambari• Agent Installation• Service Installation

• Service Management• Cluster Configuration

• OverAll Dashboard• By Host• By Service• Alert Trigger

Provisioning

Monitoring Management

Page 53: Manage Hadoop Cluster with Ambari

Q&A

• https://ambari.apache.org• https://issues.apache.org/jira/secure/attachment/12559939/Ambari_Architecture.pdf• http://www.slideshare.net/hortonworks/ambari-agentregistrationflow-17041261• https://cwiki.apache.org/confluence/display/AMBARI/Technology+Stack• http://www.throughwave.co.th/2011/11/10/enterprise-apache-hadoop/• http://www.slideshare.net/hortonworks/apache-ambari-whats-new-in-200

Page 54: Manage Hadoop Cluster with Ambari

Thank you