Yahoo! JAPAN の Ambari 活用事例 #yjdsnight

60
Copyright (C) 2016 Yahoo Japan Corporation. All Rights Reserved. 無断引・転載禁 ヤフー株式会社 データ&サイエンスソリューション統括本部 データプラットフォーム本部 開発2部グリッド 深澤 開 20160428 Yahoo! JAPAN の Ambari 活事

Transcript of Yahoo! JAPAN の Ambari 活用事例 #yjdsnight

  • Copyright (C) 2016 Yahoo Japan Corporation. All Rights Reserved.

    &

    2

    20160428

    Yahoo! JAPAN Ambari

  • Copyright (C) 2016 Yahoo Japan Corporation. All Rights Reserved.

    (2013)

    Hadoop

    ()Splatoon ()

  • Copyright (C) 2016 Yahoo Japan Corporation. All Rights Reserved.

    Ambari

    2015/03v1.7.0

    2015/05v2.0.0

    2015/07v2.1.0

    2015/10v2.1.1

    2015/11v2.1.2

    2016/02v2.2.0.0

    2016/05v2.2.2.0

  • Copyright (C) 2016 Yahoo Japan Corporation. All Rights Reserved.

    Yahoo! JAPAN Ambari Ambari

    (Yahoo! JAPAN )

    MoveNameNode

  • Copyright (C) 2016 Yahoo Japan Corporation. All Rights Reserved.

    Yahoo! JAPAN Ambari

    Cluster13PB

    (2011/06)

    Cluster220PB

    (2013/01)

    Cluster458PB

    (2015/12)

    Cluster330PB

    (2014/4)

    Cluster590PB

    (2016/06)

  • Copyright (C) 2016 Yahoo Japan Corporation. All Rights Reserved.

    Yahoo! JAPAN Ambari

    Cluster13PB

    (2011/06)

    Cluster220PB

    (2013/01)

    Cluster458PB

    (2015/12)

    Cluster330PB

    (2014/4)

    Cluster590PB

    (2016/06)

    2Ambari

  • Copyright (C) 2016 Yahoo Japan Corporation. All Rights Reserved.

    Yahoo! JAPAN Ambari

    Cluster13PB

    (2011/06)

    Cluster220PB

    (2013/01)

    Cluster458PB

    (2015/12)

    Cluster330PB

    (2014/4)

    Cluster590PB

    (2016/06)

    800 800

  • Copyright (C) 2016 Yahoo Japan Corporation. All Rights Reserved.

    Ambari

    https://ambari.apache.org/

    Hadoop100%

  • Copyright (C) 2016 Yahoo Japan Corporation. All Rights Reserved.

    Ambari

    View

  • Copyright (C) 2016 Yahoo Japan Corporation. All Rights Reserved.

    Ambari

    View

  • Copyright (C) 2016 Yahoo Japan Corporation. All Rights Reserved.

    Ambari

    View

  • Copyright (C) 2016 Yahoo Japan Corporation. All Rights Reserved.

    Ambari

    View

    AmbariHA

  • Copyright (C) 2016 Yahoo Japan Corporation. All Rights Reserved.

    Ambari

    View

  • Copyright (C) 2016 Yahoo Japan Corporation. All Rights Reserved.

  • Copyright (C) 2016 Yahoo Japan Corporation. All Rights Reserved.

    Agent

  • Copyright (C) 2016 Yahoo Japan Corporation. All Rights Reserved.

    Agent

    /

  • Copyright (C) 2016 Yahoo Japan Corporation. All Rights Reserved.

    Agent

    Agent

  • Copyright (C) 2016 Yahoo Japan Corporation. All Rights Reserved.

    Ambari Server

    Ambari Agent

  • Copyright (C) 2016 Yahoo Japan Corporation. All Rights Reserved.

    Ambari Server

    Ambari Agent

  • Copyright (C) 2016 Yahoo Japan Corporation. All Rights Reserved.

    Ambari Server

    Ambari Agent

  • Copyright (C) 2016 Yahoo Japan Corporation. All Rights Reserved.

    Master Node / Application

    Data Node

    Ambari Server

    Ambari Agent

  • Copyright (C) 2016 Yahoo Japan Corporation. All Rights Reserved.

    Master Node / Application

    Data Node

    Agent

    Ambari Server

    Ambari Agent

  • Copyright (C) 2016 Yahoo Japan Corporation. All Rights Reserved.

    Master Node / Application

    Data Node

    Agent

    Ambari Server

    Ambari Agent

    800 Node

  • Copyright (C) 2016 Yahoo Japan Corporation. All Rights Reserved.

    Master Node / Application

    Data Node

    Agent

    Ambari Server

    Ambari Agent

    800 Node

    800 Ambari Agent

  • Copyright (C) 2016 Yahoo Japan Corporation. All Rights Reserved.

    Master Node / Application

    Data Node

    Agent

    Ambari Server

    Ambari Agent

    800 Node

  • Copyright (C) 2016 Yahoo Japan Corporation. All Rights Reserved.

    Start/StopNode Decommission/Recommission

  • Copyright (C) 2016 Yahoo Japan Corporation. All Rights Reserved.

    Start/StopNode Decommission/Recommission

    WebUINode

  • Copyright (C) 2016 Yahoo Japan Corporation. All Rights Reserved.

    Start/StopNode Decommission/Recommission Node

  • Copyright (C) 2016 Yahoo Japan Corporation. All Rights Reserved.

  • Copyright (C) 2016 Yahoo Japan Corporation. All Rights Reserved.

  • Copyright (C) 2016 Yahoo Japan Corporation. All Rights Reserved.

  • Copyright (C) 2016 Yahoo Japan Corporation. All Rights Reserved.

  • Copyright (C) 2016 Yahoo Japan Corporation. All Rights Reserved.

    1000

  • Copyright (C) 2016 Yahoo Japan Corporation. All Rights Reserved.

    1000hdfs

    chmod r

  • Copyright (C) 2016 Yahoo Japan Corporation. All Rights Reserved.

    1000

  • Copyright (C) 2016 Yahoo Japan Corporation. All Rights Reserved.

    Service

  • Copyright (C) 2016 Yahoo Japan Corporation. All Rights Reserved.

    Service

    Config

  • Copyright (C) 2016 Yahoo Japan Corporation. All Rights Reserved.

    ServiceAmbariConfig

  • Copyright (C) 2016 Yahoo Japan Corporation. All Rights Reserved.

    MoveNameNode

  • Copyright (C) 2016 Yahoo Japan Corporation. All Rights Reserved.

    MoveNameNode

    NameNode

    AmbariMoveNameNode

    23-4

  • Copyright (C) 2016 Yahoo Japan Corporation. All Rights Reserved.

    MoveNameNode

  • Copyright (C) 2016 Yahoo Japan Corporation. All Rights Reserved.

    MoveNameNode

    sudo su hdfs l c hdfs namenode -bootstrapStandby

  • Copyright (C) 2016 Yahoo Japan Corporation. All Rights Reserved.

    MoveNameNode

  • Copyright (C) 2016 Yahoo Japan Corporation. All Rights Reserved.

    MoveNameNode

  • Copyright (C) 2016 Yahoo Japan Corporation. All Rights Reserved.

    MoveNameNode

  • Copyright (C) 2016 Yahoo Japan Corporation. All Rights Reserved.

    MoveNameNode

  • Copyright (C) 2016 Yahoo Japan Corporation. All Rights Reserved.

    MoveNameNode

  • Copyright (C) 2016 Yahoo Japan Corporation. All Rights Reserved.

    MoveNameNode

  • Copyright (C) 2016 Yahoo Japan Corporation. All Rights Reserved.

    MoveNameNode

  • Copyright (C) 2016 Yahoo Japan Corporation. All Rights Reserved.

    MoveNameNode

    NameNode3

  • Copyright (C) 2016 Yahoo Japan Corporation. All Rights Reserved.

    MoveNameNode

    NameNode3

  • Copyright (C) 2016 Yahoo Japan Corporation. All Rights Reserved.

    MoveNameNode

    NameNodeinstallBootstrap

  • Copyright (C) 2016 Yahoo Japan Corporation. All Rights Reserved.

    MoveNameNode

  • Copyright (C) 2016 Yahoo Japan Corporation. All Rights Reserved.

    MoveNameNode

  • Copyright (C) 2016 Yahoo Japan Corporation. All Rights Reserved.

    MoveNameNode

    NameNode4

  • Copyright (C) 2016 Yahoo Japan Corporation. All Rights Reserved.

    MoveNameNode

    NameNode4

    1 + APINameNode

  • Copyright (C) 2016 Yahoo Japan Corporation. All Rights Reserved.

    MoveNameNode

    NameNode4

    1 + APINameNode

    Bug

  • Copyright (C) 2016 Yahoo Japan Corporation. All Rights Reserved.

    WebUI

    I Ambari

  • Copyright (C) 2016 Yahoo Japan Corporation. All Rights Reserved.

    Appendix

    Ambari#yjdsw3http://www.slideshare.net/techblogyahoo/ambariyjdsw3Yahoo! JAPAN Ambari #ambarimeetup

    http://www.slideshare.net/techblogyahoo/yahoo-japan-ambari-ambarimeetup

  • Copyright (C) 2016 Yahoo Japan Corporation. All Rights Reserved.

    Copyright (C) 2015 Yahoo Japan Corporation. All Rights Reserved.


Ambari Meetup: 2nd April 2013: Teradata Viewpoint Hadoop Integration with Ambari

Ambari Meetup: 2nd April 2013: Teradata Viewpoint Hadoop Integration with Ambari

An Overview of Ambari

An Overview of Ambari

Hortonworks Data Platform - Apache Ambari Minor …...Hortonworks Data Platform November 15, 2018 2 2. Getting Ready to Upgrade Ambari and HDP When preparing to upgrade Ambari and

Hortonworks Data Platform - Apache Ambari Minor …...Hortonworks Data Platform November 15, 2018 2 2. Getting Ready to Upgrade Ambari and HDP When preparing to upgrade Ambari and

Apache ambari

Apache ambari

Apache Ambari: Past, Present, Future

Apache Ambari: Past, Present, Future

Apache Ambari - What's New in 1.6.0

Apache Ambari - What's New in 1.6.0

Presto in Yahoo! JAPAN #yjdsnight

Presto in Yahoo! JAPAN #yjdsnight

Apache Ambari - What's New in 2.0.0

Apache Ambari - What's New in 2.0.0

Hortonworks Data Platform - Apache Ambari Major Upgrade · After adding or resetting the Ambari NameNode restart parameter, restart your Ambari server before starting the HDP upgrade.

Hortonworks Data Platform - Apache Ambari Major Upgrade · After adding or resetting the Ambari NameNode restart parameter, restart your Ambari server before starting the HDP upgrade.

Using Ambari Book-20131022

Using Ambari Book-20131022

Hortonworks Technical Workshop: Apache Ambari

Hortonworks Technical Workshop: Apache Ambari

Apache Ambari - What's New in 1.2.5

Apache Ambari - What's New in 1.2.5

Ambari Meetup: Ambari Futures

Ambari Meetup: Ambari Futures

可視化までのとある方法 #yjdsnight

可視化までのとある方法 #yjdsnight

Ambari Meetup: APIs and SPIs of Ambari

Ambari Meetup: APIs and SPIs of Ambari

 · O"ganize /VaÆtigaé AMBARNAKLiYAT Ambar Nakliyat, Gebze nakliyat ambari, istanbul Nakliyat Ambari, Erzurum Nakliyat Ambari, izmir Nakliya Ambari, Bodrum Nakliyat

 · O"ganize /VaÆtigaé AMBARNAKLiYAT Ambar Nakliyat, Gebze nakliyat ambari, istanbul Nakliyat Ambari, Erzurum Nakliyat Ambari, izmir Nakliya Ambari, Bodrum Nakliyat

Setup HSP Ambari Cluster - Connect › download › epcra › hsp0171.pdfHortonworks Ambari VM for HSP Deployment Guide MK-95HSP017-01 1 Setup HSP Ambari Cluster Prerequisites 1. An

Setup HSP Ambari Cluster - Connect › download › epcra › hsp0171.pdfHortonworks Ambari VM for HSP Deployment Guide MK-95HSP017-01 1 Setup HSP Ambari Cluster Prerequisites 1. An

Bk Using Ambari Book-20130114

Bk Using Ambari Book-20130114

BlueData Integration with Apache Ambari

BlueData Integration with Apache Ambari

Ambari Install v20

Ambari Install v20