Hadoop YARN - Under the Hoodarchive.apachecon.com/eu2012/presentations/06... · Node Manager...
Transcript of Hadoop YARN - Under the Hoodarchive.apachecon.com/eu2012/presentations/06... · Node Manager...
![Page 2: Hadoop YARN - Under the Hoodarchive.apachecon.com/eu2012/presentations/06... · Node Manager Container Container Node Manager App Mstr Container Node Manager Container App Mstr Node](https://reader030.fdocuments.net/reader030/viewer/2022040922/5e9bd0f45e900742fe5bb7df/html5/thumbnails/2.jpg)
About me
![Page 3: Hadoop YARN - Under the Hoodarchive.apachecon.com/eu2012/presentations/06... · Node Manager Container Container Node Manager App Mstr Container Node Manager Container App Mstr Node](https://reader030.fdocuments.net/reader030/viewer/2022040922/5e9bd0f45e900742fe5bb7df/html5/thumbnails/3.jpg)
Recap: Hadoop 1.0 Map-Reduce JobTracker
Manages cluster resources and job scheduling
TaskTracker Per-node agent Manage tasks
![Page 4: Hadoop YARN - Under the Hoodarchive.apachecon.com/eu2012/presentations/06... · Node Manager Container Container Node Manager App Mstr Container Node Manager Container App Mstr Node](https://reader030.fdocuments.net/reader030/viewer/2022040922/5e9bd0f45e900742fe5bb7df/html5/thumbnails/4.jpg)
YARN Architecture
ResourceManager
MapReduce Status
Job Submission
Client
NodeManager
NodeManager
Container
NodeManager
App Mstr
Node Status
Resource Request
ResourceManager
Client
MapReduce Status
Job Submission
Client
NodeManager
NodeManager
App Mstr Container
NodeManager
App Mstr
Node Status
Resource Request
ResourceManager
Client
MapReduce Status
Job Submission
Client
NodeManager
Container Container
NodeManager
App Mstr Container
NodeManager
Container App Mstr
Node Status
Resource Request
![Page 5: Hadoop YARN - Under the Hoodarchive.apachecon.com/eu2012/presentations/06... · Node Manager Container Container Node Manager App Mstr Container Node Manager Container App Mstr Node](https://reader030.fdocuments.net/reader030/viewer/2022040922/5e9bd0f45e900742fe5bb7df/html5/thumbnails/5.jpg)
What the new Architecture gets us?
Scale Compute Platform
![Page 6: Hadoop YARN - Under the Hoodarchive.apachecon.com/eu2012/presentations/06... · Node Manager Container Container Node Manager App Mstr Container Node Manager Container App Mstr Node](https://reader030.fdocuments.net/reader030/viewer/2022040922/5e9bd0f45e900742fe5bb7df/html5/thumbnails/6.jpg)
Scale for a compute platform • Application Size
• No of sub-tasks • Application level state
• eg. Counters
• Number of Concurrent Tasks in a single cluster
![Page 7: Hadoop YARN - Under the Hoodarchive.apachecon.com/eu2012/presentations/06... · Node Manager Container Container Node Manager App Mstr Container Node Manager Container App Mstr Node](https://reader030.fdocuments.net/reader030/viewer/2022040922/5e9bd0f45e900742fe5bb7df/html5/thumbnails/7.jpg)
Application size scaling in Hadoop 1.0
JTHeap!TotalTasks,Nodes, JobCounters
![Page 8: Hadoop YARN - Under the Hoodarchive.apachecon.com/eu2012/presentations/06... · Node Manager Container Container Node Manager App Mstr Container Node Manager Container App Mstr Node](https://reader030.fdocuments.net/reader030/viewer/2022040922/5e9bd0f45e900742fe5bb7df/html5/thumbnails/8.jpg)
Application size scaling in YARN is by
Architecture
![Page 9: Hadoop YARN - Under the Hoodarchive.apachecon.com/eu2012/presentations/06... · Node Manager Container Container Node Manager App Mstr Container Node Manager Container App Mstr Node](https://reader030.fdocuments.net/reader030/viewer/2022040922/5e9bd0f45e900742fe5bb7df/html5/thumbnails/9.jpg)
Why a limitation on cluster size ?
Cluster Utilization
Cluster Size
Hadoop 1.0
![Page 10: Hadoop YARN - Under the Hoodarchive.apachecon.com/eu2012/presentations/06... · Node Manager Container Container Node Manager App Mstr Container Node Manager Container App Mstr Node](https://reader030.fdocuments.net/reader030/viewer/2022040922/5e9bd0f45e900742fe5bb7df/html5/thumbnails/10.jpg)
JobTracker JIP TIP Scheduler
Heartbeat Request
Heartbeat Response
• Synchronous Heartbeat Processing
• JobTracker Global Lock
JT transaction rate limit: 200 heartbeats/sec
![Page 11: Hadoop YARN - Under the Hoodarchive.apachecon.com/eu2012/presentations/06... · Node Manager Container Container Node Manager App Mstr Container Node Manager Container App Mstr Node](https://reader030.fdocuments.net/reader030/viewer/2022040922/5e9bd0f45e900742fe5bb7df/html5/thumbnails/11.jpg)
Highly Concurrent Systems • scales much better (if done
right) • makes effective use of multi-
core hardware • managing eventual
consistency of states hard • need for a systemic framework
to manage this
![Page 12: Hadoop YARN - Under the Hoodarchive.apachecon.com/eu2012/presentations/06... · Node Manager Container Container Node Manager App Mstr Container Node Manager Container App Mstr Node](https://reader030.fdocuments.net/reader030/viewer/2022040922/5e9bd0f45e900742fe5bb7df/html5/thumbnails/12.jpg)
Event Queue
Component A
Component B
Component N
Event Model
Event Dispatcher
• Mutations only via events • Components only expose Read APIs • Use Re-entrant locks • Components follow clear lifecycle
![Page 13: Hadoop YARN - Under the Hoodarchive.apachecon.com/eu2012/presentations/06... · Node Manager Container Container Node Manager App Mstr Container Node Manager Container App Mstr Node](https://reader030.fdocuments.net/reader030/viewer/2022040922/5e9bd0f45e900742fe5bb7df/html5/thumbnails/13.jpg)
Heartbeat Listener Event Q
Heartbeat Request
Heartbeat Response
Asynchronous Heartbeat Handling
NodeManager Meta
Get commands
![Page 14: Hadoop YARN - Under the Hoodarchive.apachecon.com/eu2012/presentations/06... · Node Manager Container Container Node Manager App Mstr Container Node Manager Container App Mstr Node](https://reader030.fdocuments.net/reader030/viewer/2022040922/5e9bd0f45e900742fe5bb7df/html5/thumbnails/14.jpg)
YARN: Better utilization bigger cluster
Cluster Utilization
Cluster Size
YARN
Hadoop 1.0
![Page 15: Hadoop YARN - Under the Hoodarchive.apachecon.com/eu2012/presentations/06... · Node Manager Container Container Node Manager App Mstr Container Node Manager Container App Mstr Node](https://reader030.fdocuments.net/reader030/viewer/2022040922/5e9bd0f45e900742fe5bb7df/html5/thumbnails/15.jpg)
State Management
![Page 16: Hadoop YARN - Under the Hoodarchive.apachecon.com/eu2012/presentations/06... · Node Manager Container Container Node Manager App Mstr Container Node Manager Container App Mstr Node](https://reader030.fdocuments.net/reader030/viewer/2022040922/5e9bd0f45e900742fe5bb7df/html5/thumbnails/16.jpg)
![Page 17: Hadoop YARN - Under the Hoodarchive.apachecon.com/eu2012/presentations/06... · Node Manager Container Container Node Manager App Mstr Container Node Manager Container App Mstr Node](https://reader030.fdocuments.net/reader030/viewer/2022040922/5e9bd0f45e900742fe5bb7df/html5/thumbnails/17.jpg)
State management in JT Very Hard to Maintain Debugging even harder
![Page 18: Hadoop YARN - Under the Hoodarchive.apachecon.com/eu2012/presentations/06... · Node Manager Container Container Node Manager App Mstr Container Node Manager Container App Mstr Node](https://reader030.fdocuments.net/reader030/viewer/2022040922/5e9bd0f45e900742fe5bb7df/html5/thumbnails/18.jpg)
Complex State Management • Light weight State Machines Library • Declarative way of specifying the state
Transitions • Invalid transitions are handled automatically • Fits nicely with the event model • Debug-ability is drastically improved.
Lineage of object states can easily be determined
• Handy while recovering the state
![Page 19: Hadoop YARN - Under the Hoodarchive.apachecon.com/eu2012/presentations/06... · Node Manager Container Container Node Manager App Mstr Container Node Manager Container App Mstr Node](https://reader030.fdocuments.net/reader030/viewer/2022040922/5e9bd0f45e900742fe5bb7df/html5/thumbnails/19.jpg)
Declarative State Machine
![Page 20: Hadoop YARN - Under the Hoodarchive.apachecon.com/eu2012/presentations/06... · Node Manager Container Container Node Manager App Mstr Container Node Manager Container App Mstr Node](https://reader030.fdocuments.net/reader030/viewer/2022040922/5e9bd0f45e900742fe5bb7df/html5/thumbnails/20.jpg)
High Availability
![Page 21: Hadoop YARN - Under the Hoodarchive.apachecon.com/eu2012/presentations/06... · Node Manager Container Container Node Manager App Mstr Container Node Manager Container App Mstr Node](https://reader030.fdocuments.net/reader030/viewer/2022040922/5e9bd0f45e900742fe5bb7df/html5/thumbnails/21.jpg)
MR Application Master Recovery • Hadoop 1.0
• Application need to resubmit Job • All completed tasks are lost
• YARN • Application execution state check pointed in
HDFS • Rebuilds the state by replaying the events
![Page 22: Hadoop YARN - Under the Hoodarchive.apachecon.com/eu2012/presentations/06... · Node Manager Container Container Node Manager App Mstr Container Node Manager Container App Mstr Node](https://reader030.fdocuments.net/reader030/viewer/2022040922/5e9bd0f45e900742fe5bb7df/html5/thumbnails/22.jpg)
Resource Manager HA • Based on Zookeeper • Coming Soon
• YARN-128
![Page 23: Hadoop YARN - Under the Hoodarchive.apachecon.com/eu2012/presentations/06... · Node Manager Container Container Node Manager App Mstr Container Node Manager Container App Mstr Node](https://reader030.fdocuments.net/reader030/viewer/2022040922/5e9bd0f45e900742fe5bb7df/html5/thumbnails/23.jpg)
YARN: New Possibilities
• Open MPI - MR-2911 • Master-Worker – MR-3315 • Distributed Shell • Graph processing – Giraph-13 • BSP – HAMA-431 • CEP
• S4 – S4-25 • Storm -
https://github.com/nathanmarz/storm/issues/74 • Iterative processing - Spark
https://github.com/mesos/spark-yarn/
![Page 24: Hadoop YARN - Under the Hoodarchive.apachecon.com/eu2012/presentations/06... · Node Manager Container Container Node Manager App Mstr Container Node Manager Container App Mstr Node](https://reader030.fdocuments.net/reader030/viewer/2022040922/5e9bd0f45e900742fe5bb7df/html5/thumbnails/24.jpg)
YARN - a solid foundation to take Hadoop to next level
on
Scale, High Availability, Utilization And
Alternate Compute Paradigms
![Page 25: Hadoop YARN - Under the Hoodarchive.apachecon.com/eu2012/presentations/06... · Node Manager Container Container Node Manager App Mstr Container Node Manager Container App Mstr Node](https://reader030.fdocuments.net/reader030/viewer/2022040922/5e9bd0f45e900742fe5bb7df/html5/thumbnails/25.jpg)
Thank You
@twitter: sharad_ag