Hadoop @ eBay: Past, Present, and Future

39
Hadoop @ eBay: Past, Present and Future Ryan Hennig Hadoop Platform Team

description

An overview of eBay's experience with Hadoop in the Past and Present, as well as directions for the Future. Given by Ryan Hennig at the Big Data Meetup at eBay in Netanya, Israel on Dec 2, 2013

Transcript of Hadoop @ eBay: Past, Present, and Future

Page 1: Hadoop @ eBay: Past, Present, and Future

Hadoop @ eBay: Past, Present and Future

Ryan Hennig

Hadoop Platform Team

Page 2: Hadoop @ eBay: Past, Present, and Future

ABOUT ME

Page 3: Hadoop @ eBay: Past, Present, and Future

COMPUTE AND DATA INFRASTRUCTURE 3

RYAN HENNIG

Born and raised in Seattle, WA

Studied Computer Science at University of Washington in Seattle

Worked on Microsoft SQL Server 2006 – 2012- Shipped SQL Server 2008, 2008 R2, 2012

Joined eBay Hadoop team in early 2012

- Based in Bellevue, suburb of Seattle

Page 4: Hadoop @ eBay: Past, Present, and Future

AGENDA

Past: Growth of Hadoop at eBay

Present: Hadoop Use Cases, Operations Tools

Future: Hadoop 2.0

Page 5: Hadoop @ eBay: Past, Present, and Future

HADOOP AT EBAY: PASTGrowth of Hadoop at eBay

Adventures in Forking

Partnership with Hortonworks

Page 6: Hadoop @ eBay: Past, Present, and Future

HADOOP AT EBAY: PAST 6

HADOOP EVOLUTION @ eBay

2007

Single digit nodes

2010Shared cluster• 100s nodes• 1000s +

core• PB• CDH2

2011• Shared

clusters• 1000s node• 10,000+ core• 10s PB• Wilma (0.20)

2012• Shared

clusters• 1000s node• 10,000+ core• 10s PB• Argon (0.22)

2013• Shared

clusters• 4k+ node• 40,000+ core• 50s PB• HDP 1.x

2009Search• 10s-

nodes

Page 7: Hadoop @ eBay: Past, Present, and Future

HADOOP AT EBAY: PAST 7

ADVENTURES IN FORKING

• 2007-2010: eBay runs shared clusters on Cloudera Distribution of Hadoop

• 2010-2012: eBay runs shared clusters on custom Hadoop versions– 2010: Wilma (based on 0.20)– 2011: Argon (based on 0.22)– 2012: Custom branch abandoned

• Lessons Learned– Forking a fast-changing open source project is difficult and risky

• Balancing Development and operations needs• Development team size

– Facebook had 100– eBay had 15

• Coordination with open source community = lots of overhead• Divergence from open source: Push changes early and often

Page 8: Hadoop @ eBay: Past, Present, and Future

HADOOP AT EBAY: PAST 8

Page 9: Hadoop @ eBay: Past, Present, and Future

HADOOP AT EBAY: PAST 9

EBAY AND HORTONWORKS

• 2012: eBay enters partnership with HortonWorks– Goals

• Focus on eBay-specific development internally• Leverage HortonWorks expertise for general Hadoop Development• Avoid source code divergence by making open source contribution a priority

– Benefits to HortonWorks• Credibility enhanced by having a well-known customer• Ability to test at large scale

Page 10: Hadoop @ eBay: Past, Present, and Future

HADOOP AT EBAY:PRESENTShared and Dedicated Clusters

Job Distribution

Use Case Examples

eBay Data Platform Overview

Page 11: Hadoop @ eBay: Past, Present, and Future

11

SHARED AND DEDICATED CLUSTERS

Shared clusters– 10s of PB and 10s of thousands of slots per cluster– Used primarily for analytics of user behavior and inventory– Mix of production and ad-hoc jobs– Mix of MR, Hive, PIG, Cascading etc.– Hadoop and HBase security enabled

Dedicated clusters– Very specific use cases like Index Building– Tight SLAs for jobs (in order of minutes)– Immediate revenue impact– Usually smaller than our shared clusters, but still big (100s of nodes…)

HADOOP AT EBAY: PRESENT

Page 12: Hadoop @ eBay: Past, Present, and Future

HADOOP AT EBAY: PRESENT 12

JOB DISTRIBUTION BY TYPE

Page 13: Hadoop @ eBay: Past, Present, and Future

HADOOP AT EBAY: PRESENT 13

USE CASE EXAMPLES

•Cassini, eBay’s new search engine:– Use MR to build full and incremental near-real-time indexes– Raw Data is stored in HBase for efficient updates and random read– Strong SLAs: < 10 minutes– Run on dedicated clusters

•Related and similar Items recommendations:– Use transactional data, click stream data, search index, etc.– Production MR jobs on a shared cluster

•Analytics dashboard:– Run Mobius MR jobs to join click stream data and transactional data – Store summary data in HBase– Web application to query HBase

Page 14: Hadoop @ eBay: Past, Present, and Future

HADOOP AT EBAY: PRESENT 14

HADOOP OPERATIONS

LDAP Integration

- All users stored in Active Directory, accessed via LDAP

- Access to MapReduce Queues granted via MapReduce queues

- Batch users: shared by a group of users

Security

- Kerberos as implemented by Microsoft Active Directory

- One domain for users, another for service/server principals

- Batch users authenticated via keytabs, not passwords

Misc

- 10’s of slave nodes are broken at any given time

- Often need to add several racks of machines at a time

Page 15: Hadoop @ eBay: Past, Present, and Future

HADOOP AT EBAY: PRESENT 15

HADOOP OPERATIONS

Team has Development and Operations Responsibilities- 2 Huge shared clusters- 1800+ users, exponential growth- About 10 Hadoop developers- Recently: operations work moved to dedicated team

Developed several tools to manage operations- Hadoop Management Console: user-facing web app- ldap-admin: swiss-army knife style tool for hadoop admins- Puppet: for adding machines to the clusters, many racks at a time- Decom/Recom scripts: automatic detection, repair, decommission, and

recommission of slave nodes

Page 16: Hadoop @ eBay: Past, Present, and Future

HADOOP AT EBAY: PRESENT 16

HADOOP MANAGEMENT CONSOLE

• Custom Web application built on Ruby on Rails• Self-service tools are continually added to reduce support load

– User Management• Access Requests• Group Membership

– Batch User Management• New Requests• Sudoer management

– Dataset Management• Explore Datasets• Request New dataset transfer between Teradata and Hadoop

– Metadata tools• Each dataset is stored in custom XML format• Code Generation: Hive Tables, Java POJOs

Page 17: Hadoop @ eBay: Past, Present, and Future

HADOOP AT EBAY: PRESENT 17

Page 18: Hadoop @ eBay: Past, Present, and Future

HADOOP AT EBAY: PRESENT 18

Page 19: Hadoop @ eBay: Past, Present, and Future

HADOOP AT EBAY: PRESENT 19

Page 20: Hadoop @ eBay: Past, Present, and Future

HADOOP AT EBAY: PRESENT 20

Page 21: Hadoop @ eBay: Past, Present, and Future

HADOOP AT EBAY: PRESENT 21

Page 22: Hadoop @ eBay: Past, Present, and Future

HADOOP AT EBAY: PRESENT 22

Page 23: Hadoop @ eBay: Past, Present, and Future

HADOOP AT EBAY: PRESENT 23

Page 24: Hadoop @ eBay: Past, Present, and Future

HADOOP AT EBAY: PRESENT 24

ldap-admin

•Command-line tool written in Ruby•Swiss-army knife tool, features added on demand for support issues•Often used features:

–Add a user to a group–View key details for LDAP users and groups–List all users, batch users, hadoop groups–Reset batch user passwords and keytabs–Show/add/remove sudoers for a batch account–Run user diagnostics: check permissions, keytabs, etc

Page 25: Hadoop @ eBay: Past, Present, and Future

HADOOP AT EBAY:FUTUREHDFS Federation

YARN

New Scenarios

Storage and Operational Efficiency

Page 26: Hadoop @ eBay: Past, Present, and Future

26

HDFS HA and Federation

• HDFS High-Availability for Reliability– NameNode in Hadoop 1.0 is a Single Point of Failure– Automated failover to hot standby – Depends on ZooKeeper

• HDFS Federation for Scalability and Isolation– Hadoop 1.0: Single NameNode service

• “Secondary NameNode” is not for failover• Storage scales horizontally, but Namespace scales vertically• No isolation for different tenants or applications

– Hadoop 2.0: HDFS Federation• Partition the HDFS Namespace • Many independent NameNodes• Allows direct access to Block Storage w/o going through HDFS interface

HADOOP AT EBAY: FUTURE

Page 27: Hadoop @ eBay: Past, Present, and Future

HADOOP AT EBAY: FUTURE 27

HDFS HA

Page 28: Hadoop @ eBay: Past, Present, and Future

HADOOP AT EBAY: FUTURE 28

HDFS HA

Page 29: Hadoop @ eBay: Past, Present, and Future

HADOOP AT EBAY: FUTURE 29

HDFS HA

Page 30: Hadoop @ eBay: Past, Present, and Future

HADOOP AT EBAY: FUTURE 30

HDFS Federation

Horizontal Scalability of HDFS Namespace

Multiple independent NameNodes serving a subtree of the NameSpace

Example: NN1 provides /users, NN2 provides /reports

Page 31: Hadoop @ eBay: Past, Present, and Future

HADOOP AT EBAY: FUTURE 31

YARN

Hadoop 1.0: MapReduce– JobTracker and TaskTracker services– Handles Resource Management, Job Execution

Hadoop 2.0: YARN- Refactoring Responsiblities of JobTracker and TaskTracker into more general

platform- Global ResourceManager

- Cluster-wide resource managements- Per-application ApplicationMaster

- Application-specific job control

Page 32: Hadoop @ eBay: Past, Present, and Future

HADOOP AT EBAY: FUTURE 32

YARN

Page 33: Hadoop @ eBay: Past, Present, and Future

HADOOP AT EBAY: FUTURE 33

YARN

Page 34: Hadoop @ eBay: Past, Present, and Future

HADOOP AT EBAY: FUTURE 34

YARN

Page 35: Hadoop @ eBay: Past, Present, and Future

HADOOP AT EBAY: FUTURE 35

YARN

Page 36: Hadoop @ eBay: Past, Present, and Future

HADOOP AT EBAY: FUTURE 36

New Scenarios

• Iterative Query– Stinger (Hive), Impala, etc– Rapid Data exploration and analysis

• Graph Databases– TitanDB, Giraph – Billions of vertices and edges– Complex Graph Traversals– Applications: PayPal fraud detection, Social Graph Analysis

• Real-Time Processing– Storm (Twitter), Apache S4– Reinforcement Learning, Monitoring

Page 37: Hadoop @ eBay: Past, Present, and Future

HADOOP AT EBAY: FUTURE 37

Efficiency and Reliability

• Storage Efficiency– HDFS introduces a 3x storage cost for its replicas– HDFS-RAID: more reliability for 1.5x storage cost

• Reed-Solomon• Locally Repairable Codes (Project Xorbas)

– Tradeoff: the cost of repairing lost data is much higher• Operational Efficiency

– More automation– More self-service tools– Better Monitoring

Page 38: Hadoop @ eBay: Past, Present, and Future

HADOOP AT EBAY: FUTURE 38

Open Source

• HMC Metadata– Long term goal: standardize on open source technologies (HCatalog)– Short term: explore what should be open sourced

• Hadoop Management Console– Hadoop Access Request Automation– Batch user creation and management– Metadata management– Code generation of dataset to Hive tables and Java POJOs

• ldap_admin tools– Very useful but tightly coupled to eBay’s LDAP configuration– Willing to open source if there is interest

Page 39: Hadoop @ eBay: Past, Present, and Future

THANK YOU

Questions?