Apache Bigtop: a crash course in deploying a Hadoop platformApache Bigtop: a crash course in deploying a Hadoop platform

“3 + 7 + 9 + 1”The complexity of the stack

“3 + 7 + 9 + 1”The complexity of the stack

Hadoop ecosystem• Many components• Gazillions of versions• Lot of patches if you like it hot and dirty

Presenters• Dr. Konstantin Boudnik, Roman Shaposhnik• Initial co-inventors of Apache Bigtop• Active contributors, committers, and PMCs on

multiple Apache TLPs in Hadoop ecosystem• Expertise in

• Compilers• Operating systems, Virtual machines, JVM• Distributed systems• Complex software stacks

Hadoop ≠ HDFS + MR• Today's Hadoop is more than storage + MR• Baby elephant outgrew his cradle

• Hbase• SQL frontends• In-memory processing• Storage caching• Connectors• DSL languages• <your name is here>

Complexity in the extreme

“One to bring them all”Bigtop is a simple answer “One to bring them all”

Bigtop is a simple answer

Bigtop stack: take it & go• Modify a stack BOM

• Build• Deploy• Configure w/ Puppet (included)• Test (scenarios are provided / easy to add)• Grab an appliance if short of hardware

Rinse and repeat

Field case studies:Pivotal

WANdisco

Field case studies:Pivotal

WANdisco

From 0 to full stack in 28 days:WANdisco case study

From 0 to full stack in 28 days:WANdisco case study

New distro in 4 weeks• Major challenges for a new player:

• Need to have a stable development platform

• Offering Apache Hadoop certified binaries• Team with no prior expertise in the field• Very complex Hadoop ecosystem

landscape

Bigtop to help• Define component versions in the BOM• Make component changes if needed

• Run Bigtop build• Deploy the cluster w/ provided puppet• Test the cluster with integration suite• Rinse and repeat as needed

• Seamless integration into CI• Easy provisioning and incremental updates

Hadoop in the cloudPivotal case study

Hadoop in the cloudPivotal case study

Hadoop @Pivotal: history• Started at Greenplum (GPHD)

• A stand-alone Hadoop distribution• Based on a fork of Bigtop 0.3-incubating• Hadoop 1.0.1 based ecosystem• No community interaction

• Graduated as a PHD at Pivotal• A PivotalONE vision• Based on a fork of Bigtop 0.4• Hadoop 2.x based ecosystem• Integrated with HAWQ

Lessons for Pivotal• Think of Bigtop as “Fedora”• Become part of the community • Say “no” to forking• Work on custom requirements upstream• Participate in release planning• Leverage Bigtop's infrastructure internally

Lessons for Bigtop• Promote common build and RE infrastructure

• “Codifying” it• Avoid broken windows syndrome

• Bigtop releases don't patch, but vendors do• Make tests easier to use• Invest in documentation

• Whitepapers, demos, etc.

Road aheadRoad ahead

Packaging/deployment• Deployment environments:

• vanilla servers == packages ?• vanilla Vms/containers == JeOS/baking ?• specializes VMs == Osv

• Baking vs frying• Rolling upgrades

• side-by-side install

Validation• Investing in iTest

• Easier test management and execution• Cluster discovery and management

framework• More test cases

• Lowering entry-level barriers• Less rigid user-facing interfaces: Gradle• Mix-n-match built-in integration tests

• Becoming a TCK for Hadoop ecosystem• Engaging more into trunk testing

Growing the ecosystem• 1st Hadoop distribution including Apache

Spark• But there's more:

• GridGain• HBase indexer• Lipstick (not for Pig)• Ambrose• Launch it all to the Stratosphere?

Growth in last 6 months• Apache Spark: In-memory analytics• Phoenix: Hbase SQL frontend• Groovy• Unification of user-facing interfaces

• Gradle build system• Received proposal in include GridGain

in-memory platform

Community• 23 committers• Over 100 contributors• Quarterly releases• Hackathons and more

Demo (worth 1k words)

https://bigtop.apache.org/

https://blogs.apache.org/bigtop

https://cwiki.apache.org/confluence/display/BIGTOP

dev@bigtop.apache.org

user@bigtop.apache.org

@ASFbigtop

Come & join us

Q & AQ & A

Thank you@c0sin@rhatr

Thank you@c0sin@rhatr