Ironfan
• Open-source core (Apache) with enterprise features
• http://github.com/infochimps-labs/ironfan
• Based on Chef (http://www.opscode.com/chef)
Principles
• Progressive - match what you’d whiteboard
• Declaratory - show all (and only) what’s interesting
• Coherent - full state is versioned as a whole
• Decoupled - minimize shared knowledge
• Helpful - common things are simple
• Inevitable - declarations imply actions
• one-ish:• HBase master• HBase secondary master• Hadoop Namenode• Hadoop 2NN• Hadoop Jobtracker• Zookeeper Quorum x3
• many:• HBase RegionServer• HBase Stargate• HBase Thrift• Hadoop Datanode• Hadoop Tasktracker
HBase Cluster
• HBase
• ElasticSearch
• Cassandra
• TokyoTyrant
• SimpleDB
• MongoDB
• MySQL
• Redis
• sqlite
• whisper (graphite)
• file system
• S3
Data Stores in Production
• Emacs
• Keynote
• Preview
• Chrome
• ruby (pry)
• Skitch
• finder
• flickr.com
• google image search
• ssh
Programs Used for This Talk
gibbon-nnjt-0
gibbon-worker-0
gibbon-hadoop-namenodegibbon-hadoop-jobtracker
¿dónde está mi namenode?
gibbon-nnjt-0
ack ack ack
gibbon-nnjt-0
gibbon-worker-0
¿dónde está mi namenode?
gibbon-nnjt-0
¿dónde está mi jobtracker?
gibbon-nnjt-0
gibbon-hadoop-namenodegibbon-hadoop-jobtracker
ack ack ack
gibbon-nn-0
gibbon-worker-0
¿dónde está mi namenode?
gibbon-nn-0
¿dónde está mi jobtracker?
gibbon-jt-0
ack ack ack
gibbon-jt-0
gibbon-hadoop-namenode
gibbon-hadoop-jobtracker
Amenities
• Logs: rotation, archiving
• Ports: remote monitoring (on and off)
• Daemons: process monitoring
• Dashboards: Dashboard dashboard
• Libs / Executables: Decoupled imports
• Versions: Shims, Discovery
hard coded
fails on bsd
not DRY
SECURITY HOLEdaemon-writeable
conf dir
10 lines, 5 flaws
‘0755’ not 755
Inevitabilitymust do this:
to get a log dir
gibbon-hadoop-namenode
so when you announce,
AWESOMENESS IS INEVITABLE
• Region Server (dominate)• TaskTracker + Children (wussy)• DataNode (moderate)
• DataNode (moderate)• TaskTracker + Children (dominate)
HBase
Hadoop
We’re Hiring
github.com/ infochimps-labs
• System Diagram made real• Decoupled Components connect• Components Announce their capabilities• Announcements enable Service Discovery• Components announce cross-cutting Aspects• Aspects enable zero-conf Amenities• Announcement as Contract• Contracts enable zero-conf specification testing• Specs + monitoring enable zero-conf integration
testing• Systems Bind to provisioned resources• Resource Sharing
Top Related