Benjamin Hindman – @benh

Apache Mesoshttp://incubator.apache.org/mesos@ApacheMesos

historyBerkeley research project including Benjamin Hindman, Andy Konwinski, Matei Zaharia, Ali Ghodsi, Anthony D. Joseph, Randy Katz, Scott Shenker, Ion Stoica

http://incubator.apache.org/mesos/research.html

Mesos aims to make it easier to build distributed applications/frameworks and share cluster resources

applications/frameworks

services analytics

analyticsservices

applications/frameworks

how?

MesosNode Node Nod

e Node

Hadoop service …

Node

Node

Hadoop

Node

Node

service…

level of abstractionmore easily share the resources via multi-tenancy and elasticity (improve utilization)run on bare-metal or virtual machines – develop against Mesos API, run in private datacenter (Twitter), or the cloud, or both!

Hadoop

Spark

service

shared cluster

static partitioning sharing with Mesos

featuresAPIs in C++, Java, Pythonhigh-availability via zookeeperisolation via linux control groups (LXC)

in progress»official apache release»more linux cgroup support (oom and i/o, in

particular, networking)»resource usage monitoring, reporting»new allocators (priority based, usage

based)»new frameworks (storm)»scheduler management (launching,

watching, re-launching, etc)

400+ nodes running production servicesgenomics researchers using Hadoop and SparkSpark in use by Yahoo! ResearchSpark for analyticsHadoop and Spark used by machine learning researchersYour Name

Here

demonstration

linux environment$ yum install -y gcc-c++$ yum install -y java-1.6.0-openjdk-devel.x86_64$ yum install -y make.x86_64$ yum install -y patch.x86_64$ yum install -y python26-devel.x86_64$ yum install -y ant.noarch

get mesos$ wget http://people.apache.org/~benh/mesos-0.9.0-incubating-RC3/mesos-0.9.0-incubating.tar.gz$ tar zxvf mesos-0.9.0-incubating.tar.gz$ cd mesos-0.9.0

build mesos$ mkdir build$ cd build$ ../configure.amazon-linux-64$ make$ make install

deploy mesos (1)/usr/local/var/mesos/deploy/masters:

ec2-50-17-28-135.compute-1.amazonaws.com

/usr/local/var/mesos/deploy/slaves:

ec2-184-73-142-43.compute-1.amazonaws.comec2-107-22-145-31.compute-1.amazonaws.com

deploy mesos (2)on slaves (i.e., ec2-184-73-142-43.compute-1.amazonaws.com, ec2-107-22-145-31.compute-1.amazonaws.com)

/usr/local/var/mesos/conf/mesos.conf:

master=ec2-50-17-28-135.compute-1.amazonaws.com

deploy mesos (3)$ /usr/local/sbin/mesos-start-cluster.sh

build hadoop$ make hadoop$ mv hadoop/hadoop-0.20.205.0 /etc/hadoop$ cp protobuf-2.4.1.jar /etc/hadoop$ cp src/mesos-0.9.0.jar /etc/hadoop

configure hadoop (1)conf/mapred-site.xml:

<configuration> <property> <name>mapred.job.tracker</name> <value>ip-10-108-207-105.ec2.internal:9001</value> </property> <property> <name>mapred.jobtracker.taskScheduler</name> <value>org.apache.hadoop.mapred.MesosScheduler</value> </property>

<property> <name>mapred.mesos.master</name> <value>ip-10-108-207-105.ec2.internal:5050</value> </property></configuration>

configure hadoop (2)conf/hadoop-env.sh:

#!/bin/sh

export JAVA_HOME=/usr/lib/jvm/jre

# Google protobuf (necessary for running the MesosScheduler). export PROTOBUF_JAR=${HADOOP_HOME}/protobuf-2.4.1.jar

# Mesos.export MESOS_JAR=${HADOOP_HOME}/mesos-0.9.0.jar

# Native Mesos library. export MESOS_NATIVE_LIBRARY=/usr/local/lib/libmesos.so

export HADOOP_CLASSPATH=${HADOOP_HOME}/build/contrib/mesos/classes:${MESOS_JAR}:${PROTOBUF_JAR}

...

configure hadoop (3)conf/core-site.sh:

<configuration> <property> <name>fs.default.name</name> <value>hdfs://ip-10-108-207-105.ec2.internal:9000</value> </property></configuration>

configure hadoop (4)conf/masters:

ec2-50-17-28-135.compute-1.amazonaws.com

conf/slaves:

ec2-184-73-142-43.compute-1.amazonaws.comec2-107-22-145-31.compute-1.amazonaws.com

configure hadoop (4)conf/masters:

ec2-50-17-28-135.compute-1.amazonaws.com

conf/slaves:

ec2-184-73-142-43.compute-1.amazonaws.comec2-107-22-145-31.compute-1.amazonaws.com

starting hadoop$ pwd/etc/hadoop$ ./bin/hadoop jobtracker

running wordcount$ ./bin/hadoop jar hadoop-examples-0.20.205.0.jar wordcount macbeth.txt output

starting another hadoop<configuration> <property>    <name>mapred.job.tracker</name>    <value>ip-10-108-207-105.ec2.internal:9002</value> </property>

<property>    <name>mapred.job.tracker.http.address</name>    <value>0.0.0.0:50032</value>  </property>

  <property>   <name>mapred.task.tracker.http.address</name>   <value>0.0.0.0:50062</value>  </property></configuration>

get and build spark$ git clone git://github.com/mesos/spark.git$ cd spark$ git checkout --track origin/mesos-0.9$ sbt/sbt compile

configure spark$ cp conf/spark-env.sh.template conf/spark-env.sh

conf/spark-env.sh:

#!/bin/sh

export SCALA_HOME=/root/scala-2.9.1-1export MESOS_NATIVE_LIBRARY=/usr/local/lib/libmesos.soexport SPARK_MEM=1g

run spark shell$ pwd/root/spark$ MASTER=$HOSTNAME:5050 ./spark-shell

setting log_diron slaves (i.e., ec2-184-73-142-43.compute-1.amazonaws.com, ec2-107-22-145-31.compute-1.amazonaws.com)

/usr/local/var/mesos/conf/mesos.conf:

master=ec2-50-17-28-135.compute-1.amazonaws.comlog_dir=/tmp/mesos

re-deploy mesos$ /usr/local/sbin/mesos-stop-slaves.sh

$ /usr/local/sbin/mesos-start-slaves.sh