Mesos by zigi

download Mesos by zigi

of 34

  • date post

  • Category


  • view

  • download


Embed Size (px)

Transcript of Mesos by zigi

  • 1KrDAG ZIGI zigispace.netKrDAG ZIGI

  • 2KrDAG ZIGI zigispace.netKrDAG ZIGI



    Apache Mesos abstracts CPU, memory, storage, and other compute resources away from machines (physical or virtual), enabling fault-tolerant and elastic distributed systems to easily be built and run effectively.

    Mesos is built using the same principles as the Linux kernel, only at a different level of abstraction. The Mesos kernel runs on every machine and provides applications with APIs for resource management and scheduling across entire datacenter and cloud environments.

  • 3KrDAG ZIGI zigispace.netKrDAG ZIGI

    Mesos Machine( Node) (CPU, Memory, Storage ) .

    Cloud Infrastructure Computing Engine .

  • 4KrDAG ZIGI zigispace.netKrDAG ZIGI

    UC Berkeley Nexus .

    Mesos Apache .


    MesosHistory &Reference

  • 5KrDAG ZIGI zigispace.netKrDAG ZIGI

    Scalability to 10,000s of nodes

    Fault-tolerant replicated master and slaves using ZooKeeper

    Support for Docker containers

    Native isolation between tasks with Linux Containers

    Multi-resource scheduling (memory, CPU, disk, and ports)

    Java, Python and C++ APIs for developing new parallel applications

    Web UI for viewing cluster state


  • 6KrDAG ZIGI zigispace.netKrDAG ZIGI

    Utility ,

    View .

    OS Lock-In H/W OS .


  • 7KrDAG ZIGI zigispace.netKrDAG ZIGI

    Architecture : Master


    Slave Framework


    - Active Master 1

    - Master Active / ZooKeeper

    - Active Master Standby

    Master Framework Slave Resource


  • 8KrDAG ZIGI zigispace.netKrDAG ZIGI

    Architecture : Slave


    Mesos (Task)


  • 9KrDAG ZIGI zigispace.netKrDAG ZIGI

    Architecture : Frameworks


    Executor .


    - Master .


    - Task

  • 10KrDAG ZIGI zigispace.netKrDAG ZIGI



  • 11KrDAG ZIGI zigispace.netKrDAG ZIGI

    Aurora is a service scheduler that runs on top of Mesos, enabling you to run long-running services that take advantage of Mesos' scalability, fault-tolerance, and resource isolation.

    Marathon is a private PaaS built on Mesos. It automatically handles hardware or software failures and ensures that an app is always on.

    Singularity is a scheduler (HTTP API and web interface) for running Mesos tasks: long running processes, one-off tasks, and scheduled jobs.

    SSSP is a simple web application that provides a white-label Megaupload for storing and sharing files in S3.

    MesosFramework : Long Running Services

  • 12KrDAG ZIGI zigispace.netKrDAG ZIGI

    Cray Chapel is a productive parallel programming language. The Chapel Mesos scheduler lets you run Chapel programs on Mesos.

    Dpark is a Python clone of Spark, a MapReduce-like framework written in Python, running on Mesos.

    Exelixi is a distributed framework for running genetic algorithms at scale.

    Hadoop Running Hadoop on Mesos distributes MapReduce jobs efficiently across an entire cluster.

    Hama is a distributed computing framework based on Bulk Synchronous Parallel computing techniques for massive scientific computations e.g., matrix, graph and network algorithms.

    MPI is a message-passing system designed to function on a wide variety of parallel computers.

    Spark is a fast and general-purpose cluster computing system which makes parallel jobs easy to write.

    Storm is a distributed realtime computation system. Storm makes it easy to reliably process unbounded streams of data, doing for realtime processing what Hadoop did for batch processing.

    MesosFramework : Big Data Processing

  • 13KrDAG ZIGI zigispace.netKrDAG ZIGI

    Chronos is a distributed job scheduler that supports complex job topologies. It can be used as a more fault-tolerant replacement for Cron.

    Jenkins is a continuous integration server. The mesos-jenkins plugin allows it to dynamically launch workers on a Mesos cluster depending on the workload.

    JobServer is a distributed job scheduler and processor which allows developers to build custom batch processing Tasklets using point and click web UI.

    MesosFramework : Batch Scheduling

  • 14KrDAG ZIGI zigispace.netKrDAG ZIGI

    Cassandra is a performant and highly available distributed database. Linear scalability and proven fault-tolerance on commodity hardware or cloud infrastructure make it the perfect platform for mission-critical data.

    ElasticSearch is a distributed search engine. Mesos makes it easy to run and scale.

    Hypertable is a high performance, scalable, distributed storage and processing system for structured and unstructured data.

    MesosFramework : Data Storage

  • 15KrDAG ZIGI zigispace.netKrDAG ZIGI

    High availability


    Mesos is configured for high availability, the leading master will abort itself when it discovers it has been partitioned away from the Zookeeper quorum. This is a safety precaution to ensure the previous leader doesnt continue communicating in an unsafe state.

  • 16KrDAG ZIGI zigispace.netKrDAG ZIGI

    Master Quorum 2N-1 , N Master .

    Master Quorum 3 5

    MesosMaster Quorum

    Master Quorum Size Failure Tolerance

    1 1 0

    3 2 1

    5 3 2

    2N-1 N N-1

  • 17KrDAG ZIGI zigispace.netKrDAG ZIGI


  • 18KrDAG ZIGI zigispace.netKrDAG ZIGI



    Step 1. Slave Master .

    Slave Master

    Resource register

  • 19KrDAG ZIGI zigispace.netKrDAG ZIGI



    Step 2. Master Slave Framework .

    Master Framework

    Resource Inform

  • 20KrDAG ZIGI zigispace.netKrDAG ZIGI



    Step 3. Framework Job 1 .

    Admin Framework

    Task (Job)

  • 21KrDAG ZIGI zigispace.netKrDAG ZIGI



    Step 4. Master Job 1 Slave .

    MasterFramework Slave

  • 22KrDAG ZIGI zigispace.netKrDAG ZIGI



    Step 5. Slave



  • 23KrDAG ZIGI zigispace.netKrDAG ZIGI

    Example of resource offer


  • 24KrDAG ZIGI zigispace.netKrDAG ZIGI

    Datacenter OS

  • 25KrDAG ZIGI zigispace.netKrDAG ZIGI

    Datacenter operating system - DCOS

    The DCOS runs in any modern Linux environment: public and private cloud, virtual machines and bare metal.

    Built in to the DCOS, Mesos pools your infrastructure, automatically allocating resources and scheduling tasks based on demands and policy.

    Control your entire DCOS-powered datacenter from a single command line and a single web user interface.

    Run all your apps and workloads, from PaaS-based microservices to big data to databases and everything in between.

  • 26KrDAG ZIGI zigispace.netKrDAG ZIGI


  • 27KrDAG ZIGI zigispace.netKrDAG ZIGI


  • 28KrDAG ZIGI zigispace.netKrDAG ZIGI



  • 29KrDAG ZIGI zigispace.netKrDAG ZIGI

    Dashboard : Frameworks


  • 30KrDAG ZIGI zigispace.netKrDAG ZIGI

    Dashboard : Slaves


  • 31KrDAG ZIGI zigispace.netKrDAG ZIGI

    Dashboard : Application


  • 32KrDAG ZIGI zigispace.netKrDAG ZIGI

    Dashboard : Application Detail


  • 33KrDAG ZIGI zigispace.netKrDAG ZIGI



    Pushing control to the frameworks has two benefits. First, it allows frameworks to implement diverse approaches to various problems in the cluster (e.g., achieving data locality, dealing with faults), and to evolve these solutions independently. Second, it keeps Mesos simple and minimizes the rate of change required of the system, which makes it easier to keep Mesos scalable and robust.

  • 34KrDAG ZIGI zigispace.netKrDAG ZIGI