Mesos by zigi

download Mesos by zigi

of 34

  • date post

    14-Apr-2017
  • Category

    Technology

  • view

    130
  • download

    5

Embed Size (px)

Transcript of Mesos by zigi

  • 1KrDAG ZIGI zigispace.netKrDAG ZIGI zigispace.net

  • 2KrDAG ZIGI zigispace.netKrDAG ZIGI zigispace.net

    Introduce

    Mesos

    Apache Mesos abstracts CPU, memory, storage, and other compute resources away from machines (physical or virtual), enabling fault-tolerant and elastic distributed systems to easily be built and run effectively.

    Mesos is built using the same principles as the Linux kernel, only at a different level of abstraction. The Mesos kernel runs on every machine and provides applications with APIs for resource management and scheduling across entire datacenter and cloud environments.

  • 3KrDAG ZIGI zigispace.netKrDAG ZIGI zigispace.net

    Mesos Machine( Node) (CPU, Memory, Storage ) .

    Cloud Infrastructure Computing Engine .

  • 4KrDAG ZIGI zigispace.netKrDAG ZIGI zigispace.net

    UC Berkeley Nexus .

    Mesos Apache .

    Mesos

    MesosHistory &Reference

  • 5KrDAG ZIGI zigispace.netKrDAG ZIGI zigispace.net

    Scalability to 10,000s of nodes

    Fault-tolerant replicated master and slaves using ZooKeeper

    Support for Docker containers

    Native isolation between tasks with Linux Containers

    Multi-resource scheduling (memory, CPU, disk, and ports)

    Java, Python and C++ APIs for developing new parallel applications

    Web UI for viewing cluster state

    MesosFeatures

  • 6KrDAG ZIGI zigispace.netKrDAG ZIGI zigispace.net

    Utility ,

    View .

    OS Lock-In H/W OS .

    MesosWhy?

  • 7KrDAG ZIGI zigispace.netKrDAG ZIGI zigispace.net

    Architecture : Master

    Mesos

    Slave Framework

    Master

    - Active Master 1

    - Master Active / ZooKeeper

    - Active Master Standby

    Master Framework Slave Resource

    .

  • 8KrDAG ZIGI zigispace.netKrDAG ZIGI zigispace.net

    Architecture : Slave

    Mesos

    Mesos (Task)

    .

  • 9KrDAG ZIGI zigispace.netKrDAG ZIGI zigispace.net

    Architecture : Frameworks

    Mesos

    Executor .

    scheduler

    - Master .

    executor

    - Task

  • 10KrDAG ZIGI zigispace.netKrDAG ZIGI zigispace.net

    Architecture

    Mesos

  • 11KrDAG ZIGI zigispace.netKrDAG ZIGI zigispace.net

    Aurora is a service scheduler that runs on top of Mesos, enabling you to run long-running services that take advantage of Mesos' scalability, fault-tolerance, and resource isolation.

    Marathon is a private PaaS built on Mesos. It automatically handles hardware or software failures and ensures that an app is always on.

    Singularity is a scheduler (HTTP API and web interface) for running Mesos tasks: long running processes, one-off tasks, and scheduled jobs.

    SSSP is a simple web application that provides a white-label Megaupload for storing and sharing files in S3.

    MesosFramework : Long Running Services

  • 12KrDAG ZIGI zigispace.netKrDAG ZIGI zigispace.net

    Cray Chapel is a productive parallel programming language. The Chapel Mesos scheduler lets you run Chapel programs on Mesos.

    Dpark is a Python clone of Spark, a MapReduce-like framework written in Python, running on Mesos.

    Exelixi is a distributed framework for running genetic algorithms at scale.

    Hadoop Running Hadoop on Mesos distributes MapReduce jobs efficiently across an entire cluster.

    Hama is a distributed computing framework based on Bulk Synchronous Parallel computing techniques for massive scientific computations e.g., matrix, graph and network algorithms.

    MPI is a message-passing system designed to function on a wide variety of parallel computers.

    Spark is a fast and general-purpose cluster computing system which makes parallel jobs easy to write.

    Storm is a distributed realtime computation system. Storm makes it easy to reliably process unbounded streams of data, doing for realtime processing what Hadoop did for batch processing.

    MesosFramework : Big Data Processing

  • 13KrDAG ZIGI zigispace.netKrDAG ZIGI zigispace.net

    Chronos is a distributed job scheduler that supports complex job topologies. It can be used as a more fault-tolerant replacement for Cron.

    Jenkins is a continuous integration server. The mesos-jenkins plugin allows it to dynamically launch workers on a Mesos cluster depending on the workload.

    JobServer is a distributed job scheduler and processor which allows developers to build custom batch processing Tasklets using point and click web UI.

    MesosFramework : Batch Scheduling

  • 14KrDAG ZIGI zigispace.netKrDAG ZIGI zigispace.net

    Cassandra is a performant and highly available distributed database. Linear scalability and proven fault-tolerance on commodity hardware or cloud infrastructure make it the perfect platform for mission-critical data.

    ElasticSearch is a distributed search engine. Mesos makes it easy to run and scale.

    Hypertable is a high performance, scalable, distributed storage and processing system for structured and unstructured data.

    MesosFramework : Data Storage

  • 15KrDAG ZIGI zigispace.netKrDAG ZIGI zigispace.net

    High availability

    Mesos

    Mesos is configured for high availability, the leading master will abort itself when it discovers it has been partitioned away from the Zookeeper quorum. This is a safety precaution to ensure the previous leader doesnt continue communicating in an unsafe state.

  • 16KrDAG ZIGI zigispace.netKrDAG ZIGI zigispace.net

    Master Quorum 2N-1 , N Master .

    Master Quorum 3 5

    MesosMaster Quorum

    Master Quorum Size Failure Tolerance

    1 1 0

    3 2 1

    5 3 2

    2N-1 N N-1

  • 17KrDAG ZIGI zigispace.netKrDAG ZIGI zigispace.net

    Mesos

  • 18KrDAG ZIGI zigispace.netKrDAG ZIGI zigispace.net

    Flow

    Mesos

    Step 1. Slave Master .

    Slave Master

    Resource register

  • 19KrDAG ZIGI zigispace.netKrDAG ZIGI zigispace.net

    Flow

    Mesos

    Step 2. Master Slave Framework .

    Master Framework

    Resource Inform

  • 20KrDAG ZIGI zigispace.netKrDAG ZIGI zigispace.net

    Flow

    Mesos

    Step 3. Framework Job 1 .

    Admin Framework

    Task (Job)

  • 21KrDAG ZIGI zigispace.netKrDAG ZIGI zigispace.net

    Flow

    Mesos

    Step 4. Master Job 1 Slave .

    MasterFramework Slave

  • 22KrDAG ZIGI zigispace.netKrDAG ZIGI zigispace.net

    Flow

    Mesos

    Step 5. Slave

    Running

    Slave

  • 23KrDAG ZIGI zigispace.netKrDAG ZIGI zigispace.net

    Example of resource offer

    Mesos

  • 24KrDAG ZIGI zigispace.netKrDAG ZIGI zigispace.net

    Datacenter OS

  • 25KrDAG ZIGI zigispace.netKrDAG ZIGI zigispace.net

    Datacenter operating system - DCOS

    The DCOS runs in any modern Linux environment: public and private cloud, virtual machines and bare metal.

    Built in to the DCOS, Mesos pools your infrastructure, automatically allocating resources and scheduling tasks based on demands and policy.

    Control your entire DCOS-powered datacenter from a single command line and a single web user interface.

    Run all your apps and workloads, from PaaS-based microservices to big data to databases and everything in between.

  • 26KrDAG ZIGI zigispace.netKrDAG ZIGI zigispace.net

    DATACENTER OPERATING SYSTEM - DCOS

  • 27KrDAG ZIGI zigispace.netKrDAG ZIGI zigispace.net

    Mesos

  • 28KrDAG ZIGI zigispace.netKrDAG ZIGI zigispace.net

    Dashboard

    Mesos

  • 29KrDAG ZIGI zigispace.netKrDAG ZIGI zigispace.net

    Dashboard : Frameworks

    Mesos

  • 30KrDAG ZIGI zigispace.netKrDAG ZIGI zigispace.net

    Dashboard : Slaves

    Mesos

  • 31KrDAG ZIGI zigispace.netKrDAG ZIGI zigispace.net

    Dashboard : Application

    Marathon

  • 32KrDAG ZIGI zigispace.netKrDAG ZIGI zigispace.net

    Dashboard : Application Detail

    Marathon

  • 33KrDAG ZIGI zigispace.netKrDAG ZIGI zigispace.net

    Etc.

    Mesos

    Pushing control to the frameworks has two benefits. First, it allows frameworks to implement diverse approaches to various problems in the cluster (e.g., achieving data locality, dealing with faults), and to evolve these solutions independently. Second, it keeps Mesos simple and minimizes the rate of change required of the system, which makes it easier to keep Mesos scalable and robust.

  • 34KrDAG ZIGI zigispace.netKrDAG ZIGI zigispace.net

    End.