Mesos Gets Pluggable - Introducing Mesos Modules

download Mesos Gets Pluggable - Introducing Mesos Modules

of 31

  • date post

  • Category


  • view

  • download


Embed Size (px)

Transcript of Mesos Gets Pluggable - Introducing Mesos Modules

  • Kapil Arya & Niklas Nielsen

    Mesos Gets Pluggable Introducing Mesos Modules

  • 2015 Mesosphere, Inc. 2



  • 2015 Mesosphere, Inc. 3

    Mesos Modules & HooksModules & HooksArwwwww

  • 2015 Mesosphere, Inc. 4

    How and why modules was introduced in Mesos

    Our humble thoughts on how modules and extensibility in Mesos can evolve in the future

    How Mesos Modules work and give you concrete examples of modules in action

  • 2015 Mesosphere, Inc.



  • 2015 Mesosphere, Inc.

    Different organizationsDifferent needs


    Hardly anyone run clusters the same way Different scales Different hardware Different workloads Different external tooling Different security needs

    One cluster with turbo chargers please

  • 2015 Mesosphere, Inc.

    Mesos was built with this in mind! The subsystems are lightweight insight

    and control over HTTP

    Excellent for tooling around

    Different subsystems can be enabled and configured in a modular way

    Most notable: Isolation mechanisms

    Good news!


  • 2015 Mesosphere, Inc.

    New extensions to subsystems like isolators had to be upstreamed


    Mesos can be made even more customizable and extendable

    Not all organizations can share their work

    Support proprietary and experimental integrations

    Not create bespoke forks of Mesos



  • 2015 Mesosphere, Inc.

    Tie into and control task launch

    Dynamically setup execution environments

    Pass signatures through Mesos

    All of this, transparently to the framework and user

    We needed it to support bespoke security subsystems


  • 2015 Mesosphere, Inc.

    Be able to extend and replace any component in Mesos Allocator algorithms

    Authentication mechanisms

    Advanced scheduling features like oversubscription


    The general thought of Modules was bigger


    Imagine ifI could write my own?

  • 2015 Mesosphere, Inc.

    Modules are old news

    Many large software systems support libraries to

    Extend behavior Isolate and abstract complexity Make this a configuration rather than a

    build exercise

    For example

    Browsers (Firefox) Server software (Apache Webserver) Linux kernel


    Wish I had modules already

  • 2015 Mesosphere, Inc.

    What is a module anyway?

    Module, plugin, extension, library

    Adds or replace a full component

    For example:

    An isolator (works together with existing ones) in the agent

    The allocator and authenticators in the master


  • 2015 Mesosphere, Inc.

    And how about hooks?

    More often than not, you dont want to replace a full component

    Just want to tie into events and their context

    For example:

    Launch task requests at the master

    Launch task requests at the agent

    Exit and cleanup events


    Psst - I just launched a task

  • 2015 Mesosphere, Inc.

    And who is using it?

    Powering new exciting features and integrations!

    Oversubscription modules Static (fixed) estimator

    Dynamic estimator and QoS Controller, project Serenity

    Networking integration with project Calico


  • 2015 Mesosphere, Inc. 15

    Module Mechanics

  • 2015 Mesosphere, Inc.

    A demo!


    A hook module that tags TaskStatus messages

  • 2015 Mesosphere, Inc.



    Isolator InterfaceIsolator Module

    Hook Module H1

    Hook Module H2

    Hook Interface

    Mesos Master/Agent

    Module library


    Module spec JSON


    Initialize subsystems

    use module objects

    Module libraryinitialize modules

    get module object


    call hooks

  • 2015 Mesosphere, Inc.



    First phase: load module libraries compatibility checks, etc. libprocess not available

    Second phase initialize a specific module module-specific parameters libprocess available

  • 2015 Mesosphere, Inc.

    class TestHook : public Hook{public: Result slaveTaskStatusLabelDecorator( const FrameworkID& frameworkId, const TaskStatus& status) { Labels labels; if (status.state() == TASK_RUNNING) { Label* newLabel = labels.add_labels(); newLabel->set_key("whereami"); newLabel->set_value("mesoscon"); } return labels; }};

    A Hook Module


    // Create and return an object or TestHook type. static Hook* createHook(const Parameters& parameters){ // Any initialization checks go here.

    return new TestHook();}

    // Declares a Hook module named org_apache_mesos_TestHook'mesos::modules::Module org_apache_mesos_TestHook( MESOS_MODULE_API_VERSION, MESOS_VERSION, "Apache Mesos", "", "Test Hook module.", NULL, createHook);

  • 2015 Mesosphere, Inc.

    { "libraries": [ { "file": "/path/to/", "modules": [

    { "name": "org_apache_mesos_TestHookModule", "parameters": [ { "key": "agent_addr", "value": "" }, { "key": "...", "value": "..." } ] } ] } ]}

    Specifying Modules to Master/Agent


  • 2015 Mesosphere, Inc.

    Build without building Mesos Just have Mesos installed

    Modules compile into a shared libraries Multiple modules per library

    Specify modules on command --modules=file:///path/to/modules.json --isolation=my_isolator --hooks=my_hook

    Using Modules


  • 2015 Mesosphere, Inc.

    Add/replace a full component Implement the interface Asynchronous (actor model)

    Existing modularized interfaces: Allocator Authentication Authorizer Isolator QoSController ResourceEstimator

    Replacement Modules


  • 2015 Mesosphere, Inc.

    Listen/Intercept interesting calls Occasionally modify the behavior

    Trigger initialization/cleanup Allows us to tag certain tasks, statuses, etc.

    Two broad categories Task launch sequence Status updates

    Hook Modules


  • 2015 Mesosphere, Inc.

    Co-exists with the parent process separate thread of execution

    Create Master/Agent http listen endpoints No callbacks

    Anonymous Modules


    One module to rule them all!

  • 2015 Mesosphere, Inc.

    Do not block Hooks are synchronous Use libprocess/pthreads

    Exit semantics Avoid assertions

    Writing Modules


  • 2015 Mesosphere, Inc.

    Logs stdout/stderr

    Run debug module with non-debug Master/Agent gdb



    What crashed the Master?

  • 2015 Mesosphere, Inc.

    Dependency on other modules Compatibility within set of modules Upgrade path

    rebuild modules when updating Mesos

    Dependency and Compatibility


  • 2015 Mesosphere, Inc. 28

    Future Work

  • 2015 Mesosphere, Inc.

    Safeguard against unsafe modules Limit data exposure Execute modules in a separate process

    Module certification ACLs Runtime functionality checks

    whitelist services can it add routes or not

    Better Safety and Security


  • 2015 Mesosphere, Inc.

    More module interfaces Load/Unload a module without rebooting Master/Agent Upgrade path Express dependability on other modules Inter-module communication Non-C++ modules

    Future Work


  • Thanks for listening!


    Modules repo:

    Mailing list: