Building a cloud management platform using megam.io

Yeshwanth KumarPlatform EngineerMegam systems

Agenda :

1. Introduction to cloud orchestration2. Megam’s Cloud Management Platform3. Architecture4. Cloud In A Box5. Demo

What is cloud orchestration?

In the world of software, orchestration is the entity that manages complex cross domain (system, enterprise, firewall) processes, and that handles exceptions.

How does cloud orchestration help?

➔ Rapid application deployment

➔ end-to-end automation

➔ Multiple cloud support, so no vendor lock-ins

megam’s Cloud Management Platform

megam’s Cloud Management Platform

➔ Omni scheduler (VM, containers, unikernel…)

➔ Launching apps/services

➔ Namespace grouping (organization &domain)

➔ Metering and monitoring, logs

➔ SCM and continuous integration

https://console.megam.io

Architecture:

Megam’s architecture

● REST-ful● Stateless● Asynchronous● Loosely coupled● API Driven● Declarative

Web & CLI

API Gateway server

RabbitMQ

PaaS Engine - megamd

Gulpd agent

Gateway Server - Powerful REST API

● Scala● Play framework● Stateless● Super secure - HMAC based auth● Adheres to CAMP/TOSCA

Gateway Server

Robust functional techniques

match use_functional() {

case Some(succ) => “Yay! easy to scale, me :)”

case None => “me :(”

}

Functional concise code high scalability

API Gateway server

Error handling - scalaz

Why ValidationNel?

def create(email: String, input: String): ValidationNel[Throwable, Option[EventsResult]] = {

(mkGunnySack(email, input) leftMap { err: NonEmptyList[Throwable] =>

new ServiceUnavailableError(input, (err.list.map(m => m.getMessage)).mkString("\n"))

}).toValidationNel.flatMap { gs: Option[GunnySack] =>

(riak.store(gs.get) leftMap { t: NonEmptyList[Throwable] => t }).

flatMap { maybeGS: Option[GunnySack] =>

maybeGS match {

case Some(thatGS) => (parse(thatGS.value).extract[EventsResult].some).successNel[Throwable]

case None => {

play.api.Logger.warn(("%-20s -->[%s]").format("Events created. success", "Scaliak returned => None.

Thats OK."))

(parse(gs.get.value).extract[EventsResult].some).successNel[Throwable];

}

Transport layer

A Request that comes to megam_gateway gets funnelled

HTTPRequest

(Headers, Body)

|

\ /

|

(becomes)

FunneledRequest(maybeEmail,clientAPIHmac, clientAPIDate,

clientAPIPath, clientAPIBody)

Transport layer - marshalling/unmarshalling

toJSON

override implicit val writer = new JSONW[AccountResult] {

override def write(h: AccountResult): JValue = { JObject( JField(IdKey, toJSON(h.id)) :: JField(FirstNameKey, toJSON(h.first_name)) :: JField(LastNameKey, toJSON(h.last_name)) }}

fromJSON

override implicit val reader = new JSONR[AccountResult] {

override def read(json: JValue): Result[AccountResult] = { val idField = field[String](IdKey)(json) val firstNameField = field[String](FirstNameKey)(json) val lastNameField = field[String](LastNameKey)(json) val phoneField = field[String](PhoneKey)(json)

● Store validated schema

Declarative based platform

Assemblies

Assembly Assembly

Components Components Components

#tosca_definitions_version: tosca_simple_yaml_1_0

description: Template for deploying a spring app

node_templates:

orion: type: tosca.app.java

properties: # omitted here for sake of brevity

requirements: - host: megam_host - domain: megambox.com - source: https://github.com/megamsys/spring-mvc.git

JSON

Assembly

{

“name”: “assembly_name”,

“components” : [“component_1”, “”],

“inputs” : [ “any_global_inputs(tab, sheet, id)”, “ ”],

“operations” : “restart/reboot”,

“groups” : {

“placement_policy” : {

“ha_policy” :{

“scaling_policy” :{

}

}

Components

{“name”: “component_1”,“tosca_type”: “tosca.web.Java”,

“requirements” : {

“host”: “cloud_setting_id },

“inputs” : { “domain”: “megam.co”, “port”: “6379”,

“username”: “ ”,

“password”: ” “,

“version”: “ “ ,

“source”: “ “ ,

}, “service_inputs”: { “dbname”: “ “,

“dbpassword”: “ “ },

“status”: “RUNNING”, }

Workings..

Data from UI

(JSON )Controller

Models GunnySack

serialize

CloudPublish(RabbitMQ)

reqFunneled

db

Async, it is! - Leveraging message brokers - RabbitMQ

● open source

● More flexibility for namespace groupings

● Independant control of each apps, services and VMs

Megam queues -

➔ CloudStandup ➔ CloudPerNode

megamd engine/scheduler

➔ Multiple transport protocols - mini servers that run

concurrently (http, queues) in its own space

➔ Extensible provisioner approach (Docker, vm,

unikernel, lxc ....)

➔ Action based

➔ Recoverable actions (good/bad) with state rollback

facility though

Workings...

Queue engine

Provisioner Action

Launch

Store (Riak)

reqData from Queue

(JSON)

Concurrency - goroutines vs akka workersfunc (self *Server) ListenAndServe() error {

log.Info("Starting admin interface on port")

//var etcdServerList [2]string

var queueInput [3]string

queueInput[0] = "cloudstandup"

queueInput[1] = "events"

self.Checker()

// Queue input

for i := range queueInput {

listenQueue := queueInput[i]

queueserver := queue.NewServer(listenQueue)

go queueserver.ListenAndServe()

}

control app state on the cloud - gulpd agent

● start

● stop

● restart

● apply policy - Bind

- HA

Full working of megam

UI

CLI

API Gateway RabbitMQ megamd

app1(gulpd)

app2(gulpd)

app3(gulpd)

cloudstandupapp1.megam.comapp2.megam.comapp3.megam.com

ChefDocker

What does megam provide?

Hybrid

Megam IaaS Megam CIB

● Easy to move from public to private cloud

Demo

Lets launch application, service & bind them seperately..

Thank you

Questions?

Yeshwanth Kumar

Platform Engineer

Megam Systems

(www.megam.io)

Twitter: @megamsys Docs: docs.megam.io

Devcenter: devcenter.megam.io