1

Deploy an Elastic, Resilient, Load-Balanced Cluster in 5

Minutes with Senlin

Mark Voelker (VMware)Qiming Teng (IBM)Xinhui Li (VMware)

2

Requirements for Resource Pool Management

Senlin Clustering Service Overview

An All-in-one Sample

Agenda

3

Managing Resource Pools

4

Manageability cluster membership

Elasticity key feature of a cloud, pay-as-you-go

Load-Balanced distribute loads evenly on all members

Flexibility customizable to meet the needs of real-life use cases

Extensibility easy to add resource types and policies

Generic Requirements

5

Heat: The Orchestrator AutoScalingGroup +

Scaling Policy + Alarm Heat AutoScaling modeled after AWS

AutoScaling It may suit some limited use cases, but not

all scenarios considering that OpenStackaims to build a platform for both private andpublic clouds

But, Heat's Mission... Orchestrate composite cloud apps using a declarative template format

through an OpenStack-native REST API It is not about any new service, not auto-scaling, not HA, not load-balance ...

What we have on OpenStack

Heat as a CompilerYAML

Template

Heat

Nova

Cinder

Neutron

Keystone

• CREATE• GET• UPDATE• DELETE

6

Cross Availability Zone Placement Cross Region Placement Anti-Affinity Placement Choose a Specific Node to Delete when Scaling-In Trigger Autoscaling Using Application-Level Metrics Manual Scaling Auto Recover Node when Failures Detected Migrate Nodes from a Stand-By Cluster for Rapid Provisioning Soft Scaling ...

Some Specific Requirements

7

Senlin Overview

8

What Do We Really Need?

Scalable

Load-Balanced

Highly-Available

Manageable

......

of any (OpenStack) objects

- What is missing from OpenStack?A Clustering Service

- Auto-scaling?Just one of the usage scenario of a cluster.

- Auto-Healing (HA)?Just another usage scenario.

- We can address the concerns by making policies orthogonal

9

Senlin Architecture

Senlin Engine

Senlin API Senlin

DatabaseSenlin Client

REST RPC

Profiles

Policies

10

Senlin Features Profiles: A specification for the objects to be managed Policies: Rules to be checked/enforced before/after actions are performed

Docker

Senlin

Nova

Kubernetes

Heat

Ironic BareMetal

VMs

Stacks

Containersplacement

deletion

scaling

health

load-balance

batching

Policies as Plugins Profiles as Plugins Cluster/Nodes Managed

11

Senlin Server Architecture

openstacksdkidentity

compute

orchestration

network

...

engineengine lock

scheduler

actions

nodecluster

service

registry

receiverparser

drivers

openstack

dummy

(others)

dbapi

sqlalchemy

rpc clientpolicies

placement

deletionscaling

health

load-balance

batching

receiverwebhoookMsgQueue

extension points for external monitoring services

extension points facilitating a smarter cluster management

extension points to talk to different endpoints for object CRUD operations

extension points for interfacing with different services or clouds

profilesos.heat.stack

(others)

os.nova.server

senlin-apiWSGI

middleware

apiv1

12

Senlin Operations (Actions)

Cluster• CREATE• DELETE• UPDATE• LIST• SHOW• ADD_NODES• DEL_NODES• SCALE_OUT• SCALE_IN• POLICY_ATTACH• POLICY_DETACH• POLICY_UPDATE

Node• CREATE• DELETE• UPDATE• LIST• SHOW

Policy• CREATE• UPDATE• DELETE• LIST• SHOW

Profile• CREATE• UPDATE• DELETE• LIST• SHOW

Action• LIST• SHOW

Event• LIST• SHOW

Receiver• CREATE• DELETE• LIST• SHOW

13

profile/cluster/node operations$ senlin profile-create -s <spec> <name>$ senlin profile-list$ senlin profile-show <profile>$ senlin profile-delete <profile>$ senlin profile-update -s <spec> <profile>$ senlin cluster-create -p <profile> -n <size> <name>$ senlin cluster-delete <cluster>$ senlin cluster-list$ senlin cluster-show <cluster>$ senlin node-create -p <profile> -c <cluster> <name>$ senlin node-delete <node>$ senlin node-show <node>$ senlin node-join -c <cluster> <node>$ senlin node-leave <node>$ senlin cluster-node-add -n <nodes> <cluster>$ senlin cluster-node-del -n <nodes> <cluster>$ senlin cluster-node-list <cluster>

14

policies operations

$ senlin policy-create -s <spec> <name>$ senlin policy-list$ senlin policy-show <policy>$ senlin policy-delete <policy>$ senlin policy-update -n <name> ... <policy>

$ senlin cluster-policy-attach <cluster> <policy>$ senlin cluster-policy-detach <cluster> <policy>$ senlin cluster-policy-list ... <cluster>$ senlin cluster-policy-show <cluster> <policy>$ senlin cluster-policy-enable <cluster> <policy>$ senlin cluster-policy-disable <cluster> <policy>$ senlin cluster-policy-update ... <cluster> <policy>

15

Other Useful Operations$ senlin cluster-scale-in [-c <count>] <cluster>$ senlin cluster-scale-out [-c <count>] <cluster>$ senlin cluster-resize <options> <cluster>

$ senlin profile-type-list$ senlin profile-type-show <type>$ senlin policy-type-list$ senlin policy-type-show <type>

$ senlin action-list$ senlin action-show <action-id>

$ senlin event-list$ senlin event-show <type>

16

senlin cluster-resizeusage: senlin cluster-resize [-c <CAPACITY>] [-a <ADJUSTMENT>] [-p <PERCENTAGE>] [-t <MIN_STEP>] [-s] [-n MIN] [-m MAX] <CLUSTER>

Positional arguments: <CLUSTER> Name or ID of cluster to operate on.

Optional arguments: -c <CAPACITY>, --capacity <CAPACITY> The desired number of nodes of the cluster. -a <ADJUSTMENT>, --adjustment <ADJUSTMENT> A positive integer meaning the number of nodes to add, or a negative integer indicating the number of nodes to remove. -p <PERCENTAGE>, --percentage <PERCENTAGE> A value that is interpreted as the percentage of size adjustment. This value can be positive or negative. -t <MIN_STEP>, --min-step <MIN_STEP> An integer specifying the number of nodes for adjustment when <PERCENTAGE> is specified. -s, --strict A boolean specifying whether the resize should be performed on a best-effort basis when the new capacity may go beyond size constraints. -n MIN, --min-size MIN New lower bound of cluster size. -m MAX, --max-size MAX New upper bound of cluster size. A value of -1 indicates no upper limit on cluster size.

EXACT CAPACITY?

CAPACITY BY PERCENTAGE?

CHANGE IN CAPACITY?

MINIMUM STEP?

STRICT/BEST-EFFORT?

ADJUST MIN_SIZE?

ADJUST MAX_SIZE?

17

ALL-In-ONE:AS (AutoScaling) + HA (High-Availability) + LB (Load-Balancing)

18

Architecture OverviewHeat Template

Cluster

Scaling Policy LB Policy

Health Policy

LBaaS

Alarm

Receivers

Compute

SCALE IN

SCALE OUT

RECOVER

Profile

19

Profile and Cluster

profile: type: OS::Senlin::Profile properties: type: os.nova.server-1.0 properties: flavor: {get_param: flavor} image: {get_param: image} key_name: {get_param: key_name} networks: - network: {get_param: network} security_groups: - {get_resource: security_group}

cluster: type: OS::Senlin::Cluster properties: desired_capacity: 2 min_size: 2 profile: {get_resource: profile}

20

LB Policylb_policy: type: OS::Senlin::Policy properties: type: senlin.policy.loadbalance-1.0 bindings: [cluster: {get_resource: cluster}] properties: pool: protocol: HTTP protocol_port: 80 subnet: {get_param: pool_subnet} vip: subnet: {get_param: vip_subnet} protocol: HTTP protocol_port: 80 health_monitor: type: HTTP delay: 10 timeout: 5 max_retries: 4

21

Scaling Policies scale_in_policy: type: OS::Senlin::Policy properties: type: senlin.policy.scaling-1.0 bindings: - cluster: {get_resource: cluster} properties: event: CLUSTER_SCALE_IN adjustment: type: CHANGE_IN_CAPACITY number: 1

scale_out_policy: type: OS::Senlin::Policy properties: type: senlin.policy.scaling-1.0 bindings: - cluster: {get_resource: cluster} properties: event: CLUSTER_SCALE_OUT adjustment: type: CHANGE_IN_CAPACITY number: 1

22

Health Policyhealth_policy: type: OS::Senlin::Policy properties: type: senlin.policy.health-1.0 bindings: - cluster: {get_resource: cluster} properties: detection: type: NODE_STATUS_POLLING options: interval: 60 recovery: actions: - RECREATE

23

Receivers and Alarmsreceiver_scale_in: type: OS::Senlin::Receiver properties: cluster: {get_resource: cluster} action: CLUSTER_SCALE_IN type: webhook

scale_in_alarm: type: OS::Ceilometer::Alarm properties: meter_name:network.services.lb.incoming.bytes.rate statistic: avg period: 180 evaluation_periods: 1 threshold: 12000 repeat_actions: True alarm_actions: - get_attr: - receiver_scale_in - channel - alarm_url comparison_operator: le

24

Receivers and Alarmsreceiver_scale_out: type: OS::Senlin::Receiver properties: cluster: {get_resource: cluster} action: CLUSTER_SCALE_OUT type: webhook

scale_out_alarm: type: OS::Ceilometer::Alarm properties: meter_name: network.services.lb.incoming.bytes.rate statistic: avg period: 60 evaluation_periods: 1 threshold: 28000 repeat_actions: True alarm_actions: - get_attr: - receiver_scale_out - channel - alarm_url comparison_operator: ge

25

Short Video, For Free, :-)

26

Questions?

27

Blueprints on Reworking Heat AutoscalingBP Priorit

yDescription

autoscaling-api-resources

high Heat resources invoking AS APIs

as-api-group-resource

high ScalingGroup resource wrapping AS API's group functionality

as-api-policy-resource

high ScalingPolicy resource wrapping AS API's policy functionality

as-api-webhook-resource

high Webhook resource wrapping AS API's execution of webhooks

autoscaling-api-client

high A python client for Heat to interact with AS API

autoscaling-api - A separate service for the implementation of autoscaling w/ Heat

as-engine - A separate engine/service for autoscaling support AS API

as-engine-db - A DB dedicated to autoscaling, using schema created in as-lib-db

as-lib - A separate module to be used by the AS service

as-lib-db - A DB for autoscaling bookkeeping

28

Relation To Other Projects Senlin provides the array data type for cloud programming

Senlin

Ceilometer

Heat

Nova Cinder Neutron Swift Keystone

Horizon

Primitive Data Types

Complex Data Types

struct person { int age; char name[0];}

person team[10]; // Senlin cluster of Heat stacks

// Senlin cluster of nova servers

// Heat stack containing senlin clusters

Mistral MagnumSaharaMonasca Tacker Zaqar