Automatické škálování s OpenStack Rafał Szmigiel

OpenStack Solution Architect/Red Hat

Ahoj!

● Rafał Szmigiel – rafal@redhat.com

● Red Hat OpenStack Solutions Architect since December 2015

● Previously:

● OpenStack Operations Engineer @ Rackspace● Systems Engineer @ Wydawnictwo Naukowe PWN S.A.

What is CLOUD COMPUTING?

Cloud Computing

Cloud computing is a type of Internet-based computing that provides shared computer processing resources and data to computers and other devices on demand. It is a model for enabling ubiquitous, on-demand access to a shared pool of configurable computing resources (e.g., computer networks, servers, storage, applications and services), which can be rapidly provisioned and released with minimal management effort.

Source: https://en.wikipedia.org/wiki/Cloud_computing

Cloud Computing

Cloud computing is a type of Internet-based computing that provides shared computer processing resources and data to computers and other devices on demand. It is a model for enabling ubiquitous, on-demand access to a shared pool of configurable computing resources (e.g., computer networks, servers, storage, applications and services), which can be rapidly provisioned and released with minimal management effort.

Source: https://en.wikipedia.org/wiki/Cloud_computing

The problem.

Workload underestimation

Monday Tuesday Wednesday Thursday Friday Saturday Sunday0

20

40

60

80

100

120

140

160

180

Load capacity

Workload overestimation

Monday Tuesday Wednesday Thursday Friday Saturday Sunday0

20

40

60

80

100

120

140

160

180

Load capacity

The solution: auto-scaling infrastructure

Monday Tuesday Wednesday Thursday Friday Saturday Sunday0

20

40

60

80

100

120

140

160

180

Load capacity

Heat.( The orchestrator )

OpenStack Orchestrator - Heat

● Provides human and machine accessible orchestration service● Managing the entire life cycle of cloud infrastructure and applications● Uses templates to describe and configure components and...● … relations and dependencies between them.

It is HOT – Heat Orchestration Template

HOT is a template format supported by the Heat along with the other template format, i.e. the Heat CloudFormation-compatible format (CFN). It is expressed in YAML format.

The structure of a Heat Orchestration Template has three main sections:

● Parameters - These are settings passed to heat, which provides a way to customize a stack, and any default values for parameters without passed values. These are defined in the parameters section of a template.

● Resources - These are the specific objects to create and configure as part of a stack. OpenStack contains a set of core resources that span across all components. These are defined in the resources section of a template.

● Output - These are values passed from heat after the stack’s creation. You can access these values either through the heat API or client tools. These are defined in the output section of a template.

Simple Heat template

heat_template_version: 2015-04-30

description: Simple template to deploy a single compute instance

resources:

my_instance:

type: OS::Nova::Server

properties:

key_name: my_ssh_key

image: RHEL7-x86

flavor: m1.small

Heat – resources$ openstack orchestration resource type list

| OS::Glance::Image || OS::Neutron::FloatingIP || OS::Neutron::FloatingIPAssociation || OS::Neutron::Net || OS::Neutron::NetworkGateway || OS::Neutron::Port || OS::Neutron::ProviderNet || OS::Neutron::RBACPolicy || OS::Neutron::SecurityGroup || OS::Nova::Flavor || OS::Nova::FloatingIP || OS::Nova::FloatingIPAssociation || OS::Nova::HostAggregate || OS::Nova::KeyPair || OS::Nova::Server || OS::Glance::Image |

Heat – stack’s lifecycle management

Create a stack$ openstack stack create -t autoscaling.yaml -e autoscaling-environment.yaml stack-wordpress

+---------------------+--------------------------------------+| Field | Value |+---------------------+--------------------------------------+| id | 703987b8-39aa-4252-95af-0f25066486db || stack_name | stack-wordpress || description | AutoScaling Wordpress || creation_time | 2016-10-31T17:54:30 || updated_time | None || stack_status | CREATE_IN_PROGRESS || stack_status_reason | Stack CREATE started |+---------------------+--------------------------------------+

Heat – stack’s lifecycle managementList stacks

$ openstack stack list

| ID | Stack Name | Stack Status | Creation Time |+-----+-----------------+-----------------+---------------------+| 703 | stack-wordpress | CREATE_COMPLETE | 2016-10-31T17:54:30 |+-----+-----------------+-----------------+---------------------+

Get stack’s resources$ openstack stack resource list ${STACK_NAME_OR_UUID}

| resource_name | resource_type | resource_status |+----------------+----------------------------+-----------------+| lb | OS::Neutron::LoadBalancer | CREATE_COMPLETE || cpu_alarm_high | OS::Ceilometer::Alarm | CREATE_COMPLETE || lb_floating | OS::Neutron::FloatingIP | CREATE_COMPLETE || db | OS::Nova::Server | CREATE_COMPLETE || monitor | OS::Neutron::HealthMonitor | CREATE_COMPLETE |+----------------+----------------------------+-----------------+

Heat – stack’s lifecycle management

Few other example commands to manage stack’s lifecycle:$ openstack stack resource show ${STACK_NAME_OR_UUID} ${RESOURCE_ID}

$ openstack stack output list ${STACK_NAME_OR_UUID}

$ openstack stack suspend ${STACK_NAME_OR_UUID}

$ openstack stack resume ${STACK_NAME_OR_UUID}

$ openstack stack update --dry-run ${ARGS} ${STACK_NAME_OR_UUID}

$ openstack stack delete ${STACK_NAME_OR_UUID}

OpenStack’stelemetry services.

OpenStack Telemetry

● Ceilometer – data collection service● Aodh – alarming service● Gnocchi – time-series DB and resource indexing service

Auto-scaling( automagically )

Heat – auto scaling group asg: type: OS::Heat::AutoScalingGroup properties: min_size: 1 max_size: 6 resource: type: OS::Nova::Server properties: flavor: {get_param: flavor} image: {get_param: image} key_name: {get_param: key_name} metadata: {get_param: metadata} user_data: {get_param: user_data} user_data_format: RAW networks: [{network: {get_param: network} }] (...)

Heat – auto scaling policyweb_server_scaleup_policy:

type: OS::Heat::ScalingPolicy

properties:

adjustment_type: change_in_capacity

auto_scaling_group_id: {get_resource: asg}

cooldown: 60

scaling_adjustment: 1

web_server_scaledown_policy:

type: OS::Heat::ScalingPolicy

properties:

adjustment_type: change_in_capacity

auto_scaling_group_id: {get_resource: asg}

cooldown: 60

scaling_adjustment: -1

Heat – alarm definition cpu_alarm_high:

type: OS::Ceilometer::Alarm

properties:

description: Scale-up if the average CPU > 50% for 30 seconds

meter_name: cpu_util

statistic: avg

period: 30

evaluation_periods: 1

threshold: 50

alarm_actions:

- {get_attr: [web_server_scaleup_policy, alarm_url]}

matching_metadata: {'metadata.user_metadata.stack': {get_param: "OS::stack_id"}}

comparison_operator: gt

Externally triggered scaling action

It is as easy as HTTP POST query:

curl -X POST http://10.0.123.10:8000/v1/signal/arn%3Aopenstack%3Aheat%3A%3A00cac25bdaae4d20b875954630e2dbe4%3Astacks%2Fstack-wordpress%2F6a5308ec-8dee-4b9e-9d8d-0f473e2fce85%2Fresources%2Fweb_server_scaleup_policy?Timestamp=2016-10-31T19%3A41%3A34Z&SignatureMethod=HmacSHA256&AWSAccessKeyId=df7b944c180a47c9a958fb504f0d6590&SignatureVersion=2&Signature=dGrYqZsnOdgtjzG9XNuSiqwaWPL1jmX1cRyfN16%2B1oQ%3D

What’s happening there?!( And why I got 1000 instances on Sunday morning? )

Heat$ openstack stack event list ${STACK_NAME_OR_UUID}

2016-10-31 22:56:16 [web_server_scaledown_policy]: SIGNAL_COMPLETE alarm state changed from alarm to alarm (Remaining as alarm due to 1 samples outside threshold, most recent: 4.9663441268)

2016-10-31 22:57:17 [web_server_scaledown_policy]: SIGNAL_COMPLETE alarm state changed from alarm to alarm (Remaining as alarm due to 1 samples outside threshold, most recent: 2.15981422408)

2016-10-31 22:59:30 [web_server_scaleup_policy]: SIGNAL_COMPLETE alarm state changed from ok to alarm (Transition to alarm due to 1 samples outside threshold, most recent: 75.6321721782)

2016-10-31 23:03:32 [web_server_scaleup_policy]: SIGNAL_COMPLETE alarm state changed from alarm to alarm (Remaining as alarm due to 1 samples outside threshold, most recent: 134.256260301)

2016-10-31 23:04:40 [web_server_scaleup_policy]: SIGNAL_COMPLETE alarm state changed from alarm to alarm (Remaining as alarm due to 1 samples outside threshold, most recent: 65.1931917983)

Aodh

$ aodh alarm list

| alarm_id | type | name | state | severity | enabled |+-----------+-----------+--------------------+-------+----------+---------+| 17110da8- | threshold | stack-cpu_alarm_lo | alarm | low | True || a54bc325- | threshold | stack-cpu_alarm_hi | ok | low | True |+-----------+-----------+--------------------+-------+----------+---------+

$ aodh alarm-history show a54bc325-3bdb-4745-bbcc-3419d3a13f56

| timestamp | type | detail |+-----------+------------------+--------------------+| 23:05:04 | state transition | {"state": "ok"} || 22:59:03 | state transition | {"state": "alarm"} || 20:59:03 | state transition | {"state": "ok"} |+-----------+------------------+--------------------+

Ceilometer $ ceilometer statistics -m cpu_util -a avg -p 60| Period | Start | Avg | Duration || 60 | 22:53:58 | 3.10023188636 | 37.049 || 60 | 22:54:58 | 9.87736496115 | 37.053 || 60 | 22:55:58 | 1.53177061562 | 51.76 || 60 | 22:56:58 | 12.168116509 | 37.063 || 60 | 22:57:58 | 26.5699939046 | 37.047 || 60 | 22:58:58 | 38.3967840398 | 37.062 || 60 | 22:59:58 | 67.6228733554 | 48.498 || 60 | 23:00:58 | 46.8100887795 | 48.555 || 60 | 23:01:58 | 48.5380955223 | 48.478 || 60 | 23:02:58 | 22.4152868243 | 48.509 || 60 | 23:03:58 | 32.4806136739 | 37.55 || 60 | 23:04:58 | 50.8631014266 | 48.473 || 60 | 23:05:58 | 20.1688076082 | 51.637 || 60 | 23:06:58 | 17.0933995846 | 48.455 || 60 | 23:07:58 | 2.24006997852 | 48.499 || 60 | 23:08:58 | 4.19847201624 | 48.464 || 60 | 23:09:58 | 2.82519301419 | 48.44 |

Questions?

Thank you

● https://keithtenzer.com/

● http://docs.openstack.org/developer/heat/template_guide/hot_guide.html

● https://github.com/openstack/heat-templates