D3.3: BIGSEA QoS infrastructure services initial version · 2017. 5. 23. · Mesos/Marathon/Chronos...

www.eubra-bigsea.eu | [email protected] @bigsea_eubr 1

D3.3: BIGSEA QoS infrastructure services initial version

Author(s) Ignacio Blanquer (UPV), Germán Moltó (UPV)

Status Approved

Version V1

Date 30.12.2016

Dissemination Level

X PU: Public

PP: Restricted to other programme participants (including the Commission)

RE: Restricted to a group specified by the consortium (including the Commission)

CO: Confidential, only for members of the consortium (including the Commission)

Abstract: Europe - Brazil Collaboration of BIG Data Scientific Research through Cloud-Centric Applications (EUBra-BIGSEA) is a medium-scale research project funded by the European Commission under the Cooperation Programme, and the Ministry of Science and Technology (MCT) of Brazil in the frame of the third European-Brazilian coordinated call. The document has been produced with the co-funding of the European Commission and the MCT. The purpose of this report is to describe the implementation of the cloud service cluster, as well as to provide access guidelines to it.

EUBra-BIGSEA is funded by the European Commission under the Cooperation Programme, Horizon 2020 grant agreement No 690116.

Este projeto é resultante da 3a Chamada Coordenada BR-UE em Tecnologias da Informação e Comunicação (TIC), anunciada pelo Ministério de Ciência, Tecnologia e Inovação (MCTI)

EUBra-BIGSEA D3.3 : QoS Infrastructure services initial version

2

www.eubra-bigsea.eu | [email protected] |@bigsea_eubr

Document identifier: EUBRA BIGSEA –WP3-D3.3

Deliverable lead UPV

Related work package WP3

Author(s) Ignacio Blanquer (UPV), Germán Moltó (UPV), Andy S Alic (UPV)

Contributor(s) See versioning and contribution history

Due date 31/12/2016

Actual submission date 31/12/2016

Reviewed by Athanasia Evangelinou (POLIMI), Jussara Almeida (UFMG)

Approved by PMB

Start date of Project 01/01/2016

Duration 24 months

Keywords Cloud Services, QoS

Versioning and contribution history

Version Date Authors Notes

0.1 17/11/16 Ignacio Blanquer (UPV) TOC

0.2 5/12/16 Ignacio Blanquer (UPV) First draft for section 3

0.3 12/12/16 Raquel Vigolvino (UFCG), Falvio

Ramalho (UFCG), Andrey Brito (UFCG)

Section 4

0.4 14/12/16 Ignacio Blanquer (UPV) Complete section 3, section 5 and 6.

0.5 27/12/16 Ignacio Blanquer (UPV) Address review comments from Jussara Almeida (UFMG) and Athanasia Evangelinou (POLIMI)

Copyright notice: This work is licensed under the Creative Commons CC-BY 4.0 license. To view a copy of this license, visit https://creativecommons.org/licenses/by/4.0. Disclaimer: The content of the document herein is the sole responsibility of the publishers and it does not necessarily represent the views expressed by the European Commission or its services. While the information contained in the document is believed to be accurate, the author(s) or any other participant in the EUBra-BIGSEA Consortium make no warranty of any kind with regard to this material including, but not limited to the implied warranties of merchantability and fitness for a particular purpose. Neither the EUBra-BIGSEA Consortium nor any of its members, their officers, employees or agents shall be responsible or liable in negligence or otherwise howsoever in respect of any inaccuracy or omission herein. Without derogating from the generality of the foregoing neither the EUBra-BIGSEA Consortium nor any of its members, their

officers, employees or agents shall be liable for any direct or indirect or consequential loss or damage caused by or arising from any

information advice or inaccuracy or omission herein.


3


TABLE OF CONTENT

1 Introduction .............................................................................................................................. 6

1.1 Scope of the Document .................................................................................................... 6

1.2 Target Audience ............................................................................................................... 6

1.3 Structure ........................................................................................................................... 6

2 Cloud Services Architecture ..................................................................................................... 7

2.1 Status of Requirements .................................................................................................... 7

2.2 Layers and Components ................................................................................................. 10

2.3 Interactions among Components .................................................................................... 10

2.4 Interface with upper layers .............................................................................................. 11

3 Configuration and Deployment ............................................................................................... 12

3.1 Platforms supported ........................................................................................................ 12

3.2 Deployment of the system............................................................................................... 13

3.2.1 Client setup and configuration .................................................................................. 13

3.2.2 Update of templates and launching .......................................................................... 14

3.2.3 Log-in, power-on and validation ............................................................................... 16

3.2.4 End-points ............................................................................................................... 17

4 Monitoring .............................................................................................................................. 17

4.1 Monasca Installation ....................................................................................................... 18

4.1.1 Requirements .......................................................................................................... 18

4.2 Deployment .................................................................................................................... 18

4.2.1 Using DevStack ....................................................................................................... 18

4.2.2 Using Ansible ........................................................................................................... 20

4.3 Monasca basics .............................................................................................................. 20

4.3.1 Metrics ..................................................................................................................... 20

4.3.2 Agent ....................................................................................................................... 21

4.4 Monasca Probes ............................................................................................................. 26

4.4.1 How to monitor the progress of a batch application? ................................................ 29

4.5 Infrastructure instrumentation for vertical scaling ............................................................ 29

4.5.1 Scaler ...................................................................................................................... 30

4.5.2 Actuator ................................................................................................................... 30

5 Security model ....................................................................................................................... 30

6 Conclusions ........................................................................................................................... 31

7 GLOSSARY ........................................................................................................................... 31


4


LIST OF FIGURES

Figure 1. A description of the interaction among the components ................................................. 11

Figure 2. Interfacing with the upper layers .................................................................................... 12

Figure 3. Schema of the deployment of the BIGSEA cloud cluster ................................................ 14

Figure 4. Snapshot from OpenNebula Sunstone ........................................................................... 14

Figure 5. Monitoring Cycle ............................................................................................................ 22

Figure 6. Scaling Method .............................................................................................................. 29

LIST OF TABLES

Table 1. Status of the requirements ................................................................................................ 9


5


EXECUTIVE SUMMARY

EUBra-BIGSEA project aims at developing a set of cloud services empowering Big Data analytics to ease the

development of massive data processing applications. EUBra-BIGSEA models lie on a predictive and reactive

cloud infrastructure developed and integrated in WP3. The cloud services serve the programming models

and the privacy framework for the execution of the workload defined by the application areas.

This document describes the implementation of the Quality of Service (QoS) platform that supports the

executions of the data analytic jobs. This platform follows the architecture design defined in the D3.1.

This deliverable is actually a prototype, and this document serves as a guide to describe, deploy and access

such infrastructure. The document briefly describes the architecture and explains how to deploy your own

instance in a public or on-premise provider, respectively. The document also describes the tests of the

infrastructure and the monitoring system.

The BIGSEA platform is deployed by means of a client that includes the following services: a Mesos master,

several Mesos agents and several Hadoop Datanodes. The services include Marathon, Chronos, Spark, Docker

and a shared file system.

The monitoring of the system is based on MONASCA, a scalable monitoring infrastructure for cloud-based

infrastructures.


6


1 INTRODUCTION

1.1 Scope of the Document

This document describes the cloud services to be used by the other components in the project to execute

the processing workload. The document covers the implementation details of the architecture depicted in

D3.1 QoS Monitoring System Architecture. The document describes how to deploy a fully functional cluster,

how to access it and how the components are integrated.

Through the document you will find the instructions to tune-up the BIGSEA cloud services recipes to deploy

them on different IaaS, as well as the security mechanisms for deploying frameworks.

Finally, it covers the QoS Monitoring system, based on Monasca, describing how to implement probes and

deploy agents.

1.2 Target Audience

The document is mainly intended for internal use, although it is publicly released. There is an internal version

that includes an annex with the IPs, users and credentials needed to access the system.

1.3 Structure

The document is structured into 6 sections. After this introduction, a summary of the Cloud Service

Architecture is provided in section 2. Then, the configuration, deployment and access to the Cloud-service

platform is described in section 3. The monitoring service is described in section 4 and section 5 covers the

security model used, which will be updated by WP6, and section 6 deals with the conclusions.


7


2 CLOUD SERVICES ARCHITECTURE

The Cloud services architecture of EUBra-BIGSEA differentiates two main layers: the Physical resources management and the logical (and virtual resources)

management.

2.1 Status of Requirements

D3.1 analyses the requirements identified by the applications and the rest of the WPs. We include a column that describe the degree of

achievement of each one of the requirements.

RE.1. Unrestricted Batch jobs

RE.2. Unrestricted Bag of Tasks

RE.3. QoS Batch jobs

RE.4. Deadline based

scheduling

RE.5. Self adapting elasticity

RE.6. Short-jobs

R01 - The RMF must enable a scheduler to allocate resources.

x x x x x Fulfilled

R02 - The RMF must support multiple types of workloads.

x x x x x Marathon, Chronos, YARN, COMPSs in

process

R03 - An application topology may involve several containers.

x x In progress

R04 - The status of the resources should be accessible by an API.

x Mesos and Monasca API

R05 - RMFs should expose an API to change the number of resources.

x x x Mesos/Marathon/Chronos API

R06 - Monitor the usage of resources and the application health.

x x x Through Monasca and Monasca API

R07 - A central list of containers images should be available.

x x x Docker Hub Repository


8


R08 - Updates should be automated. x x x Docker Hub Repository / automated build

R09 - A user should be able to modify its own application.

x x x Through

Mesos/Marathon/Chronos API

R10 - Capability of executing container-based batch jobs.

x x x Docker

R11 - Integration with the selected CMF x x ONE, OSTACK, Azure, AWS, Google Cloud

R12 - Capability of executing jobs involving concurrent processes.

x Through COMPSs

R13 - Support of Spark jobs. x x x x x Fulfilled

R14 - Support of deadline-QoS periodic jobs. x x In progress

R15 - High availability for long-running jobs. x x x Through Marathon

R16 - Support for interactive Jobs. x Fulfilled

R17 - Deployment of TOSCA blueprints. x x x x x x Currently RADL, in progress

R18 - Update of TOSCA blueprints to reconfigure the system.

x Currently RADL, in

progress

R19 - Support of multiple platforms. x x x x x x ONE, OSTACK, Azure,

AWS, Google Cloud

R20 - Automatic scaling up of resources when new job requests arise.

x x x Fulfilled through CLUES


9


R21 - Automatic deallocation of resources when idle for a given period.

x Fulfilled through CLUES

R22 - Customization policies for elasticity. x Fulfilled through CLUES

R23 - Reallocation of memory size and number of resources.

x In progress

R24 - Transparent management of memory allocation.

x Through CLOUDVAMP

R25 - Automatic reconfiguration of execution kernels.

x In progress

Table 1. Status of the requirements


10


2.2 Layers and Components

The EUBraBIGSEA platform defines two layers:

- The physical resources management components, which comprise the components required to

configure resources, allocate and monitor workload and provide elasticity.

- The logical components, which comprise those components used by the developers to build their

programs.

For each one of the layers, we have the following components:

- Logical components:

- Lemonade: An Integrated Development Environment for data analytics.

- COMPSs: A platform-agnostic framework for the execution of concurrent applications.

- Spark (*): A framework for In-memory data analytics.

- Quality Enactment architecture: A framework for defining the proper allocation of

resources for an application.

- Physical resource management components:

- Marathon (*): A fault-tolerant container-based scheduler for Mesos.

- Chronos (*): A fault-tolerant scheduler with ISO8601 based schedules for Mesos.

- Mesos (*): A resource management framework.

- IM: A platform-agnostic contextualization service.

- CLUES: A Cluster horizontal elasticity service.

- EC3: A framework that integrates IM and CLUES to deploy self-managed elastic clusters.

- CLOUDVAMP: A Cloud Virtual machine Automatic Memory Procurement for Vertical

elasticity at the level of memory.

- MONASCA (*): A multi-tenant monitoring-as-a-service solution.

- CMF: Any Cloud Management Framework, such as OpenNebula, OpenStack, etc.

Components marked with the star (*) are external components.

2.3 Interactions among Components

Figure 1 shows the component interactions in the EUBraBIGSEA architecture and defines the roles among

them.


11


Figure 1. A description of the interaction among the components

Basically, the interactions are defined in the next order:

1. A user codes his/her application using Spark and Lemonade, or directly on COMPSs.

2. Proactive Policies provides with the values for the memory and cpu requests.

3. If the application does not require any additional software dependency, the user can proceed with the

execution via any of the Frameworks

- spark-submit if it is a pure Spark code.

- COMPSs client for mixed or pure COMPSs code.

4. The Framework requests Mesos the amount of resources proposed by the Proactive Policies.

5. If there are enough resources, Mesos will create the framework and the scheduler will run the job.

6. If there are not, the request will be temporarily blocked.

- In this case, EC3 will detect this issue and request the Cloud Management Framework for more

resources.

- IM configures the new node and reconfigures the whole cluster automatically to make the new

agent available.

7. The monitor system registers the usage of the resources and decides to adjust the allocation of resources

for the framework. It also feeds back to the Quality Enactment.

8. Finally, vertical elasticity at the level of the resource is managed through CLOUDVAMP, which reallocates

the ratio physical vs virtual memory.

2.4 Interface with upper layers

Figure 2 defines the interactions with the upper layers, from the application developer point of view.

IDE(Lemonade)

Prog. Model(COMPSs/Spark)

SchedulerEC3/IM

Mesos

QualityEnactment

Cloud Manag. Framew

Resources

CLOUDVAMPMonitor

Generates &

combines codeCreates &

executes runtime

Defines resource

allocation

Creates

frameworks

Request & configure

VMs

[un] deploys VMsDetect starvation/idle

Triggers vertical

elasticity

Provides agents

Physical Resources

Logical

Components


12


Figure 2. Interfacing with the upper layers

The user can customize the basic image of the container with specific dependencies and register it in Docker

Hub, so it can be used in further executions. For this purpose, the user will pull the base container (1), update

it with the dependencies and push it back to the repository, either through automated build or directly

through Docker push (3). The user develops his/her application in Lemonade and COMPSs and submits the

request to the BIGSEA scheduler by adding the QoS constraints, which are passed to the Quality Enactment

Architecture.

3 CONFIGURATION AND DEPLOYMENT

The EUBra-BIGSEA platform can be instantiated with minimal user or system administration intervention and

without a deep knowledge of the components. Client tools are available as a Docker container, including all

the configuration recipes to deploy all the components of the system. Next sections describe the

requirements and brief instructions to configure and deploy a system.

3.1 Platforms supported

The EUBra-BIGSEA cloud platform can be deployed (at least partially) on different computing backends. The

configuration service supports both public clouds (Amazon AWS, Microsoft Azure and Google Cloud) and

open-source cloud management frameworks (OpenStack and OpenNebula), as well as libvirt, Docker Swarm,

Kubernetes and OCCI.

For the deployment of the system is needed:

- A Linux machine with Docker installed.

- Credentials to access a cloud infrastructure (among the ones described above).


13


- Access to a base Linux Golden Image installed in the cloud infrastructure. We have made the tests

with Ubuntu 14.04, although other images could work also.

The EUBra-BIGSEA cluster launching system is ec3.

3.2 Deployment of the system

Three steps are required to launch the system: Set-up of the client and configuration of the credentials;

update of the templates for the cluster and deployment; cluster log-in, power-on and validation.

3.2.1 Client setup and configuration

The client is available both in GitHub (https://github.com/EUBra-BIGSEA/ec3client) and as a Docker image in

Docker Hub (https://hub.docker.com/r/eubrabigsea/ec3client/). The ec3 system documentation is online

available at http://ec3.readthedocs.io/en/devel/. The easiest way to deploy the client is by pulling and

running the container image. It is strong recommended to execute it in the background for keeping the client

database active. In any case, once you have retrieved the IP, login name and password, you may not need

the client container again.

$ docker pull eubrabigsea/ec3client $ docker run -d -i -t --name mybigseaclucli eubrabigsea/ec3client /bin/bash $ docker exec -i -t mybisgseaclulci /bin/bash

In order to launch a BIGSEA infrastructure you need to have the proper credentials. The credentials should

be located in a file with multiple lines following the next structure:

id = id_value ; type = value_of_type ; username = myusername ; password = mypassword

You must include at least two lines:

type = InfrastructureManager ; username = yourIMusername ; password = anypassword id = myiaas ; type = IAASTYPE ; host = host:port ; username = youriaasuser ; password =

youriaaspass

Where yourIMusername is a username for the Infrastructure Manager configuration system. If this is the

first time you invoke it, a new account will be created with the password provided in the same line. If not, it

will try to authenticate with the credentials provided. By using the im_client you can query any

deployment you may have created. The second line contains the access details for your cloud infrastructure.

The supported infrastructure flavours, defined in the keyword “type” of the authorization file are the

following: OpenNebula, EC2, FogBow, OpenStack, OCCI, LibCloud, Docker, GCE, Azure and

Kubernetes. The host corresponds to the endpoint where the IaaS is accessible. In the case of EC2 you do

not need to provide any host, and username refers to the Access Key ID while password refers to the

Secret Access Key.

Create an “auth.dat” file with the previous information and place it in any directory. You will refer it

through the -a argument.

https://github.com/EUBra-BIGSEA/ec3client

https://hub.docker.com/r/eubrabigsea/ec3client/

http://ec3.readthedocs.io/en/devel/


14


3.2.2 Update of templates and launching

The configuration states in the “ubuntu-one.radl” template the configuration of the base image and network

of a specific infrastructure. You should check the following information before updating it:

- The name of the base image you want to use and the associated credentials.

- The name of the cluster nodes you want to deploy.

You can use Sunstone1 or Horizon2 to discover this information. Figure 3 shows a schema of the deployment.

In the case of EC2 you should identify a valid AMI id (e.g. an Ubuntu 14.04).

Figure 3. Schema of the deployment of the BIGSEA cloud cluster

For example, in the UPV’s case, we use the image 95. The IM system automatically identifies the “public”

and “private” networks and automatically associate them to the definitions of “public” and “private” in

the recipes. Next figure shows a snapshot from OpenNebula Sunstone. In the left image, the list of VMIs is

listed. Image 95 corresponds to a Ubuntu 14.04 image. In the right side, a list of the available networks is

shown.

Figure 4. Snapshot from OpenNebula Sunstone

1 https://archives.opennebula.org/documentation:archives:rel2.2:sunstone 2 http://docs.openstack.org/developer/horizon/


15


Therefore, the items marked in red should be changed in the “ubuntu-one.radl” file should be modified,

as shown:

disk.0.image.url = 'one://ramses.i3m.upv.es/95' and disk.0.os.credentials.username = 'existing_user_in_the_image' and disk.0.os.credentials.password = 'valid_passwd_for_the_user'

In the case of AWS, the image url will point out to the EC2 AMI (e.g. aws://us-east-1/ami-c93a83de) The

names of the cluster are embedded in the “mesos.radl” file, as well as the access credentials. If you want

to update so, change the following lines:

system front ( cpu.count>=1 and memory.size>=4g and net_interface.0.connection = 'private' and net_interface.0.dns_name = 'InternalServerName' and net_interface.1.connection = 'public' and net_interface.1.dns_name = 'PublicServerName' and queue_system = 'mesos' and ec3_templates contains 'mesos' )

and

configure framework_vars ( @begin principal: yourprincipal secret: yoursecret @end )

and

configure front ( @begin --- ... - copy: content: | {% for number in range(1, NNODES|int + 1) %} agentbasename{{number}} {% endfor %} dest: "/etc/clues2/mesos_vnodes.info"

@end )


16


Finally, you should add the users you want to give access to by including them in the listings provided in the

files hadoop.radl and bigseausers.radl. In those files you can find one single user (apart from the one already

existing in the VMI) so you can easily add new users to be automatically created in all the VMIs, including the

creation of the HDFS directories and the NFS mounted home directories.

To launch the cluster, you should run the launch command with the proper configuration files.

./ec3 launch myBIGSEACluster ubuntu-one mesos bigseausers docker spark nfs hadoop monasca -a

/root/auth.dat

The name myBIGSEACluster can be replaced by any other name. If the process finishes successfully, the

system will show a message like:

Infrastructure successfully created with ID: d0835294-b892-11e6-a466-300000000002 Front-end configured with IP 158.42.105.63 Transferring infrastructure Front-end ready!

3.2.3 Log-in, power-on and validation

You can get information about the cluster with ./ec3 show myBIGSEACluster, including the new

password for logging in the front-end. You can also log-in the front end with ./ec3 ssh myBIGSEACluster

(or the name you used to created it).

Once you are inside the front-end, you will need to power on the three first WNs, which will host the HDFS

storage, and disable them to prevent the elastic system to power them off.

$ ./ec3 ssh myBIGSEACluster2 node state enabled time stable (cpu,mem) used (cpu,mem) total ----------------------------------------------------------------------------------------------- bsagent1 off enabled 00h14'29" 0,0 1,1572864000 bsagent2 off enabled 00h14'29" 0,0 1,1572864000 bsagent3 off enabled 00h14'29" 0,0 1,1572864000 bsagent4 off enabled 00h14'29" 0,0 1,1572864000 bsagent5 off enabled 00h14'29" 0,0 1,1572864000 bsagent6 off enabled 00h14'29" 0,0 1,1572864000 bsagent7 off enabled 00h14'29" 0,0 1,1572864000

ubuntu@mesosserver:~$ clues poweron bsagent1 bsagent2 bsagent3 node bsagent1 powered on node bsagent2 powered on node bsagent3 powered on ubuntu@mesosserver:~$ clues disable bsagent1 bsagent2 bsagent3


17


Then, you can manually power on additional nodes (they will be configured as mesos slaves) or leave the

elastic system to do it for you when you submit a job. Nodes can be disabled through clues disable bsagent1

bsagent2 bsagent3, to avoid CLUES to power them off (this should the case for the first three nodes that are

defined as Hadoop data nodes).

Once the system is working, you can test it by using the testing suite provided in the GitHub repository

https://github.com/EUBra-BIGSEA/Elastic-Cluster-Testing

The tests provided in the above repository include:

- Submit a job through Chronos

- Submit a job through Marathon

- Submit a container-based job through Chronos

- Submit a container-based job through Marathon

- Submit Map-Reduce jar jobs through hadoop

- Submit a Python job through spark-submit with no interaction with HDFS

- Submit a Python job through spark-submit with interaction with HDFS

- Deploy the Spark-shell and run a job in Scala.

3.2.4 End-points

The cluster deploys automatically five services deployed in the private addresses of the front-end node, but

that can be accessed from outside of the cluster through NAT. The end-points are:

- Mesos master service at http://PublicServerName:5050

- Marathon service at http://PublicServerName:8080

- Chronos service at http://PublicServerName:4400

- History server for Spark at http://PublicServerName:18080

- Grafana at http://PublicServerName:5000

4 MONITORING

Monitoring everything is the most important activity to achieve quality of service (QoS). Monitoring services

may be useful to predict fails, make decisions (e.g. when and where to allocate some resource), balance load,

monitor health status, and others. A good platform to collect and store monitoring information is Monasca,

a powerful alternative considering the cloud environment. Monasca is an open-source multi-tenant, highly

scalable, performant, fault-tolerant monitoring-as-a-service solution that integrates with OpenStack. It uses

a REST API for high-speed metrics processing and querying and both streaming alarm and notification

engines. Thus, with Monasca we can gather a huge quantity of monitoring information from many different

hosts using simple agents or directly by using an API.

https://github.com/EUBra-BIGSEA/Elastic-Cluster-Testing


18


4.1 Monasca Installation

Many deployment scenarios are possible. In the following sections we show information about Monasca

installation regarding important scenarios in the context of the BIGSEA project.

4.1.1 Requirements

Monasca must be installed in a machine with some capacity restrictions and other dependencies. This is

important to guarantee high performance processing metrics and fault-tolerance.

It is strongly recommended to follow these minimal resource configurations to deploy Monasca without

compromising its performance:

- An endpoint Keystone v3;

- Machine with at least 12 GB RAM, at least 4 CPU cores (~2.3GHz) and at least 20 GB Disk.

4.2 Deployment

There are some different ways to deploy Monasca:

4.2.1 Using DevStack

This is a developer solution used mainly for contributors in OpenStack community to make changes, since

those changes are reflected instantly for tests. This is the less stable way to deploy Monasca, especially if you

are using the master version of OpenStack components and it often doesn't work well with other components

deployed together. But, if you disable other components like Ironic, Cinder, Neutron, etc., this alternative

can be simple and fast.

To deploy your DevStack is necessary to make the following steps:

1. Clone the DevStack repo:

$ git clone https://git.openstack.org/openstack-dev/devstack

2. Add the following to the DevStack local.conf file in the root of the devstack directory. You may need to

create the local.conf if it does not already exist.

# BEGIN DEVSTACK LOCAL.CONF CONTENTS [[local|localrc]] DATABASE_PASSWORD=secretdatabase RABBIT_PASSWORD=secretrabbit

ADMIN_PASSWORD=secretadmin SERVICE_PASSWORD=secretservice SERVICE_TOKEN=111222333444 LOGFILE=$DEST/logs/stack.sh.log LOGDIR=$DEST/logs LOG_COLOR=False # The following two variables allow switching between Java and Python for the

implementations # of the Monasca API and the Monasca Persister. If these variables are

not set, then the # default is to install the Python implementations of both the

Monasca API and the Monasca Persister.

https://git.openstack.org/openstack-dev/devstack


19


# Comment one of the following two lines to choose Java or Python for the Monasca API. MONASCA_API_IMPLEMENTATION_LANG=${MONASCA_API_IMPLEMENTATION_LANG:-java} MONASCA_API_IMPLEMENTATION_LANG=${MONASCA_API_IMPLEMENTATION_LANG:-python} # Comment of the following two lines to choose Java or Python for the Monasca Pesister.

MONASCA_PERSISTER_IMPLEMENTATION_LANG=${MONASCA_PERSISTER_IMPLEMENTATION_LANG:-java}

MONASCA_PERSISTER_IMPLEMENTATION_LANG=${MONASCA_PERSISTER_IMPLEMENTATION_LANG:-python} # Ccomment one of the following two lines to choose either InfluxDB or Vertica.

MONASCA_METRICS_DB=${MONASCA_METRICS_DB:-influxdb} #

MONASCA_METRICS_DB=${MONASCA_METRICS_DB:-vertica} # This line will enable all of Monasca. enable_plugin monasca-api

git://git.openstack.org/openstack/monasca-api # END DEVSTACK LOCAL.CONF CONTENTS

3. Run './stack.sh' from the root of the devstack directory.

Using Vagrant and OpenStack Provider

Vagrant is also a developer solution for Monasca deployment that provides easy configuration, reproducible,

and portable work environments. Basically it consists of playbooks to orchestrate deployments of a DevStack

integrating Monasca.

To deploy your DevStack using Vagrant is necessary to perform the following steps:

1. First step is to install Vagrant and its dependencies:

$ sudo apt-get install virtualbox $ sudo apt-get install vagrant $ sudo apt-get install virtualbox-dkms

2. To play with Monasca source code, you'll need a Devstack with Monasca environment running. To get this

environment up and running one can easily use Vagrant + OpenStack Provider

[https://github.com/ggiamarchi/vagrant-openstack-provider] using the following commands:

# Create a Vagrantfile to look like this: http://paste.openstack.org/show/545644/ $ vagrant plugin install vagrant-openstack-provider $ vagrant up --provider=openstack --debug

After you run the commands below, it is expected to have an instance in your OpenStack cloud with a

DevStack deployed integrated with a Monasca services.

https://github.com/ggiamarchi/vagrant-openstack-provider

http://paste.openstack.org/show/545644/


20


4.2.2 Using Ansible

Ansible is an IT automation tool. It can configure systems, deploy software, and orchestrate more advanced

IT tasks such as continuous deployments or zero downtime rolling updates. To deploy Monasca using Ansible

it is necessary to use a set of playbooks, that are Ansible’s configuration, deployment, and orchestration

language. They can describe a policy you want your remote systems to enforce, or a set of steps in a general

IT process. Using a simple analogy, they are just like automatized manuals that executes a sequence of steps

to configure a specific host. In this case, the goal is to set the configurations needed to deploy Monasca and

execute the deployment of Monasca’s components into the host.

There is a set of templates already done to deploy Monasca using ansible. Using it is quite simple. It is possible

to find the playbooks at the git repository (https://git.lsd.ufcg.edu.br/monag/monasca-ansible/) and perform

the following commands:

$ sudo apt-get update && sudo apt-get install git -y $ git clone https://git.lsd.ufcg.edu.br/monag/monasca-ansible.git $ cd monasca-ansible $ sudo scripts/bootstrap-ansible.sh # Add keystone and monasca IP address to the inventory file $ ansible-playbook setup-everything.yml

These Ansible playbooks can be used to install and configure the following services:

● monasca-api, used to interact with the Monasca system remotely. It is used by both the application

containers to register their status and the applications to register custom metrics.

● monasca-persister, for the storage of the monitoring data.

● monasca-notification, which consumes alarm state transition messages and sends notifications.

● monasca-thresh, which computes thresholds on metrics and publishes alarms when exceeded.

● monasca-agent (setting up default plugins), which includes monitoring probes to retrieve different

metrics.

● monitoring dashboard for monasca in OpenStack Horizon.

4.3 Monasca basics

4.3.1 Metrics

Metrics are the core of the monitoring services. They can explain a lot about the application, the service or

the host we are interested in monitoring. But it is not always easy to use the metrics in the best possible way

and for this reason it is necessary to filter the most relevant information about the monitored targets.

Monasca has a high-performance analytics database that can store massive amounts of metrics in real-time

and can also support interactive queries. Currently, Vertica and InfluxDB are supported. The metrics can be

created in Monasca using agents or directly using the API. The metric is identified by its name and other

characteristics called dimensions, following these examples:

https://git.lsd.ufcg.edu.br/monag/monasca-ansible/


21


name dimensions

mem.swap_free_mb hostname: c4-compute11

service: monitoring

mem.swap_free_mb hostname: c4-compute12

service: monitoring device: em1

load.avg_15_min hostname: c4-compute12

appliance_errors

When the information represented by the metric is actually retrieved from the monitored source, we have a

set of measurements like the examples below:

name dimensions measurement_id timestamp value value_meta

metric1 instance_id:123

service:ourservice 8732 2016-05-

08T21:46:4

7Z

10.2

metric1 instance_id:123

service:ourservice 8733 2016-05-

08T06:56:3

2Z

1

metric2 8734 2016-05-

09T23:21:0

8Z

1.3 rc:123

It is up to the agent developer to define the name and dimensions of the metrics and how they are filled

when a new measurement comes up. For instance, the developer defines that there is one dimension called

hostname and that it must be filled with the hostname of the compute node from which the measurement

was sent.

4.3.2 Agent

Any resource can be monitored, providing information that can be used by other services to take actions. To

have that information available for other services you need to have an agent that is capable of handling all

the information of interest from each resource in terms of Monasca metrics. The monitoring agents can

gather the information from resources using the logs or an API provided by the resource, all the information

is transformed in metrics to be send to the Monasca API and those metrics can be retrieved by any service.


22


Figure 5. Monitoring Cycle

How to create a new agent to monitor an application

In order to monitor your application, you need to create an agent that will be responsible for collecting all

measurements that the resource can provide via API, logs or anything else. Also your agent must be able to

publish all the metric measurements to Monasca. There are some ways that you can use to publish metrics

to Monasca.

1. Using the Monasca CLI

Dependencies: At the machine your agent is running, you need to install python-monascaclient.

Usage:

monasca metric-create <metric-name> <value> --dimensions

<dimension1=value1>,<dimension2=value2>

$ monasca metric-create mymetric 0.1 --dimensions service=compute

2. Using the Monasca python API

Dependencies: Same dependencies for the client should apply here.

Example: Publishing metric using Python API.

# Copyright (c) 2014 Hewlett-Packard Development Company, L.P. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License.

"""An example using monascaclient via the Python API""" from monascaclient import client import monascaclient.exc as exc import time

# In order to use the python api directly, you must first obtain an

http://docs.openstack.org/cli-reference/monasca.html

https://github.com/openstack/python-monascaclient#python-api


23


# auth token and identify which endpoint you wish to speak to. endpoint = 'http://192.168.10.4:8070/v2.0'

# The api version of monasca-api api_version = '2_0'

# Pass in the keystone authentication kwargs to construct a monasca client. # The monasca_client will try to authenticate with keystone one time # when it sees a 401 unauthorized resp, to take care of a stale token. # In this example no token is input, so it will get a 401 when executing the # first metrics.create request, and will authenticate and try again. auth_kwargs = {'username': 'mini-mon', 'password': 'password', 'project_name': 'mini-mon', 'auth_url': 'http://localhost:35357/v3/'} monasca_client = client.Client(api_version, endpoint, **auth_kwargs)

# post a metric dimensions = {'instance_id': '12345', 'service': 'nova'} fields = {} fields['name'] = 'metric1' fields['dimensions'] = dimensions # time in milliseconds fields['timestamp'] = time.time() * 1000 fields['value'] = 222.333 try: resp = monasca_client.metrics.create(**fields) except exc.HTTPException as he: print('HTTPException code=%s message=%s' % (he.code, he.message)) else: print(resp) print('Successfully created metric')

3. Using Monasca API

To publish metrics using the Monasca API you need to provide at least the following arguments:

● Headers:

○ X-Auth-Token → Keystone token

○ Content-Type → application/json

● Request Body:

○ name → the name of the metric.

○ timestamp → the timestamp in milliseconds.

○ value → the value of the metric.

Dependencies: At the machine your agent is running make sure you can access the

Keystone and Monasca endpoints.

POST /v2.0/metrics HTTP/1.1

Host: 192.168.10.4:8070

Content-Type: application/json

https://github.com/openstack/monasca-api/blob/master/docs/monasca-api-spec.md#create-metric


24


X-Auth-Token: 27feed73a0ce4138934e30d619b415b0

Cache-Control: no-cache

{

"name":"spark.application_progress",

"dimensions":{

"application_id":"app-12345",

"service":"spark"

},

"timestamp":1405630174123,

"value":0.1

}

4. Using Monasca Plugin:

Plugins are the way to extend the Monasca Agent. They add extra functionality, which allows the

Monasca agent to perform checks on other applications, servers or services. Some plugins may have

corresponding Detection Plugins to automatically detect, configure, and activate certain Agent

plugins.

The location where the plugin must be installed depends on how the information gathered by the plugin is

obtained. For instance, if the application exposes a REST API and the plugin uses this API to get information

from the application, then, the plugin does not need to be installed in the same machine as the application.

It can be installed in any machine where there is an agent and access granted to the application REST API.

The plugin must be properly configured to know the addresses of the applications monitored as well as the

Monasca API. On other situation, the plugin may need to be installed in the same machine as the application,

to gather information that is more sensitive, and in this case the agent may be also needed to publish the

metrics.

Dependencies: At the machine your Plugin is installed and running you only need to install monasca-

agent.

a. Custom Check Plugins: This type of plugin will monitor the specified service in all hosts

according to the configuration in the yaml file. The documentation on how to write Check

Plugin can be found here.

Example: Agent to collect the temperature of the host using ipmitool.

ipmi_temperature.py

import subprocess import time

from monasca_agent.collector.checks import AgentCheck

class IPMICheck(AgentCheck):

def check(self, instance): dimensions = self._set_dimensions(None, instance)

https://github.com/openstack/monasca-agent/blob/master/docs/Customizations.md#creating-custom-plugins


25


# req_args = { ‘address': instance.get('address'), 'username': instance.get('username'), 'password': instance.get('password') } ambient_temperature = int(self.get_temperature(**req_args)) self.gauge(‘ipmi.ambient.temperature’, ambient_temperature, dimensions=dimensions, device_name=’idrac’)

def get_temperature(self, address, username, password): return subprocess.check_output(“ipmitool -H “ + address + “ -U “ + username + “ -P

“ + password + “ sdr type Temperature | grep 0Eh | awk ‘{print $10}’”, shell=True).rstrip()

Ipmi Check Plugin Configuration YAML (ipmi_temperature.yaml)

init_config: null instances: - name: "danio" address: "10.15.0.2" username: "<idrac_user>" password: "<idrac_user_password>"

- name: “localhost”

address: “127.0.0.1” username: “<idrac_user>” Password: “<idrac_user_password>”

b. Custom Detection Plugin: This type of plugin is responsible for auto-detecting services in the

host that the monasca-agent is being installed, if successful it will create the configuration

file for the related Check Plugin. The implementation of this plugin should be

/usr/lib/monasca/agent/custom_detect.d and you need to extend the Plugin Class and

implement three methods: _detect(self), build_config(self) and

dependencies_installed(self). The documentation on how to write the Detection Plugin can

be found here.

# (C) Copyright 2016 Hewlett Packard Enterprise Development Company LP

#

# Licensed under the Apache License, Version 2.0 (the "License"); you may

# not use this file except in compliance with the License. You may obtain

# a copy of the License at

#

# http://www.apache.org/licenses/LICENSE-2.0

#

# Unless required by applicable law or agreed to in writing, software

# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT

# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the

# License for the specific language governing permissions and limitations

# under the License.

import logging

https://github.com/openstack/monasca-agent/blob/master/docs/Customizations.md#creating-a-custom-detection-plugin


26


import os

import yaml

import monasca_setup.agent_config

import monasca_setup.detection

log = logging.getLogger(__name__)

class Bind(monasca_setup.detection.Plugin):

"""Detect Bind daemon and setup configuration to monitor it.

"""

def _detect(self):

"""Run detection, set self.available True if the service is detected.

"""

if monasca_setup.detection.find_process_cmdline('named') is not None:

self.available = True

def build_config(self):

"""Build the config and return.

"""

log.info("\tWatching the named process.")

return monasca_setup.detection.watch_process(

['named'], 'bind', exact_match=False)

def dependencies_installed(self):

return True

4.4 Monasca Probes

In the context of BigSea, it is important to monitor some aspects of the applications running on top of the

infrastructure.

How to monitor the progress of a Spark application?

The Spark API can provide the information about the progress of a running application using the endpoint

‘api/v1/applications/[app-id]/jobs’. The Agent sends requests to the Spark API and extracts the progress from

the json received to publish the progress to the Monasca API.

SparkMonitoring Agent

This agent is started when we start any Spark application, we get the job progress and elapsed time in

percentage to publish into Monasca using the ManagerMonasca.


27


import requests import time import tzlocal import pytz

from ManagerMonasca import *

EXPECTED_TIME=100

class SparkMonitoring():

def __init__(self, spark_submisson_url, monitoring_info): self.submission_url = spark_submisson_url self.monasca = ManagerMonasca(monitoring_info['auth_url'], monitoring_info['username'], monitoring_info['password'], monitoring_info['project_name'], monitoring_info['api_version'])

def getRunningApp(self): try: all_app = requests.get(self.submission_url + '/api/v1/applications?status=running') for app in all_app.json(): if app['attempts'][0]['completed'] == False: return app['id'],app['name'] return None except: print "No application found" return None

def get_elapsed_time(self, gmt_timestamp): local_tz = tzlocal.get_localzone() date_time = datetime.strptime(gmt_timestamp, '%Y-%m-%dT%H:%M:%S.%fGMT') date_time = date_time.replace(tzinfo=pytz.utc).astimezone(local_tz) elapsed_time = datetime.now() - date_time.replace(tzinfo=None) print 'Elapsed time is %s' % elapsed_time.seconds return elapsed_time.seconds

def getSingleJobProgress(self): # try to get application running app = None for i in range(0,50): print 'Attempt %s' % i application = self.getRunningApp() if application is not None: app = application break time.sleep(5)

if app is not None: app_id = app[0]


28


while True: try: job_request =

requests.get(self.submission_url+'/api/v1/applications/'+app_id+'/jobs') job_progress = {} time_progress = {} for result in job_request.json(): timestamp = time.time() * 1000

job_progress['name'] = 'spark.job_progress' job_progress['value'] =

result['numCompletedTasks']/float(result['numTasks']) job_progress['timestamp'] = timestamp job_progress['dimensions'] = {'service': 'spark', 'application_id': app_id}

time_progress['name'] = 'spark.elapsed_time' time_progress['value'] =

self.get_elapsed_time(result['submissionTime'])/float(EXPECTED_TIME) time_progress['timestamp'] = timestamp time_progress['dimensions'] = {'service': 'spark', 'application_id': app_id} self.monasca.send_metrics([job_progress, time_progress]) except Exception as ex: print ex.message

Monasca Manager

It is the component that allows the connection with Monasca and publishes all metrics.

from monascaclient import client as monclient, ksclient

import monascaclient.exc as exc

class ManagerMonasca():

def __init__(self, auth_url, username, password, project_name, api_version): self.auth_url = auth_url self.username = username self.password = password self.project_name = project_name self.api_version = api_version

def _get_monasca_client(self):

# Authenticate to Keystone ks = ksclient.KSClient( auth_url=self.auth_url, username=self.username, password=self.password, project_name=self.project_name


29


) # Monasca Client monasca_client = monclient.Client( self.api_version, ks.monasca_url, token=ks.token ) return monasca_client

def send_metrics(self, measurements): batch_metrics = {'jsonbody': measurements} try: monasca_client = self._get_monasca_client() monasca_client.metrics.create(**batch_metrics) except exc.HTTPException as httpex: print httpex.message except Exception as ex: print ex.message

4.4.1 How to monitor the progress of a batch application?

There are different ways of monitoring the progress of an application, and it depends on the level of

instrumentation of the application. Ideally, the applications are built on top of well-known frameworks, like

Spark, for example. In this case, the framework offers a REST API that allows external processes to gather

specific information about the running application, including its progress. For other frameworks that does

not provide APIs, the progress can be registered directly by the end-user’s application. Other way is to get

this information by reading the specific logs of the application from which this information can be inferred.

4.5 Infrastructure instrumentation for vertical scaling

The scaling method we are developing is based on two components: a Scaler and an Actuator.

Figure 6. Scaling Method


30


4.5.1 Scaler

The Scaler is responsible for deciding what changes in the infrastructure must be made so that the application

finishes in time. It gets information on application progress and elapsed time using Monasca’s API. Then, it

uses this information to decide the best action to take.

Many scaling policies are possible. An example follows:

If application progress > elapsed time

Then scale down - remove resources (lower CPU cap value)

If application progress < used time

Then scale up - add resources (increase CPU cap value or add CPUs)

Currently the Scaler communicates directly with the Actuator. One possibility of decoupling is to publish the

chosen action on Monasca, from which other components can read.

4.5.2 Actuator

The Actuator is responsible for making the changes requested by the Scaler. It can change CPU cap or add

CPUs to an instance.

1. Change CPU cap

Currently we use ssh to access the compute node and change the cap of the desired instance. One

possibility is to run the Actuator on the compute node and get the new cap value from Monasca.

2. Add CPUs

It requires the update of the number of CPUs assigned to the instance (done outside the VM) and

to enable the new CPUs (done inside the VM). It is not possible to remove CPUs from active

instances.

Currently we do the update using virsh and enable the new CPUs using a cron script. One possibility

using Monasca is to make both “update” and “enable” scripts get from Monasca the new number

of CPUs to use.

5 SECURITY MODEL

The registration of a framework is secured by means of a principal and a secret. Both keys are stored in the

mesos.radl template, in a configuration block called “framework_vars”. You should change the values of

“principal” and “secret” to the values you consider. Please refer to section 3.3.2 for obtaining or changing

the access credentials.


31


Any framework connecting to Mesos must provide this principal and secret. By default, Chronos and

Marathon are properly configured, as well as Spark frameworks.

- Marathon: Provide the options --mesos_authentication --mesos_authentication_secret_file path_to_secret --

mesos_authentication_principal yourprincipal

- Chronos: Provide the options --mesos_authentication_secret_file path_to_secret --mesos_authentication_principal

yourprincipal

- Spark: Provide the options --conf spark.mesos.principal=yourprincipal --conf spark.mesos.secret=yoursecret

Where the path_to_secret is a file that contains (without a carriage return) the secret (yoursecret) for

the principal indicated.

The access credentials for the users in the system are listed in the file bigseausers.radl. This file can be easily

updated by updating/adding entries to the file in the following way:

- { name: 'username', uid: 'uid', password: 'userpass' }

The ‘userpass’ is generated using mkpasswd, which provides an encrypted password you can safely include

in the file.

Similarly, you have to update the rest of the configuration (ssh_keys generation and copy to

authorized_keys).

6 CONCLUSIONS

This document describes the deployment and access to the BIGSEA cluster that supports the execution of the

jobs. The cluster provides self-scaling, QoS monitoring and automated deployment of the whole

infrastructure with one single click.

With the information provided in the document, you can deploy the whole infrastructure in your preferred

cloud IaaS and execute jobs embedded in containers or Spark jobs.

The cluster will be extended with the guidelines to provide vertical elasticity, both at the framework and at

the hardware levels.

7 GLOSSARY

Acronym Explanation Usage Scope

Chronos A fault tolerant job scheduler for Mesos which handles

dependencies and ISO8601 based schedules Scheduler

CLUES Cluster Energy Saving for hpc and cloud computing Elastic Resource Management Framework


32


EC3 Elastic Compute Clusters in the Cloud Elastic Resource

Management Framework

IM Infrastructure Manager Contextualization engine

Marathon A container orchestration platform for Mesos Scheduler

Mesos A distributed systems kernel Resource Management Framework

Monasca Monasca is a open-source multi-tenant, highly scalable, performant, fault-tolerant monitoring-as-a-service solution

Monitoring service

ONE OpenNebula Cloud Management Platform

OpenStak OpenStack cloud management platform Cloud Management Platform

Spark Apache Spark™ is a fast and general engine for large-scale data processing.

Data Analytics programming kernel

D3.3: BIGSEA QoS infrastructure services initial version · 2017. 5. 23. · Mesos/Marathon/Chronos...

Documents

Transcript of D3.3: BIGSEA QoS infrastructure services initial version · 2017. 5. 23. · Mesos/Marathon/Chronos...