OpenShift PaaS Anywhere (Infrastructure.Next Ghent 2014-02-24) Diane Mueller
Deploying & Scaling OpenShift on OpenStack using Heat - OpenStack Seattle MeetUp 2014-01-23 by Diane...
-
Upload
openshift-origin -
Category
Technology
-
view
6.393 -
download
0
Transcript of Deploying & Scaling OpenShift on OpenStack using Heat - OpenStack Seattle MeetUp 2014-01-23 by Diane...
Diane MuellerOpenShift Origin Community ManagerJanuary 2014OpenStack Meetup Seattle
Deploying OpenShiftOn OpenStack
@pythondj
noun \p-thn, -thn\+\d-j\
Pythona widely used general-purpose, high-level programming ............languageSnakea very large snake that kills the animals it eats by wrapping itself around them
+ short for Djangoa high-level Python Web framework that encourages rapid development & clean design+ Disk Jockeya person who plays popular recorded music on the radio or at a party or nightclub
Red Hat's Cloud Ecosystem Evangelist
A Snake Charmer
Agenda
Learn a little about PaaS & OpenShift
Learn a little about underlying PaaS architecture
Learn how to deploy OpenShift on OpenStack
Talk about the Future of PaaS on OpenStack
Find out where to learn more
origin
Public Cloud ServiceOn-premise or Private Cloud SoftwareOpen Source Project
FLAVORS OF OPENSHIFT
What if you could use existing infractructure (bare metal or virtualized), private cloud, and public cloud in one, unified environment?
Open hybrid cloud lets you take advantage of your existing resources, and be open to new advances. Build a cloud infrastructure thats easy to integrate, agile, and future-proof.
UNIVERSAL (UNIFIED?) ENVIRONMENT: An environment that is ready any time, all the time. Because its built with technology that is standardized, interoperable, and modular.
This allows you to reuse the infrastructure you have to build more agile enterprise I.T.
OpenStack
OpenShiftPutting the PaaS in OpenStack
Cross Community Collaboration
A bit about Heat: The Heat API implements the AWS Cloud Formations API. This API provides a rest interface for creating composite VMs called Stacks from template files. The goal of the software is to be able to accurately launch AWS CloudFormation Stacks on OpenStack. We will also enable good quality high availability based upon the technologies we created in Pacemaker Cloud including escalation.
http://www.ohloh.net/
Why I love PaaS: It's Magic
Infrastructure LayerSaaS/Applications LayerInfrastructure LayerInfrastructure Layer
IBM 1401 the year I was born http://www.computerhistory.org/
First Job http://www.computerhistory.org/
SOFTWARE IS EATING THE WORLD.
Circa 1989 Predator vs. Prey
1989Prey: Meg Ryan boutique book store Predator: Tom Hanks - Fox as a Stand in for Barnes & Noble
That's Disruptive Technology at Work
http://www.theguardian.com/media-network/media-network-blog/2013/nov/18/werner-vogels-cloud-trends-amazon
4 trends according to Werner
Cloud will enable your content to follow you wherever you goCloud has changed how we interact with mobile devices. In the past content would be moved to the device, now devices are just a window to content and services that live in the cloud. This started with our smartphones and tablets, where regardless of which device we use, or the location, we have access to our content and subscriptions.
Now this approach is migrating to non-mobile devices such as Samsung smart TV's. The devices are beautifully designed and beautifully built, but the core functionally of these television sets is software connected to services running in the cloud.
This is also moving beyond traditional devices, for example my car is already connected to my Amazon Cloudplayer giving me music everywhere I go. I have seen the first treadmills where the moment I step on them they reconfigure to give access to my music and videos, my newspaper subscriptions and books, but also my documents in services like Dropbox. I no longer need to bring my content; cloud enables my content to follow me wherever I go.
Cloud based analytics enhances the offline worldThe cloud is already the place where researchers collaborate on data that flows in real-time from devices such as the Mars rover or the Ilumina DNA sequencer into cloud storage. In 2014 expect an explosion in data generation by real-world devices and where that data is stored, analysed and shared in the cloud.
For example we will see a rise in the industrial cloud where industrial environments are equipped with sensors producing data to improve efficiency and reliability. An example is the project we run with GE on instrumenting their gas turbines or with Shell where they are going to drop sensors in their oil wells that generate petabytes of data.
Also in our daily lives we will see the rise of cloud connected sensors and devices such as the Nest Thermostat or the home control applications built by energy companies like Essent. Around the world public transport companies are instrumenting their busses and trams with sensors that feed into platforms like OneBusAway that can give real-time updates to travellers.
Passengers themselves can also become sensors: services like Mooveit use the information from an application on passengers' phones to give real time transport information in the same way that Waze does for cars.
The cloud allows everyone to become a media companyIn 2014 expect a great rise in organisations that are adding media capabilities to their offerings. A good example is sports clubs; all are looking for ways to establish an engagement with their fan base beyond the two hours on a weekend. A successful way to achieve a weeklong engagement is by daily distribution or fresh, exclusive media content. The subscription revenues for clubs that often have millions of fans around the world are substantial.
Cloud based services for pre and post production, as well as distribution, are readily available such that anyone can become an internet broadcaster operating worldwide without any capital investment. A well-known case is that of the Amazon Web Services (AWS) powered LiverpoolTV, but every football club worldwide is following their example.
Another very popular case is that of performing arts organisations, from orchestras to theatre companies, which give exclusive access to their performances through cloud-based media production. This way they are able to reach a much larger audience, which would often not be able to attend their performances in person. It extends their revenue potential, which is needed in times where arts subsidies are disappearing. A good example is Berliner Philharmoniker, the world famous orchestra that gives access to their live performances through the digitalconcerthall.com that makes use of all AWS regions around the world to provide a high quality media experience.
Faster and faster, cloud moves data processing to real-timeUp until this point big data has very much focused on looking historically - people who brought product X also brought product Y, the market moved in this direction last week so is likely to move in that direction now. There has always been a close relationship between big data and cloud computing as it requires no limits in terms of compute and storage but by adding real-time processing capabilities, we will see a rise in data analytics that are able to produce results for in real-time, radically changing the products companies can build.
For example we see companies with real time recommendations, in the form of 'other people in your network are reading X'. Some of the frontrunners here are the companies working on second-screen technologies, such as Channel 4, that make use of real-time data to power the information they present to augment TV watching. A company like Netflix that processes over 40bn events a day uses real-time analytics to power their operations, their customer engagement and their business metrics.
We see almost every industry taking advantage of the cloud to radically improve the speed at which they can process their data.
Werner Vogels is vice president and CTO at Amazon.com
App Development Got Complicated
How to Build an App
Old School
For either the Entrepreneurial or the Enterprise Developer, PaaS is the way of the future.
Let's take a quick look at the before-and-after of the application development process.
In the old days, when you wanted to build a new app (or were assigned a project to build a new app), you had to jump through a million hoops to get it up and running. Everything from ordering hardware, to installing middleware, to tuning and testing every facet of the development environment.
With PaaS, life is much easier. You have an idea for an application? You just start writing the code and let OpenShift PaaS handle the rest.
Write your code, Push to OpenShift, Test with Jenkins, and Deploy when ready!
Even Gartner knows that PaaS will be the way of the future for application development.
Unhappy Developers
Then along came Cloud
Developer Expectations
Platform Flexibility
Application Portability
Increased Productivity
Infrastructure is not enough
Infrastructure as a Service gives you
Network, storage & compute as an on-demand service
Basically, servers in the cloud
Youre still on the hook to configure & manage the cloud & stack
How do I use this?
So, what you need is the ease of use and access of a SaaS application, but you need it with your purpose-built, mission-critical, applications.
PaaS gives you just that. It allows you to quickly and easily build the application that YOU need. Whether this is for your group, your enterprise, or your next BIG IDEA, you can build it and launch your specific code on a PaaS and not have to deal with the underlying infrastructure, middleware, and management headaches.
Because of the built-in auto-scaling and elasticity provided by the PaaS infrastructure, PaaS's are ideal for modern data-hungry Big Data, Mobile, and Social applications.
With a PaaS, you can focus on what you should be focused on... your application code.
And let the Cloud provide what it is suppose to: Ease, Scale and Power
Software as a Service gives you
An on-demand application
Nothing to install or configure
This is all my customers and users care about!
So, what you need is the ease of use and access of a SaaS application, but you need it with your purpose-built, mission-critical, applications.
PaaS gives you just that. It allows you to quickly and easily build the application that YOU need. Whether this is for your group, your enterprise, or your next BIG IDEA, you can build it and launch your specific code on a PaaS and not have to deal with the underlying infrastructure, middleware, and management headaches.
Because of the built-in auto-scaling and elasticity provided by the PaaS infrastructure, PaaS's are ideal for modern data-hungry Big Data, Mobile, and Social applications.
With a PaaS, you can focus on what you should be focused on... your application code.
And let the Cloud provide what it is suppose to: Ease, Scale and Power
Platform as a Service delivers
Application run-time environment in the cloud
Configures & manages both the cloud & stack for your application
The cloud is now useful!
So, what you need is the ease of use and access of a SaaS application, but you need it with your purpose-built, mission-critical, applications.
PaaS gives you just that. It allows you to quickly and easily build the application that YOU need. Whether this is for your group, your enterprise, or your next BIG IDEA, you can build it and launch your specific code on a PaaS and not have to deal with the underlying infrastructure, middleware, and management headaches.
Because of the built-in auto-scaling and elasticity provided by the PaaS infrastructure, PaaS's are ideal for modern data-hungry Big Data, Mobile, and Social applications.
With a PaaS, you can focus on what you should be focused on... your application code.
And let the Cloud provide what it is suppose to: Ease, Scale and Power
Infrastructure LayerSaaS/Applications LayerInfrastructure LayerInfrastructure Layer
Key Terms for Developers
Applications your web code and any data store. Has to be on 1 or more gears
Gears a resource-constrained container that runs one or more user-specified software stacks, also known as cartridges. Each gear has a limited amount of RAM and disk space. If an application needs more resources, it can be configured to automatically use more gears.
Cartridges A technology/framework (PHP, Perl, Java/JEE, Ruby, Python, MySQL, etc.) to build applications
OpenShift Online (http://openshift.com)
OpenShift Origin on Github
http://openshift.github.io/
OpenShift Enterprise
ENTERPRISE IT MANAGEDLet developers have the benefits of self-service PaaS while IT retains control for governance and compliance in a Private or Hybrid Cloud
OpenShift Enterprise is designed to be deployed on top of and run on Red Hat Enterprise Linux (RHEL).
What makes OpenShift different?
RHEL Platform Support
SELinux-based Secure Containers for multi-tenancy
Extensible Architecture
System Component Redundancy for High Availability
Automatic Application Stack Provisioning
Support for Java EE 6
Choice of Cloud Infrastructure, Bare Metal, or Desktop
Configurable Deployment to Support Enterprise Requirements
Automatic Application Scaling
Empowering Developersto test, launch, fail, iterate, scale
RAPIDLY
https://install.openshift.com/
Deploying OpenShift
Origin Release 3
Fedora 19 or RHEL 6.x or CentOS 6.5
Get up and running
Vagrant
Puppet
Comprehensive guide
Ansible
Heat
http://openshift.github.io
Amazon EC2
Rackspace
Bare Metal
OpenStack
RHEV
CloudStack
OpenShift PaaS
OpenShift
Runs on IaaS
Platform agnosticSELinux
OpenShift Machines
Node
Broker
Broker
Broker
Node
Nodes are where User applications live.
An OpenShift Broker can manage multiple nodes
HIgh level - 2 types of machines
OpenShift Architecture
Broker
OpenShift Machines
Node
Broker
Broker
Broker
Node
Nodes are where User applications live.
An OpenShift Broker can manage multiple nodes
Applications and Gears
Gear
Node
Gear
Jboss
MariaDB
Jboss
Gear
Node
Gear
User application
Key OpenShift Terms
Broker Management host, orchestration of Nodes
Node Compute host containing Gears
Gear Allocation of fixed memory, compute, and storage resources for running applications
Cartridge A technology/framework (PHP, Perl, Java/JEE, Ruby, Python, MySQL, etc.) to build applications
Architecture
https://install.openshift.com/
Deploying on OpenStack with Heat
Heat Overview
Entering OpenStack Integrated status in November 2013Active code base3048 commits as of September 2013
56 contributors
Cross Project functionality with OpenStack projects Keystone, Nova, Neutron, Cinder, Ceilometer, Swift, Glance, Horizon, TripleO and Tempest
OpenStack Heat provides application autoscaling today with a stable workflow model
OpenShift on OpenStack OpenShift Enterprise Templates for RHEL ready
OpenShift Origin Templates for CentOS & Fedora in progress
Heat's Mission: Orchestration
To explicitly model the relationships between OpenStack resources of all kinds; and to harness those models, expressed in forms accessible to both humans and machines, to manage infrastructure resources throughout the life-cycle of applications.
Heat Overview
Nova
ComputeNodeGlance
ImageServiceSwift
ObjectStoreCinder
VolumeServiceKeystone
IdentityService
HorizonDashboardQuantum
NetworkingHeatOrchestrationProvides AWS Cloudformation and native ReST API
Abstract configuration of services to single-template
HA/Autoscaling/Monitoring features
Openstack integrated project
KEY POINTS Heat provides a way of defining all your openstack resources / services from a single template
Ties together the underlying IAAS services, so you can provide a consolidated infrastructure service, single integration point
Compatible with AWS Cloudformation (templates and API)
Has Autoscaling and HA features
Recently became an incubated project
OpenStack Heat Architecture
Heat API
KEY POINTSLifecycle operationsCreate creates a stack from a template
Delete deletes an existing instantiated stack
Update updates an existing instantiated stack
Introspection operations List obtains a list of existing stacks
Describe obtains details of existing stacks
Events List obtains events that have occurred for a stack
List resources lists resources associated with a stack
Template describes a stack
OpenStack Heat Engine Architecture
Autoscaling
Metrics or user events drive scaling
Metrics can include CPU utilization, memory utilization, many more as well as custom dimensions
Dynamically add and reduce OS::Nova::Server resources to meet demand
Front end Neutron LBAAS or Heat provided HA-Proxy Load Balancer distributes load to server resources
Autoscaling Workflow Internal View
User instantiates template with Heat's CLI
Heat registers with Ceilometer for callbacks on Alarm events
Ceilometer tells Heat about Alarm events and Heat scales a Group based upon a Policy decision to scale up or down
OS::Nova::Server instances can also call Alarms internally
OpenShift on OpenStack Autoscaling Workflow
http://github.com/openstack/heat-templates
Reasons to precreate images:
Performance of scaling availability
Reliability of not loading at runtime image contents
Demo
Deploying OpenShift Enterprise With Heat Templates OnOpenStack
Learn more about Heat & OpenShift:
Users, testers and developers wanted!Connect via IRC on #openshift-dev@freenode
Check out the repositories:https://github.com/openstack/heathttps://github.com/openstack/heat-templateshttps://github.com/openshift
Read the Heat Documentation:http://docs.openstack.org/developer/heat
Read the OpenShift Documentation:http://openshift.github.io
Future of PaaS
https://github.com/openshift/openshift-pep/blob/master/openshift-pep-010-docker-cartridges.mdhttp://docs.docker.io/en/latest/installation/kernel/
Future of PaaS on OpenStack
Solum initiativealso http://solum.io
Current effortsConnecting Git to SolumVia project zuul (current OS tool for testing infrastructure)
Building Images that can be runDocker (normal image w/ docker pre-installed)
Normal Nova Images
OpenShift
AutoScaling Workflow
Step 1
Create DIB elements
Elements directory structure
elements/openshift-origin-broker:-rw-rw-r--. 1 sdake sdake 37 Jun 2 12:14 element-depsdrwxrwxr-x. 2 sdake sdake 4096 Jun 2 12:14 install.d-rw-rw-r--. 1 sdake sdake 176 Jun 2 12:14 README.md
elements/openshift-origin-broker/install.d:-rwxrwxr-x. 1 sdake sdake 1598 Jun 2 12:14 30-openshift-origin-broker
elements/openshift-origin-node:-rw-rw-r--. 1 sdake sdake 37 Jun 2 12:14 element-depsdrwxrwxr-x. 2 sdake sdake 4096 Jun 2 12:14 install.d-rw-rw-r--. 1 sdake sdake 172 Jun 2 12:14 README.md
elements/openshift-origin-node/install.d:-rwxrwxr-x. 1 sdake sdake 1610 Jun 2 12:14 30-openshift-origin-node
elements/openshift-origin-repos:-rw-rw-r--. 1 sdake sdake 23 Jun 2 12:14 element-depsdrwxrwxr-x. 2 sdake sdake 4096 Jun 2 12:14 pre-install.d-rw-rw-r--. 1 sdake sdake 176 Jun 2 12:14 README.md
elements/openshift-origin-repos/pre-install.d:-rwxrwxr-x. 1 sdake sdake 286 Jun 2 12:14 29-puppetlabs-release-rwxrwxr-x. 1 sdake sdake 648 Jun 2 12:14 30-openshift-origin-repos
OpenShift Autoscaling Workflow
Step 1: Create DIB Elements
Building the broker image
[sdake@freedom openshift-origin-broker]$ more element-depsopenshift-origin-repos
[sdake@freedom openshift-origin-repos]$ ls -l pre-install.d-rwxrwxr-x. 1 sdake sdake 286 Jun 2 12:14 29-puppetlabs-release-rwxrwxr-x. 1 sdake sdake 648 Jun 2 12:14 30-openshift-origin-repos
Part 1: Parse Dependencies
Part 2: Load Dependencies
Part 3: Configure Broker
[sdake@freedom openshift-origin-broker]$ ls -l install.d-rwxrwxr-x. 1 sdake sdake 1598 Jun 2 12:14 30-openshift-origin-broker
OpenShift Autoscaling Workflow
Step 1: Create DIB elements
Contents of 30-openshift-origin-broker
[sdake@freedom install.d]$ more 30-openshift-origin-broker#!/bin/bash
set -uex
install-packages \ openshift-origin-broker \ rubygem-openshift-origin-msg-broker-mcollective \ rubygem-openshift-origin-dns-nsupdate \ rubygem-openshift-origin-dns-bind \ rubygem-openshift-origin-controller \ openshift-origin-broker-util \ rubygem-passenger \ mod_passenger \ openssh \ rubygem-openshift-origin-auth-mongo \ rubygem-openshift-origin-remote-user \ rubygem-openshift-origin-console \ openshift-origin-console \ mongodb \ mongodb-server \ bind \ bind-utils \ ntpdate \ policycoreutils \ mcollective \ httpd \ openssh-server \ rhc \ activemq \ activemq-client \ git \ puppet \ ruby \ ruby-devel \ ruby-irb \
ruby-libs \ tar \ yum-plugin-priorities \ mysql-devel \ mongodb-devel \ system-config-firewall-base \ rubygem-execjs \ rubygem-uglifier \ rubygem-listen \ rubygem-sass \ rubygem-sass-rails \ autogen-libopts \ ntp \ rubygem-coffee-script-source \ rubygem-coffee-script \ rubygem-coffee-rails \ rubygem-idn \ rubygem-addressable \ rubygem-crack \ rubygem-webmock \ rubygem-fakefs \ rubygem-chunky_png \ rubygem-hpricot \ rubygem-haml \ rubygem-fssm \ rubygem-compass \ rubygem-compass-rails \ rubygem-mongo \ rubygem-jquery-rails \ rubygem-openshift-origin-dns-avahi \ rubygem-ref \ rubygem-therubyracer
sed --in-place -e \ s/Type=oneshot/"Type=oneshot\nTimeoutSec=0"/ /lib/systemd/system/cloud-final.service
OpenShift Autoscaling Workflow
Step 2: Create Heat Template - Policy
resources: OpenshiftUser: Type: AWS::IAM::User OpenshiftOriginKeys: Type: AWS::IAM::AccessKey Properties: UserName: Ref: OpenshiftUser OpenshiftOriginNodeGroup: Type: AWS::AutoScaling::AutoScalingGroup DependsOn: BrokerWaitCondition Properties: AvailabilityZones: [] LaunchConfigurationName: Ref: NodeLaunchConfig MinSize: Ref: NodeCountMinimum MaxSize: Ref: NodeCountMaximum LoadBalancerNames: [] OpenshiftOriginScaleUpPolicy: Type: AWS::AutoScaling::ScalingPolicy Properties: AdjustmentType: ChangeInCapacity AutoScalingGroupName: Ref: OpenshiftOriginNodeGroup Cooldown: '120' ScalingAdjustment: '1'
OpenshiftOriginScaleDownPolicy: Type: AWS::AutoScaling::ScalingPolicy Properties: AdjustmentType: ChangeInCapacity AutoScalingGroupName: Ref: OpenshiftOriginNodeGroup Cooldown: '60' ScalingAdjustment: '-1'
AlarmPolicyGroup
OpenShift Autoscaling Workflow
Step 2: Create Heat Template - Alarms
NodeScaleUp: Type: AWS::CloudWatch::Alarm Properties: AlarmDescription: Scale-up if event received from broker MetricName: Heartbeat Namespace: system/linux Statistic: SampleCount Period: '60' EvaluationPeriods: '1' Threshold: '0' AlarmActions: [{Ref: OpenshiftOriginScaleUpPolicy}] Dimensions: - Name: AutoScalingGroupName Value: Ref: OpenshiftOriginNodeGroup ComparisonOperator: GreaterThanThreshold
AlarmPolicyGroup
NodeScaleDown: Type: AWS::CloudWatch::Alarm Properties: AlarmDescription: Scale-down if event received from broker MetricName: Heartbeat Namespace: system/linux Statistic: SampleCount Period: '60' EvaluationPeriods: '1' Threshold: '0' AlarmActions: [{Ref: OpenshiftOriginScaleDownPolicy}] Dimensions: - Name: AutoScalingGroupName Value: Ref: OpenshiftOriginNodeGroup ComparisonOperator: GreaterThanThreshold
OpenShift Autoscaling Workflow
Step 2:Create Heat Template
Optionally Trigger Alarms
cat /etc/heat/notify-scale-up#!/bin/bash/opt/aws/bin/cfn-push-stats --credential-file /etc/heat/heat-credentials --heartbeat watch {Ref: NodeScaleUp}EOFchmod 0700 /etc/heat/notify-scale-upcat /etc/heat/notify-scale-down#!/bin/bash/opt/aws/bin/cfn-push-stats --credential-file /etc/heat/heat-credentials --heartbeat --watch{Ref: NodeScaleDown}
AlarmPolicyGroup
UserData commands:
OpenShift Autoscaling Workflow
Step 3: Register images with glance
[sdake@freedom heat-templates] glance image-create
name=openshift-origin-broker --disk-format=qcow2
--container-format=bare < openshift-origin-broker.qcow2
[sdake@freedom heat-templates] glance image-create
name=openshift-origin-node --disk-format=qcow2
--container-format=bare < openshift-origin-node.qcow2
Autoscaling Lifecycle Example
Step 4: Launch Heat Template
[sdake@freedom heat-templates] heat create
template-file OpenShiftAutoScaling.yaml parameters
KeyName=sdake;Prefix=broked.org;NodeCountMaximum=20
Click to edit the title text format