Cloud Computing mit mathematischen Anwendungen€¦ · Automated program start after machine launch...

KIT – University of the State of Baden-Württemberg and National Laboratory of the Helmholtz Association

4.Cloud Applications

www.kit.edu

Cloud Computing mit mathematischen Anwendungen Dr. habil. Marcel Kunze Engineering Mathematics and Computing Lab (EMCL) Institut für Angewandte und Numerische Mathematik IV Karlsruhe Institute of Technology (KIT)

2

Examples of Cloud Applications

1.  Rendering of movies 2.  Management of digital data 3.  Office work 4.  Collaborative work

Cloud Computing | SS2011 | M.Kunze

3

Cloud Architectures using SQS, S3 and EC2


4 Cloud Computing | SS2011 | M.Kunze

Cloud Applications (1): Rendering of Movies

!   Architecture: Weakly coupled services !   Use storage service for files !   Use compute service for work !   Organize workflow by use of a queuing service

1.  Store files 2.  Store messages with

instructions 3.  Start computation

12

Cloud Applications (2): SmugMug http://www.smugmug.com/


13

Cloud Applications (2): SmugMug http://www.smugmug.com/

Management of Pictures and Movies –  Millions of users –  Billions of pictures –  Revenue: Several million $ per year –  19 employees -  IT Services: Amazon Web Services -  User management: OpenID

Mashup M

ashu

p S3, EC2, FPS, …

Application = Mashup of services of various providers


14

Cloud Applications (3): Windows Live

!   Windows 7 ships with ready to go cloud applications !   Windows Azure Software Development Kit


15

Cloud Applications (4): Google Apps

!   Google Apps offers office applications !   Collaboration within the company (e.g. KIT)


17

docs.google.com


18

Optimize Linear Equations

!   An example of when and how to use Solve !   Let's say you're a farmer and you want to decide what to grow. You know

that each crop will bring in a certain amount of money and take a certain amount of land, capital, and fertilizer to grow

!   You can grow three things: wheat, corn, and broccoli !   Each of them has its own costs and brings in a given amount of money.

However, there are only limited resources available: !   Money available: $170 !   Land available: 70 acres !   Fertilizer available: 100 tons


19


!   Here's how you can express this information in the spreadsheet:


20


!   These are the formulas used in the spreadsheet: !   Money (Cell B7) =9*A3+8*B3+7*C3 !   Land (Cell B8) =7*A3+8*B3+9*C3 !   Fertilizer (Cell B9) =2*A3+10*C3 !   Profit (Cell B16) =6*A3+8*B3+C3

!   When you run Solve, it can tell you what to grow:



Hausaufgabe 4

Einem IaaS-Anbieter steht ein Rechenzentrum mit 100 Schränken und einer gesamten Anschlussleistung von 1.500 kW zur Verfügung. In jedem Schrank sind Rechner mit insgesamt 1.000 CPU-Cores und 1,5 TeraByte Memory installiert. Es sollen 3 verschiedene Klassen von virtuellen Maschinen angeboten werden: Micro, Small und Large. Jede Klasse hat spezifische Anforderungen an Memory, Zahl von CPU-Cores, Energieverbrauch und bringt einen gewissen Gewinn. Die Managementwerkzeuge können maximal 80.000 Instanzen verwalten. Large Small Micro Nebenbedingung Memory (GB) 10 5 1 Max. 150 TeraByte CPU-Cores 8 2 1 Max. 100.000 CPU-Cores Energiebedarf (W) 50 25 15 Max. 1.500 kW Gewinn (Cent/h) 10 4 2 Zu maximieren Verwenden Sie zur Optimierung des Gewinns die Solve-Funktion der Tabellenkalkulation aus den Google Apps.

• Wie viele Instanzen sollten von jedem Typ gefahren werden, damit der Gewinn maximal ist? • Wie groß ist der Gewinn pro Stunde? • Wie groß ist in diesem Fall der Energiebedarf?

22

Cloud Research and Development


24

Open Cirrus Cloud Computing Research Testbed http://opencirrus.org

!   An open, internet-scale global testbed for cloud computing research !   A tool for collaborative research !   Data center management & cloud services

!   Resources !   Multi-continent, multi-datacenter, cloud computing system !   Federated “Centers of Excellence” around the globe

!   each with 100–400+ nodes and up to ~PB storage !   and running a suite of cloud services

!   Structure !   Sponsors: HP Labs, Intel Research, Yahoo! !   Founding partners: IDA Singapore, KIT, UIUC !   New partners: CESGA, CMU, ETRI, MIMOS, RAS,

China Mobile, China Telecom

!   Cover story in IEEE Computer, vol 43, no 4, April 2010


25

Cloud Systems Research and Architecture

!   Perform research in the following areas: !   Cloud services !   Datacenter federation !   Datacenter management and automation !   Data-intensive applications and systems !   Cloud management tools, e.g. KOALA

http://koalacloud.appspot.com/

!   Methodology !   Perform experiments also on a low system level !   Utilize flexible cloud computing frameworks !   Compare different configurations and implementations

!   Simple, transparent, controllable cloud computing infrastructure



Proprietary Cloud Computing Stacks


Open-source Cloud Stack

!   OpenCirrus researchers have complete access to the hardware and software platform

28

Open Cirrus Blueprint

IT-Infrastructure Layer (Physical Resource Sets)

Cloud Infrastructure Services

Cloud Application Services

Virtual Resource Sets

Eucalyptus OpenNebula



Physical Resource Sets (PRS)

!   PRS service goals !   Provide mini-datacenters to researchers !   Isolate experiments from each other !   Stable base for other research

!   PRS service approach !   Allocate sets of physical co-located nodes, isolated inside VLANs. !   Leverage existing software (e.g. Utah Emulab, HP OpsWare) !   Start simple, add features as we go !   Base to implement virtual resource sets

!   Hardware as a Service (HaaS)


Virtual Resource Sets (VRS)

!   Basic idea: Abstract from physical resource by introduction of a virtualization layer

!   Concept applies to all IT aspects: CPU, storage, networks and applications, …

!   Main advantages !   Implement IT services exactly fitting customer‘s varying need !   Deploy IT services on demand !   Automated resource management !   Easily guarantee service levels !   Live migration of services

!   Infrastructure as a Service (IaaS) !   Implement compute and storage services !   De-facto standard: Amazon Web Services interface


What is it?

!   Open-Source toolkit for building cloud infrastructures !   Originates from the EC funded RESERVOIR project

!   Orchestrates storage, network and virtualization technologies to enable the dynamic placement of multi-tier services on distributed infrastructures, combining both data center resources and remote cloud resources, according to allocation policies

!   Provides internal and cloud administration and user interfaces for the full management of the IaaS cloud platform


Architecture

Cloud Plugins: Amazon EC2 and ElasticHosts connectors


Feature Function

Internal Interface •  Unix-like CLI for fully management of VM life-cycle and physical boxes •  XML-RPC API and libvirt virtualization API

Scheduler •  Requirement/rank matchmaker allowing the definition of workload and resource-aware allocation policies

•  Support for advance reservation of capacity through Haizea

Virtualization Management

•  Xen, KVM, and VMware •  Generic libvirt connector (VirtualBox planned for 1.4.2)

Image Management •  General mechanisms to transfer and clone VM images

Network Management •  Definition of isolated virtual networks to interconnect VMs

Service Management and Contextualization

•  Support for multi-tier services consisting of groups of inter-connected VMs, and their auto-configuration at boot time

Security •  Management of users by the infrastructure administrator

Fault Tolerance •  Persistent database backend to store host and VM information

Scalability •  Tested in the management of medium scale infrastructures with hundreds of servers and VMs (no scalability issues has been reported)

Installation •  Installation on a UNIX cluster front-end without requiring new services •  Distributed in Ubuntu 9.04 (Jaunty Jackalope)

Flexibility and Extensibility

•  Open, flexible and extensible architecture, interfaces and components, allowing its integration with any product or tool

Features


What is it?

!   Open-Source software infrastructure for implementing Cloud computing on clusters from UC Santa Barbara

!   EUCALYPTUS - Elastic Utility Computing Architecture for Linking Your Programs To Useful Systems

!   Implements Infrastructure as a Service (IaaS) – gives the user the ability to run and control entire virtual machine instances (Xen, KVM) deployed across a variety of physical resources

!   Interface compatible with Amazon EC2 !   Includes Walrus, a basic implementation of Amazon S3 interface !   Potential to interact with the same tools, known to work with

Amazon EC2 and S3 !   Eucalyptus is an important step to establish an open Cloud

computing infrastructure standard

37

Architecture

!   Cloud Controller (CLC) !   Collects resources information of the CCs !   Manages the S3 / EBS Services !   Cloud Meta-Scheduler


Public Network

CLC

S3 / EBS DataBase

Private Network

CC

NC NC NC

Private Network

CC

NC NC NC

Public Network

!   Cluster Controller (CC) !   Select informa5on of free

resources !   Control the alloca5ons of the

VMs on the NCs

!   Node Controller (NC) !   Controls the Xen /

KVM Hypervisor !   Sends resource

informa5on to the Cluster Controller


!   Open-Source implementation of Google App Engine !   Project driven by UC Santa Barbara !   App Engine is a platform service that allows to run Web apps in

Python (and JAVA) on the Google infrastructure ! AppScale works transparently on Cloud infrastructures like

Eucalyptus ! AppScale is compatible with Google App Engine !   Applications for Google App Engine can be developed and tested

in a private cloud

AppScale http://appscale.cs.ucsb.edu

Quelle: Navraj Chohan


Tashi http://wiki.apache.org/incubator/TashiProposal

Cluster Manager

Node

Node

Node

Node

Node

Storage Service

Virtualization Service

Node

Scheduler

!   Co-Scheduling of CPU and data

40

R&D Examples

!   Job flow to run Monte-Carlo simulation !   HPC as a Service !   Cloud management services


41

!   Grid example: Run 1000 Jobs to generate 1 million Monte-Carlo events !   The Grid is a complex system

A Grid Job Flow Scenario


42

A Cloud Job Flow looks much smarter

!   Works the same like on a local computer !  Develop and debug the program locally

!   Build a virtual machine with the following characteristics !  Automated program start after machine launch (e.g. Amazon EC2) !  Do the data processing (e.g. generate 1000 Monte-Carlo events) !  Write output to persistent cloud storage (e.g. Amazon S3) !  Automated shut down once finished to stop accounting

!   Just instantiate the suiting number of machines in the cloud !  Simply launch 1000 machines to produce 1 million events !  Cost is the same:

!  Run 1 machine 1000 hours !  Run 1000 machines 1 hour

!   No need for batch queues and scheduling (“unlimited resources”)


43

HPC as a Service

Tradi9onal HPC Architecture … !   is characterized by very specific computer clusters designed for special applica5ons !   offers pre-‐defined opera5on systems and user environments only !   serves one single applica5on at a given 5me !   provides restricted user access !   provides management privileges exclusively to administrators

Concept of HPCaaS !   Capability of using clustered servers and

storage as resource pools, fully automated management

!   Individual cluster configura5on on-‐demand !   Flexibility to serve mul5ple user groups and

applica5ons with varying requirements !   Customers gain resource management

privileges


44

Cloud Management: KOALA http://koalacloud.appspot.com


!   Mobile management of hybrid cloud resources as SaaS solution !   Plan: Develop iPad/iPhone/Android application

IaaS

PaaS

KOALA

45

!   Advantages and Disadvantages of a Cloud-‐based Management of IaaS wrt. a local solu9on (e.g.: Elas9cfox, Hybridfox) !   Advantages:

!   Flexibility concerning the used browser !   Support of EC2/S3/EBS and Eucalyptus !   No local installa5on necessary (except of the private key) !   Cloud installa5on corresponds to the cloud compu5ng idea

!   Disadvantages: !   Users have to trust the provider of the management applica5on concerning

the data privacy and opera5on availability !   Advantages and Disadvantages of KOALA wrt. the Amazon Management Tools

(especially AWS Management Console) !   Advantages:

!   Support of EC2/S3/EBS and Eucalyptus !   KOALA can run in a private cloud (via AppScale)

!   Disadvantages: !   Not all AWS-‐Features implemented (5ll now) !   No support by Amazon

KOALA hSp://koalacloud.appspot.com -‐ hSp://code.google.com/p/koalacloud


46

! AppScale runs in EC2 !   KOALA runs in AppScale !   The cloud can be managed

out of itself !   It is not necessary to store

credentials at Google

KOALA hSp://koalacloud.appspot.com -‐ hSp://code.google.com/p/koalacloud


47

Good to know

!   All course info is on http://studium.kit.edu ! http://www.math.kit.edu/mitglieder/lehre/cloudcomp2011s/


Cloud Computing mit mathematischen Anwendungen€¦ · Automated program start after machine launch...

Documents

Transcript of Cloud Computing mit mathematischen Anwendungen€¦ · Automated program start after machine launch...