OpenStack - Towards an Open CloudAIIT InfoTalk #26 2011

Adam JohnsonTwitter: @adjohn

1Friday, January 21, 2011

Application Platforms Undergoing A Major Shift

2

Applications are quickly moving to the cloud•#1 priority is virtualization•#2 is cloud computing[Based on a Gartner Study]

2Friday, January 21, 2011

Midokura Overview

3

• Founded in January 2010• Based in Tokyo• 9 employees (and growing)• Former Google, Amazon engineers

• Developing “Cloud enabling technologies”• Virtualized networking• Cloud storage• Distributed architecture and management

• We think - OpenStack is going to be HUGE

3Friday, January 21, 2011

What is OpenStack?

4

Overview of the project

4Friday, January 21, 2011

OpenStack: The Mission

5

Provision virtual machines on standard hardware at massive scale

Software to reliably store billions of objects distributed across standard hardware

creating open source software to build public and private clouds

5Friday, January 21, 2011

OpenStack: The Mission

6

"To produce the ubiquitous Open Source cloud computing platform that will meet the needs of public and private cloud providers regardless of size, by being simple to implement and massively

scalable."

6Friday, January 21, 2011

7

7Friday, January 21, 2011

OpenStack History

8

2005 2010

March May June July

Rackspace Cloud

developed

Rackspace decides to Open SourceCloud Software

NASA OpenSources Nebula

Platform

OpenStack formedb/w Rackspace and

NASA

First designSummit in Austin, TX

8Friday, January 21, 2011

OpenStack History

9

2011July October November February

OpenStack launches with 25+

partners

First ʻAustinʼ coderelease with 35+

partners

First public DesignSummit in San

Antonio

Second ʻBexarʼ coderelease planned

9Friday, January 21, 2011

OpenStack Founding Principles

10

•Apache 2.0 license (OSI), open development process

•Open design process, 2x year public Design Summits

•Publicly available open source code repository

•Open community processes documented and transparent

•Commitment to drive and adopt open standards

•Modular design for deployment flexibility via APIs

10Friday, January 21, 2011

Community with Broad Commercial Support

11

11Friday, January 21, 2011

OpenStack in Japan

12

•Japan OpenStack User Group (JOSUG)•website: http://openstack.jp•google group: openstack-ja

12Friday, January 21, 2011

Why is OpenStack Important?

13

•Open eliminates vendor lock-in

•Working together, we all go faster

•Freedom to federate, or move between clouds

13Friday, January 21, 2011

HOW TO: Turn Racks of Standard Hardware Into a

Cloud with OpenStack

14Friday, January 21, 2011

Start with an open, scalable platform

OpenStack Compute OpenStack Object Storage

CLOUD OS

OpenStack Image Service

15Friday, January 21, 2011

User Control Panel

TicketingSystem

NetworkManagement

MonitoringSystems

Host Server Management

ECOSYSTEM

OpenStack Compute OpenStack Object Storage

CLOUD OS

OpenStack Image Service

Add 3rd party tools from the ecosystem

16Friday, January 21, 2011

User Control Panel

TicketingSystem

NetworkManagement

MonitoringSystems

Host Server Management

AccountBilling

Admin CLITools

Live ChatSupport

AccountManagement

ECOSYSTEM

PUBLIC CLOUD

OpenStack Compute OpenStack Object Storage

CLOUD OS

OpenStack Image Service

17Friday, January 21, 2011

User Control Panel

TicketingSystem

NetworkManagement

MonitoringSystems

Host Server Management

ECOSYSTEM

Admin ControlPanel

Dept. Accounting Chargeback

UserManagement

Enterprise SoftwareIntegration Systems

PRIVATE CLOUD

OpenStack Compute OpenStack Object Storage

CLOUD OS

OpenStack Image Service

Integrate with existing enterprise systems18Friday, January 21, 2011

OpenStack Compute

19

Software to provision virtual machines on standard hardware at massive scale

19Friday, January 21, 2011

Asynchronous eventually consistent

communication 

REST-based API

Horizontally and massively scalable

Hypervisor agnostic: support for Xen ,XenServer, Hyper-V,

KVM, UML and ESX is coming Hardware agnostic: standard hardware, RAID not required

OpenStack Compute Key Features

20Friday, January 21, 2011

API: Receives HTTP requests, converts commands to/from API format, and sends requests to cloud controller

Cloud Controller: Global state of system, talks to LDAP, OpenStack Object Storage, and node/storage workers through a queue

User Manager

ATAoE / iSCSI

Host Machines: workers that spawn instances

Glance: HTTP + OpenStack Object Storage for server imagesOpenStack Compute

21Friday, January 21, 2011

Server Groups1 GigE Connectivity

Dual Quad CoreRAID 10 Drives

Public Network

Private Network(intra data center)

Management

Example OpenStack Compute Hardware

(other models possible)

22Friday, January 21, 2011

Compute Components

23

•API Server: Interface module for command and control requests•Designed to be modular to support multiple APIs•In current release: OpenStack API, EC2 Compatibility Module•Approved blueprint: Open Cloud Computing Interface (OCCI)

•Message Queue: Broker to handle interactions between services•Currently based on RabbitMQ

•Metadata Storage: ORM Layer using SQLAlchemy for datastore abstraction

•In current release: MySQL•In development: PostgreSQL

•User Manager: Directory service to store user identities•In current release: OpenLDAP, FakeLDAP (with Redis)

•Scheduler: Determines the placement of a new resource requested via the API

•Modular architecture to allow for optimization•Base schedulers included in Austin: Round-robin, Least busy

23Friday, January 21, 2011

Compute Components

24

•Compute Worker: Manage compute hosts through commands received on the Message Queue via the API

•Base features: Run, Terminate, Reboot, Attach/Detach •Volume, Get Console Output

•Network Controller: Manage networking resources on compute hosts through commands received on the Message Queue via the API

•Support for multiple network models•Fixed (Static) IP addresses•VLAN zones with NAT

•Volume Worker: Interact with iSCSI Targets to manage volumes•Base features: Create, Delete, Establish

•Image Store: Manage and deploy VM images to host machines

24Friday, January 21, 2011

OpenStack Object Storage

25

Software to reliably store billions of objects distributed across standard hardware

25Friday, January 21, 2011

REST-based API Data distributed evenly throughout system

Hardware agnostic: standard hardware, RAID not required

OpenStack Object Storage Key Features

No centraldatabase

Scalable to multiple petabytes, billions of objects

Account/Container/Object structure (not file system, no nesting) plus Replication (N copies of accounts, containers, objects) 

26Friday, January 21, 2011

Object Storage Architecture

27

27Friday, January 21, 2011

5 Zones2 Proxies per 25Storage Nodes

10 GigE to Proxies1 GigE to

Storage Nodes24 x 2TB Drives

per Storage Node

Public Internet

Example OpenStack Object Storage Hardware

Load Balancers (SW)

Example only – many configurations possible28Friday, January 21, 2011

Object Storage Components

29

•The Ring: Mapping of names to entities (accounts,containers, objects) on disk.

•Stores data based on zones, devices, partitions, and replicas

•Weights can be used to balance the distribution of partitions

•Used by the Proxy Server for many background processes

•Proxy Server: Request routing, exposes the public API

•Replication: Keep the system consistent, handle failures

•Updaters: Process failed or queued updates

•Auditors: Verify integrity of objects, containers, and accounts

29Friday, January 21, 2011

Object Storage Components (cont)

30

•Account Server: Handles listing of containers, stores as SQLite DB

•Container Server: Handles listing of objects, stores as SQLite DB

•Object Server: Blob storage server, metadata kept in xattrs, data in binary format

•Recommended to run on XFS

•Object location based on hash of name & timestamp

30Friday, January 21, 2011

Object Storage Software Dependencies

31

Object Storage should work on most Linux platforms with the following software (main build target for Austin release is Ubuntu 10.04):

•Python 2.6•rsync 3.0

And the following python libraries:•Eventlet 0.9.8•WebOb 0.9.8•Setuptools•Simplejson•Xattr•Nose•Sphinx

31Friday, January 21, 2011

Building a Cloud

32

OpenStack Powered

32Friday, January 21, 2011

OpenStack Isn’t Everything

33

33Friday, January 21, 2011

Technical Prerequisites

34

CLOUD HAS A MINIMUM PRACTICAL SCALE

DATACENTER MADE “CLOUD READY”

EQUIPMENT BUILT FOR CLOUD

•Proof of Concept: 5+ Servers•Pilot: 20+ Servers•Production: 40+ Servers

•Networking•Power

•CPUs with virtualization and power management support•Storage platforms with flexible workload capabilities

34Friday, January 21, 2011

Cloud Ready Datacenter Requirements

35

HOMOGEOUS CONFIGURATION

INCREASED POWER DENSITY

FAT TREE / MESH NETWORKS

“LIGHTS OUT” OPERATION

35Friday, January 21, 2011

Deploying the Cloud

36

36Friday, January 21, 2011

OpenStack GUI Options

37

• Cappuccino Web Panel

• iOS Control Panel

• Android

• Django Based Control Panel

•SimCloud - Sim City Like GUI

• Etc..

37Friday, January 21, 2011

OpenStack Development Process

38

38Friday, January 21, 2011

OpenStack Release Process: Four Phases

39

Design* Development QA Release

*Design phase and Design Summit occur every other release, 2x per year

39Friday, January 21, 2011

OpenStack Releases

40

Cactus:April 2011

Bexar: February 2011

Austin:October 2010

• OpenStack Object Storage production-ready• OpenStack Compute developer preview, ready for testing and proofs of concept

• OpenStack Compute ready for enterprise private cloud deployments and mid-size service provider deployments• Enhanced documentation• Easier to install and deploy

•OpenStack Compute ready for large service provider scale deployments

We are here!

40Friday, January 21, 2011

Openstack Compute: Austin Release

41

•Multi-hypervisor support: KVM, QEMU, User-Mode Linux, Xen and XenServer

•Introduces official OpenStack API, while maintaining EC2 API option

•New image registry and delivery service, called the Glance project

•Support for two network models on compute nodes: VLANs with DHCP and flat

with either static IP pools or DHCP

•Addition of base scheduling service

•Implements WSGI to create a standard API layer with reusable components

•Support for user-friendly naming

•Refactored ORM and networking code for simpler code that is easier to understand

•Addition of SQLAlchemy Database toolkit so users can leverage existing SQL

infrastructure

41Friday, January 21, 2011

Openstack Object Storage: Austin Release

42

•Addition of a stats system that produces per-account hourly summaries of system

usage

•Ability for users to set ACL’s and grant public access to containers

•Support for API access to account and container metadata

•Rate limiting was extended to allow requests to be slowed down and support stair

stepped rate limits based on container size

•WSGI support was improved and pulled into middleware

42Friday, January 21, 2011

What’s next for OpenStack?

43

Future releases and features in the next versions of OpenStack

43Friday, January 21, 2011

Openstack Compute: Bexar Release

44

•IPv6/IPv4 Dual Stack Support

•Sheepdog Support

•I18N Japanese messages support

•Microsoft Hyper-v support

•raw-disk images

•Web-based serial console

See all the blueprints at https://blueprints.launchpad.net/nova/bexar

44Friday, January 21, 2011

Beyond Bexar

45

•Block Storage by Sheepdog

•Plugin Network Architecture

•Virtualized Networking with Midonet

45Friday, January 21, 2011

Sheepdog

46

Distributed block storage for KVM

http://www.osrg.net/sheepdog/

46Friday, January 21, 2011

Current Networking Model

47

47Friday, January 21, 2011

Network as a First Class Service

48

48Friday, January 21, 2011

Virtualized Networking Architecture

49

49Friday, January 21, 2011

Midonet - flexible virtual networks

50

Fully Distributed Virtual Networking

• L2/L3 virtual distributed switches

• Distributed virtual load balancing

• Distributed virtual firewalls

• Scale out

• and more!

50Friday, January 21, 2011

Midonet - flexible virtual networks

51

51Friday, January 21, 2011

Questions?

52

Developing, Using, Future Trends, ...

52Friday, January 21, 2011

Sign up for the Beta

53

If you want to help us build a better

cloud, you can sign up for our beta and

try our upcoming products.

http://midokura.com/beta.html

53Friday, January 21, 2011

Want to know more?

54

• http://openstack.org

• Japan OpenStack User Group

• Contact the Midokura Team

•info@midokura.jp

• OpenStack Community

54Friday, January 21, 2011

55

Life at Midokura

55Friday, January 21, 2011

But wait there’s more!

56

• Free OpenStack t-shirts!

• Midokura is hiring!

• More Questions?

• Meet the Midokura team afterwards

56Friday, January 21, 2011

Thank You!

57

Join the OpenStack revolution!

Special thanks to others who generously provided presentation content:http://www.theregister.co.uk/2011/01/08/openstack/http://www.slideshare.net/bpiatt/openstack-tutorialhttp://www.slideshare.net/annegentle/openstack-overview-for-austin-cloud-user-grouphttp://adrianotto.com/2010/09/openstack-os-is-great-for/

57Friday, January 21, 2011