Marriage of ESX and OpenStack at PayPal
Scott Carlson, PayPal
VSVC4994
#VSVC4994
VMWORLD 2013
MULTI-VENDOR AGILITY
THE MARRIAGE OF ESX AND
OPENSTACK AT PAYPAL
3
PayPal offers flexible and innovative payment solutions for consumers and merchants of all sizes.
• 132,000,000 Users.
• $300,000 Payments processed by PayPal each minute.
• 193 markets / 25 currencies.
• PayPal is the World’s Most Widely Used Digital Wallet.
ABOUT PAYPAL
4
• 80% of the PayPal front-end is virtualized on VSphere 5.0u1
• Primary Criteria
− Stability, performance, industry expertise, availability of experts
• Standardized on VCE VBLOCK© for initial implementation
• Fully consumable API
• Load-test harness well understood in industry (specInt & vMark)
− Predictable scaling pattern for horizontally scaled workloads
WHY WE VIRTUALIZED ON ESX
5
CLOUD
6
2012/2013 Shift toward an internal cloud model
• Shift from Enterprise design model to cloud-based design
• Elastically scale and self-heal infrastructure to accommodate unpredictable usage patterns of customers and internet commerce
• Separate rapidly iterating customer experiences from core services
• reduce overall cost per transaction within the environment
PAYPAL INTERNAL CLOUD
7
CLOUD IS THE GREAT ENABLER
ENABLE THE DEVELOPER
Code Deploy Enjoy
ENABLE THE BUSINESS
Payment Delivery
One-Click Developer Self Service
Global Compute & Data Fulfillment
Self-Organizing & Optimizing Infrastructure
System Intelligence Driven Operation
8
• Technology
− Adopt Open Source Solutions where ever possible
− No Vendor Lock-in
− Industry Best Practices
− Leverage Industry/ebay Inc Investments
• Functionality
− Self-Service tool for application life cycle management.
− Robust Automation & Orchestration
− Seamless On-Demand Capacity Fulfillment
PAYPAL CLOUD PLATFORM – GUIDING PRINCIPLES
9
PayPal deploying Openstack in order to help transform our global infrastructure into an agile and open cloud platform.
Agility - time to market for customer facing services
Agility - speed to service developer requests for VM resources
Agility – utilize the engineering culture of PayPal to
deliver specialized cloud services where needed
OPENSTACK
10
INCREDIBLE INDUSTRY SUPPORT
11
TECHNOLOGY STACK
Operations Portal Horizon, Ceilometer
DEVS Deployment Portal
Orchestration Engine
Cloud Formation (Heat)
Nova, Cinder, Swift, Keystone, Quantum, Horizon
Cobbler
ISC DHCP Salt BIND RHEL 6.x Hypervisor Zabbix
x86 Compute Local Storage Network Load
Balancer
User
Interface
Orchestration
Foundational
Services
Software
Infrastructure
Hardware
Infrastructure
Traffic Mgmt Monitoring Metering Stages Workflow Monitoring
LBaaS, DNSaaS FWaaS
PP Specific
F
Z
F
Z
F
Z
F
Z
F
Z
F
Z
F
Z
F
Z
F
Z
F
Z
F
Z
F
Z
CLOUD BEFORE INTEGRATION
DATABASE & RESTRICTED ZONE
Clo
ud M
ana
gem
ent Z
one
VC
ente
r M
an
ag
em
en
t WE
B
MID
KVM
Local Disk
“Stateless & Disposable”
ESX 5.0u2
Shared Storage
ESX 5.0u2
Shared Storage
ESX 5.0u2
Shared Storage
Physical
Non-virtualized
KVM
Local Disk
ESX 5.0u2
Shared Storage
Physical
Non-virtualized
FZ = Logical Fault Zones
13
SIDE-BY-SIDE
F
Z
F
Z
F
Z
F
Z
F
Z
F
Z
F
Z
CLOUD AFTER INTEGRATION
DATABASE & RESTRICTED ZONE
KVM
Local Disk
ESX 5.0u2
Shared Storage
Physical
Non-virtualized
KVM
Local Disk
ESX 5.0u2
Shared Storage
Physical
Non-virtualized
Clo
ud M
ana
gem
ent Z
one
WE
B
MID
15
But isn’t Openstack a direct replacement for ESX? Why would you keep them both?
ESX/Vsphere != Openstack
NOVA != vSphere || vCenter || ESXi
NOVA =~ vCD, vCAC
KVM =~ ESX
To connect to any hypervisor, the Openstack cloud ‘proxies’ connections to any supported hypervisor via Nova. That abstracts the ‘Cloud’ from the hypervisor
COMPARING
16
• Equivalent functionality on KVM and ESX
• Full birth to death lifecycle management of virtual machines
− Build new, power on, power off, console, rebuild, delete
• Auto-configuration of host resources following t-shirt sizes standards
− CPU, RAM, NIC, IP, OS Version
• IP Address Management
• Build from “Snapshot”/”Template”
• Deploy resources following appropriate fault zone model
• Must work from within single Horizon/Asgard interface
BRINGING ESX ‘INTO’ THE CLOUD
17
• VSphere/ESX 5.1
− 5.0 works but many, many, many back-ports / tweaks
• Single security zone per hypervisor
− No sharing of confidential & non-confidential on same hardware (PCI)
• Openstack management network communication
− This is NOT necessarily the VKERNEL network
HYPERVISOR REQUIREMENTS
18
• Shared storage required
− Data Store Cluster
− Single Data Store support [ bug fix coming ]
• DRS Enabled with auto-placement
• Data Stores must be created in advance
− No Cinder support
STORAGE REQUIREMENTS
OPENSTACK GRIZZLY
OpenStack Object Store
OpenStack Image Ser vice OpenStack Compute
OpenStackDashboard
OpenStack Identity Service
OpenStack Compute API /
Admin API
keystone(ser vice & admin APIs)
nova-api(OS, EC2, Metadata, Admin)
nova-consoleauth
nova-cert/objectstore
nova-consolenova-*proxy
VNC/ VMRC/ Spice
OpenStack Object API
http://www.solinea.com
Queue
nova-compute
nova-scheduler
novadatabase
OpenStackCompute API
OpenStack Image API
Hor izon
OpenStack Image API
ident it y backend
swif t-proxy
objectcontaineraccount
objectstore
accountDB
containerDB
OpenStack Object API
HTTP(S)
OpenStackObject API
OpenStack Identity API
OpenStack Identity
API
OpenStack Identity
API
Internet
OpenStackImage API
OpenStack Identity
API
OpenStack Image API
catalog backend
token backend
OpenStack Identity
API
hyper visor
libvirt, XenAPI, etc.
HTTP(S)
Amazon Web Ser vices
EC2 API
OpenStack Net work Ser vice
glance-api
glance-regist r y
glancedatabase
OpenStack Block Storage
OpenStack Block Storage API
cinder-api
cinder-volume
quant um-ser ver
quant um plugin(s)
OpenStack Identity
API
cinder-scheduler
cinderdatabase
OpenStack Net work API
net workprovider
OpenStack Block Storage API
OpenStack Net work API
policy backend
Queue
OpenStack Net work API
quant umdatabase
quant um agent (s)
nova-conductor
memcached
OpenStack Identity
API
⁃ OpenStack Command Line Tools (nova-client, swif t-client, etc.)
⁃ Cloud M anagement Tools (Rightscale, Enst rat ius, etc.)
⁃ GUI tools (Cyberduck, iPhone client, etc.)
volume provider
cinder-backup
OpenStack Object API
OpenStack Identity
API
Queue
OpenStack Block Storage API
ITS ALL ABOUT NOVA
CONFIG OF NOVA
21 Confidential and Proprietary
#compute_driver = libvirt.LibvirtDriver
compute_driver = vmwareapi.VMwareVCDriver
vmwareapi_host_ip=192.168.20.50
vmwareapi_host_username=root
vmwareapi_host_password=vmware
vmwareapi_cluster_name=openstack_test
vmwareapi_wsdl_loc=https://192.168.20.50/sdk/vimService.wsdl
Vcenter 5.1 Appliance
Can be multiple
clusters now!
Nova is the project name for OpenStack Compute, a cloud computing fabric controller,
the main part of an IaaS system. Individuals and organizations can use Nova to host
and manage their own cloud computing systems.
GLANCE AND IMAGES
22 Confidential and Proprietary
glance add name=”MYMACHINE.vmdk" disk_format=vmdk container_format=bare
is_public=true vmware_adaptertype="lsiLogic" vmware_disktype="preallocated"
vmware_ostype="otherGuest" < /path/to/MYMACHINE.vmdk
Rules for Glances images for VMWare
• Saved in VMDK Format
• Imported as VMDK Format
• Thick Provisioned VMDK Required
• No split VMDK allowed (must be merged)
• In a multi-hypervisor cloud, all images are separate
‘per hypervisor’ (no launching KVM VM’s on ESX)
BUILDING AND INSTALLING OS
23 Confidential and Proprietary
• Kickstart
• Build a small root disk
• Use kickstart to image machine
• Post-install with puppet to customize machine and
add additional mount points depending on
application requirements
• Image Deploy
• Currently does not support ‘config-drive’
• Need Guest Tools to ‘duplicate’ functionality
WHAT ABOUT THE NETWORK
24 Confidential and Proprietary
• Quantum requires NVP 3.2
• Cannot talk directly to VSphere API to allocate VDS
Port to NIC
• Implemented via vAPP – integration bridge
• Configured as separate transport zone within Nicira
WHAT’S LEFT
25 Confidential and Proprietary
• Component “at-scale” testing
Currently manage “tens” at a time, need to move
to “hundreds” or “thousands”
• Most fixes scheduled to go into Havanna, every
bug-fix needs to be reviewed and possible back-
ported to Grizzly
• Multiple Data Store enumeration on a cluster
• Full Certification on VCE© VBLOCK with Vision
Intelligent Operations, auto-upgrades, and full
Openstack support of all components
READING MATERIALS
• http://www.solinea.com/2013/06/15/openstack-grizzly-architecture-revisited/ - Ken Pepple
• http://www.slideshare.net/kenhui65/getting-started-with-open-
stack?ref=http://cloudarchitectmusings.com/2013/06/16/getting-started-with-openstack/ - Kenneth Hui
• http://docs.openstack.org/trunk/openstack-compute/admin/content/config-drive.html - config-drive doc
• http://docs.openstack.org/trunk/openstack-compute/admin/content/vmware.html - Openstack VMWARE doc
• http://www.ebay.com - Buy It Now
• http://www.paypal.com - and then Pay for it Here!
Interested? [email protected]
THANK YOU
Marriage of ESX and OpenStack at PayPal
Scott Carlson, PayPal
VSVC4994
#VSVC4994
Top Related