Oreilly solinea-managing-openstack
-
Upload
viet-stack -
Category
Technology
-
view
115 -
download
0
description
Transcript of Oreilly solinea-managing-openstack
Accelerating the adoption of Cloud Computing
Beyond Installation: Managing Your OpenStack Cloud
May 6th, 2014
! Ken Pepple is the co-founder and Chief Technology Officer of Solinea
! Prior to founding Solinea, he led the introduction of Internap's OpenStack-based public cloud services while serving as their Director of Cloud Development
! Code contributor since Bexar release of OpenStack
! Author of O'Reilly Media's "Deploying OpenStack" and several other books
2
Speakers
Introduction
! Installing OpenStack gets all the attention … ! … but distributions like Red Hat OSP and
Cloudscaling are attacking this problem ! They will beat it. Then what ? ! The reality is that OpenStack management is
what we should be focusing on. ! Installation is 2 – 3 weeks … management is
forever
3
OpenStack Architecture
OpenStack Object Store
OpenStack Image Service OpenStack Compute
OpenStackDashboard
OpenStack Identity Service
OpenStack Compute API /
Admin API
keystone(service & admin APIs)
nova-api(OS, EC2, Metadata, Admin)
nova-consoleauth
nova-cert/objectstore
nova-consolenova-*proxy
VNC/Spice
OpenStack Object API
http://www.solinea.com
Queue
nova-compute
nova-scheduler
novadatabase
OpenStackCompute API
OpenStack Image API
Horizon
OpenStack Image API
identity backend
swift-proxy
objectcontaineraccount
objectstore
accountDB
containerDB
OpenStack Object API
HTTP(S)
OpenStackObject API
OpenStack Identity API
OpenStack Identity
API
OpenStack Identity
API
OpenStackImage API
OpenStack Identity
API
OpenStack Image API
catalog backendtoken backend
OpenStack Identity
API
hypervisor
libvirt, XenAPI, etc.
HTTP(S)
Amazon Web Services
EC2 API
Internet / Enterprise Network
OpenStack Network Service
glance-api
glancedatabase
OpenStack Block Storage
OpenStack Block Storage API
cinder-api
cinder-volume
neutron-server
neutron plugin(s)
OpenStack Identity
API
cinder-scheduler
cinderdatabase
OpenStack Network API
networkprovider
OpenStack Block
StorageAPI
OpenStack Network API
policy backend
Queue
OpenStack Network API
neutrondatabase
neutron agent(s)
nova-conductor
memcached
OpenStack Identity
API
⁃ OpenStack Command Line Tools (nova-client, swift-client, etc.)⁃ Cloud Management Tools (Rightscale, Enstratius, etc.)⁃ GUI tools (Cyberduck, iPhone client, etc.)
volume provider
cinder-backup
OpenStack Object API
OpenStack Identity
API
Queue
OpenStack Block Storage API
OpenStack Orchestration
heat-api
heat-engine
heatdatabase
Queue
cloudwatch-api
OpenStack Orchestration API
OpenStack Identity
API
OpenStack Compute API
OpenStack Bock Storage API
OpenStack Network API
glance-api
OpenStack Database
trove-api
trove-taskmgr
trovedatabase
trove-conductor
OpenStack Identity
API
OpenStack Database API
OpenStack Block Storage APIOpenStack Orchestration API
OpenStack Compute API
Agent
Queue
4
* Ceilometer omitted for clarity
OpenStack Deployment
5
OpenStack Management Basics
! Development and test cluster – Smaller, but representative – Same set and version of services – Reproduce problems, test fixes and practice upgrades
! Configuration management system – Chef, Puppet, Ansible, SaltStack, etc. – Your OpenStack distribution already uses one – Pick one and stick with it – everything falls under it
! Skilled and trained staff – Experienced Linux admins with virtualization skills – Network architects that understand cloud – Trained for OpenStack
6
Developing Toolkit for Management
! Troubleshooting tools – Operating system level tools – OpenStack specific tools
! Administration tools – OpenStack specific tools
! Monitoring tools – Monitoring platforms – Log management tools
! Specialized tools – Image creation
7
Troubleshooting Tools
! Tools used to investigate or fix problems within your stack
! Mostly Linux tools but some are specific to OpenStack
! These need to span virtualization, networking and normal system administration
8
Troubleshooting Hypervisor
! Vary by hypervisor, each one has it’s own tooling ! Map VM to hypervisor by OpenStack CLI with nova show!
! Investigate hypervisor through virsh tools ! Also can access backing store for VM through
hypervisor mount point or Cinder volume
9
VM Troubleshooting # nova list!+---------------------+-------+---------+------------+-------------+-------------------------------------+!| ID | Name | Status | Task State | Power State | Networks |!+---------------------+-------+---------+------------+-------------+-------------------------------------+!| f94b097d-b030-473b- | ken | ACTIVE | - | Running | rdonet=192.168.90.11 |!+---------------------+-------+---------+------------+-------------+-------------------------------------+!# nova show f94b097d-b030-473b-86a3-d501091c650b!+--------------------------------------+------------------------------------------------------------+!| Property | Value |!+--------------------------------------+------------------------------------------------------------+!| OS-EXT-AZ:availability_zone | nova |!| OS-EXT-SRV-ATTR:host | localhost.localdomain |!| OS-EXT-SRV-ATTR:hypervisor_hostname | localhost.localdomain |!| OS-EXT-SRV-ATTR:instance_name | instance-0000000e |!| OS-EXT-STS:power_state | 1 |!| OS-EXT-STS:task_state | - |!| OS-EXT-STS:vm_state | active |!| OS-SRV-USG:launched_at | 2014-05-06T06:13:01.000000 |!| created | 2014-05-06T06:11:55Z |!| flavor | m1.small (2) |!| hostId | 7e31bda83a3586907464e8e68f83a035bf9fa500d9579b2b807fa9f0 |!| id | f94b097d-b030-473b-86a3-d501091c650b |!| image | cirros-0.3.2-x86_64 (f66d54e8-f8bd-4220-930f-86b6b44dfe4d) |!| rdonet network | 192.168.90.11 |!| security_groups | default |!| status | ACTIVE |!+--------------------------------------+------------------------------------------------------------+!# vish list!Id Name State!----------------------------------------------------! 1 instance-0000000e running!
10
Troubleshooting Backing Store (Ephemeral)
# cd /var/lib/nova/!# ls!buckets CA images instances keys networks tmp!
# cd instances/!# ll!total 16!drwxr-xr-x. 2 nova nova 4096 May 6 09:03 13e86b72-7e14-43f5-ab2f-e7abf117213f!drwxr-xr-x. 2 nova nova 4096 May 2 11:25 _base!-rw-r--r--. 1 nova nova 45 May 5 23:18 compute_nodes!drwxr-xr-x. 2 nova nova 4096 Apr 30 19:28 locks!
# cd 13e86b72-7e14-43f5-ab2f-e7abf117213f/!# ll!total 208!-rw-rw----. 1 qemu qemu 0 May 6 09:04 console.log!-rw-r--r--. 1 qemu qemu 262656 May 6 09:03 disk!-rw-r--r--. 1 nova nova 79 May 6 09:03 disk.info!-rw-r--r--. 1 nova nova 1529 May 6 09:04 libvirt.xml!
# file disk!disk: Qemu Image, Format: Qcow , Version: 2!
11
The ‘disk’ file is our qcow image. The XML file is
the KVM template.
Troubleshooting Network
! Combination of Linux, OpenvSwitch and OpenStack tools
! OpenStack tools will show logical configuration of Neutron’s ports, routers and subnets – neutron port-list, net-list and router-list!
! OpenvSwitch will map internal and external bridges – ovs-vsctl and ovs-dpctl!
! Linux tools will show you inside VLAN and Linux namespaces – ip netns, iptables and tcpdump!
12
Network Troubleshooting: Router # ip netns show!qdhcp-8a496b23-ef2c-4170-9919-611d9a12180f!qrouter-e439ff2a-1973-4cda-86a4-20c977eec828!qdhcp-16b5549e-3a1a-4254-b122-f7507f003597!!!# ip netns exec qrouter-e439ff2a-1973-4cda-86a4-20c977eec828 ifconfig!qg-bbe18331-0c Link encap:Ethernet HWaddr FA:16:3E:47:63:D0! inet addr:192.168.57.132 Bcast:192.168.57.255 Mask:255.255.255.0! inet6 addr: fe80::f816:3eff:fe47:63d0/64 Scope:Link! UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1! RX packets:1335 errors:0 dropped:0 overruns:0 frame:0! TX packets:145 errors:0 dropped:0 overruns:0 carrier:0! collisions:0 txqueuelen:1000! RX bytes:197195 (192.5 KiB) TX bytes:13110 (12.8 KiB)!!qr-c4e2b047-4a Link encap:Ethernet HWaddr FA:16:3E:FD:4E:A9! inet addr:192.168.90.1 Bcast:192.168.90.255 Mask:255.255.255.0! inet6 addr: fe80::f816:3eff:fefd:4ea9/64 Scope:Link! UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1! RX packets:364 errors:0 dropped:0 overruns:0 frame:0! TX packets:309 errors:0 dropped:0 overruns:0 carrier:0! collisions:0 txqueuelen:1000! RX bytes:34760 (33.9 KiB) TX bytes:36569 (35.7 KiB)!!!!# ip netns exec qrouter-e439ff2a-1973-4cda-86a4-20c977eec828 netstat -nr!Kernel IP routing table!Destination Gateway Genmask Flags MSS Window irtt Iface!192.168.57.0 0.0.0.0 255.255.255.0 U 0 0 0 qg-bbe18331-0c!192.168.90.0 0.0.0.0 255.255.255.0 U 0 0 0 qr-c4e2b047-4a!
13
Troubleshooting OVS Bridges # ovs-vsctl show!06667946-811b-4c7b-97a5-eafc8386e9ff! Bridge br-int! Port "qvo246622d1-02"! tag: 2! Interface "qvo246622d1-02"! Port "tap3dfc8b70-ee"! tag: 1! Interface "tap3dfc8b70-ee"! Port "tapecef7610-4f"! tag: 2! Interface "tapecef7610-4f"! Port "tapc4e2b047-4a"! tag: 2! Interface "tapc4e2b047-4a"! Port br-int! Interface br-int! type: internal! Bridge br-ex! Port br-ex! Interface br-ex! type: internal! Port "eth1"! Interface "eth1"! Port "tapbbe18331-0c"! Interface "tapbbe18331-0c"! ovs_version: "1.11.0"!
14
Neutron’s integration bridge connecting VMs
Neutron’s external bridge
Physical NIC for internet access
Monitoring
! Metering is not monitoring – Ceilometer isn’t a monitoring solution
! Horizon doesn’t save history ! Monitor for FCAPS: fault, configuration,
accounting, performance and security ! Needs to be instrumented at multiple levels
– Hardware/Operating System, OpenStack, VM – Although VM monitoring may be left to the user
! Needs to be used across all elements and processes
15
Operating System Monitoring
! Required set of information as any other set of systems – CPU, memory, availability, etc.
! Process level information – RabbitMQ, database, OpenStack processes, etc.
! Should rely on host sending information to monitoring server (not ping model)
! Ideally has APIs and strong discovery to aid automation
16
! Installed as part of many distributions
! Open source ! Easy installation and
usage ! API is an add-on
module
Nagios
17
Nagios OpenStack Plugin
! Add service checks for some OpenStack services – Glance – Keystone – Nova – Swift API and dispersion
! Available in most Linux distributions – # sudo apt-get install nagios-plugins-openstack!
! More information and checks available at http://openstack.prov12n.com/monitoring-openstack-nagios-3/
18
! Open source monitoring tool used at several large service provider clouds
! Strong API and discovery modes
! Templates can be applied to host groups for monitoring
Zabbix Console
19
Zabbix Templates
! Templates created for each type of server – Compute nodes, controllers, Swift object servers, etc.
! Each template checks processes running and configuration management running – Should issue commands against processes not rely
on process table to catch hung processes ! All nodes also get default OS template ! Alerting set for pagerduty
20
Log Management
! More than just for error viewing ! Primary source of OpenStack data ! Useful for
– Finding OpenStack bugs – Understanding event timings (spin new VM) – Visualizing cluster level statistics (VMs running) – Creating dashboards
! Can be challenging to store, query and interpret data – Clusters can generate GBs per day – Use dedicated tools and data stores – May be required for legal / audit reasons
21
! Commercial log management solution
! Visualization, ad hoc queries, post processing and add-ons
! Easy to setup dashboards
! Supported with relatively easy installer
Splunk
22
! Open source alternative to Splunk
! Requires more complicated setup to parse logs correctly
! Provides ad hoc queries as well as dashboards
! Active community
Logstash, Kibana and ElasticSearch
23
Interesting Uses for Log Data
! VMs – CPUs/Instances by hypervisor (scheduler efficiency) – Total vCPUs/CPUs in cluster available versus used – Spawn success and failures – Spawn time – Top Users of VMs/vCPUs
! Authentication – Tokens generated versus invalidated – Failed authentications
! Errors – All error messages / stack traces create alert
! Logs that have stopped (zombie processes)
24
“Canary” Scripts
! Highest level check for cloud infrastructure: “Can we spin a new VM ?” – Custom written script that starts VM, attaches block
storage, assign IP address, pings outside world then terminates
– Logs to all actions with timings into log management solution
! Run every 5 to 15 minutes ! Also can be run interactively ! This should be written for your own site
25
Specialized Tools
! Many sites will want to be able to create their own custom images – CI/CD “golden images”
! Several commercial and open source alternatives – CohesiveFT Server3 – Elasticbox (https://www.elasticbox.com/) – Packer (http://www.packer.io/)
! All provide ability to create images with specified software pre-installed via command line
26
Manageability Improvements in Icehouse
! Nova live upgrade ! Swift discovery API
27
Rolling (“live”) Upgrades
! Ability to upgrade a running cluster to new release
! Upgrade controller(s) first then individual compute nodes
! Requires several pre-conditions – Neutron upgraded first – Nova-conductor being used to isolate DB schemas – Set icehouse compatibility mode
/etc/nova/nova.conf # Set a version cap for messages sent to compute services. If # you plan to do a live upgrade from havana to icehouse, you # should set this option to "icehouse-compat" before beginning # the live upgrade procedure. (string value) compute=icehouse-compat
28
! API calls to /info will return information about the cluster
! Users now able to take advantage of the unique features available in each cluster
! Turned on by default but can be disabled
# swift capabilities!Core: swift! Options:! account_listing_limit: 10000! container_listing_limit: 10000! max_account_name_length: 256! max_container_name_length: 256! max_file_size: 5368709122! max_meta_count: 90! max_meta_name_length: 128! max_meta_value_length: 256! max_object_name_length: 1024! strict_cors_mode: True! version: 1.13.1!Additional middleware: keystoneauth!Additional middleware: staticweb!Additional middleware: tempurl! Options:! methods: ['GET', 'HEAD', 'PUT']!
29
Swift Discoverability