Monitor OpenStack Environments from the bottom up and front to back
-
Upload
icinga -
Category
Technology
-
view
516 -
download
2
Transcript of Monitor OpenStack Environments from the bottom up and front to back
Monitor OpenStack Environments From the Bottom Up and Front to Back Thomas Stocking, Director of Systems Engineering, Founder. GroundWork Open Source Inc.February 16, 2016 | Icinga Camp
2
What’s ahead?
• Overview—the impact of virtualization on IT operations• How OpenStack fits into the virtualization landscape• Monitoring the changing landscape of IT infrastructure• New monitoring concepts• Selecting the right tools to fit the right process• Conclusion
www.gwos.com ©2016 Groundwork Open Source, Inc. All rights reserved worldwide. CONFIDENTIAL
3
Start with the facts: OpenStack is for real!
www.gwos.com ©2016 Groundwork Open Source, Inc. All rights reserved worldwide.
OpenStack deployments are
not just happening in a far away land…
It’s allopen source.
5000 active members,
and growing.
Not just a geek
movement.
Serious deployments for IT operations, not just in the Silicon Valley.
It’s disruptive and requires serious retooling for IT
operations.
Many corporate sponsors.
Let’s review the challenges and discover which tools best fit the new realities…
4
Once upon a time…
SysAdmin task:Add to IT infrastructure {
Deploy servers into data centerProvisioning applicationsDefine monitoring for each elementMonitoring: SSH checks and port gets
} repeat
Checking for up down and send email
www.gwos.com ©2016 Groundwork Open Source, Inc. All rights reserved worldwide.
MONITOR
SERVER
APPS
APPS
SERVER
APPS
APPS
SERVER
APPS
SWITCH SWITCH
FIREWALL
ROUTER
PORTGET
SSH
SNMP
SNMP
SNMP
5
VIRTUALIZATION
Then came along compute virtualization…
Data center compute optimization• Server resources were virtualized to
improve efficiency which was <15%.• Products like ESX allowed resource
optimization not disturbing the provisioning process.
Rollout• VM machines were provisioned.• Applications installed.• Monitoring defined as before.
API was added to the virtualization manager (example Vsphere API)
www.gwos.com ©2016 Groundwork Open Source, Inc. All rights reserved worldwide.
MONITOR
SERVER
APPS
APPS
SERVER
APPS
APPS
SERVER
APPS
SWITCH SWITCH
FIREWALL
ROUTER
PORTGET
SSH
SNMP
SNMP
SNMP
REST API
6
MONITOR
SNMP
VIRTUALIZATION
But it didn’t stop there…
Software defined everything• Compute (hypervisors)• Storage (SDS)• Network (SDN)
Hybrid Cloud Public/Private• Amazon Web Services• Rackspace• Azure
Change created blind spots in coverage. Suddenly, SSH/SNMP methods don’t cover everything.
www.gwos.com ©2016 Groundwork Open Source, Inc. All rights reserved worldwide.
SERVER
APPS
APPS
SERVER
APPS
APPS
SERVER
APPS
SWITCH SWITCH
FIREWALL
ROUTER
PORTGET
SSH
REST API
REST API
SNMP
SNMP
VIRTUALIZED NETWORK
VIRTUALIZED NETWORK
SWITCHES SWITCHESSWITCHES
7
FilesDevices
VMs
Shared I/O FogNetwork & Storage
The infrastructure landscape completely changed.
www.gwos.com ©2016 Groundwork Open Source, Inc. All rights reserved worldwide.
SIL
O V
IEW
SIL
O V
IEW
SIL
O V
IEW
SIL
O V
IEW
SIL
O V
IEW
HY
PE
RV
ISO
R M
AN
AG
ER
VIE
W
8
VIRTUALIZATION
And the moment you think you’ve seen it all…
DevOps is pushing the envelope even further…
Linux Containersare the new kids in town
API for deployment:Monitoring is somebody else’s job
www.gwos.com ©2016 Groundwork Open Source, Inc. All rights reserved worldwide.
MONITOR
SERVER
APPS
APPS
SERVER
APPS
APPS
SERVER
APPS
SWITCH SWITCH
FIREWALL
ROUTER
PORTGET
SSH
VIRTUALIZED NETWORK
VIRTUALIZED NETWORK
SWITCHES SWITCHESSWITCHES
REST API
REST API
DOCKER HOST
DOCKER HOST
DOCKER HOST
Application IsolationRapid DeploymentElastic Scalability
9
ESX
NSX
vSan
Nova
Neutron
Cinder Glance
EC2
VPC
S3
Linux KVM
Network
NFS
Diverse Virtualization Stacks
Don’t panic.
What happened over the last 5 years:• Virtualize everything• Private and public clouds
• API-centric world is not just for applications
• REST API is a standard• CORBA, SOAP and proprietary API’s
are now classified as dinosaurs
• Browser based UI/JavaScript is king• Learn to speak REST and JSON
www.gwos.com ©2016 Groundwork Open Source, Inc. All rights reserved worldwide.
vSphereAPI
OpenStack
API
AWSAPI
oVirtAPI
10
Sounds really good, doesn’t it?
• Service-oriented architecture• Pluggable hypervisors, network, storage to support a wide range of technologies• Elastic Compute Units for better virtualization efficiency• Standardized API’s to all services
But what’s the reality?
How does OpenStack fit in?
www.gwos.com ©2016 Groundwork Open Source, Inc. All rights reserved worldwide.
Object Store–Swift
Dashboard–Horizon
Image–Glance
Compute/Hypervisor–Nova Storage–Cinder Network–
Quantum-Neutron
Identity–Keystone
FaçadeService
API
API
API API API API
11
OpenStack Profile Editor
www.gwos.com ©2016 Groundwork Open Source, Inc. All rights reserved worldwide.
12
OpenStack Profile Editor
www.gwos.com ©2016 Groundwork Open Source, Inc. All rights reserved worldwide.
13
Performance Data
Virtual Infrastructure Container Metrics Provisioning/Configuration
ManageIQ/Cloud Forms
Ansible
API
Manager Stack
API
HP HelionAPI
VMWare VRealize
Mirantis Fuel
API
API
API is good, but comprehensive coverage is even better...
www.gwos.com ©2016 Groundwork Open Source, Inc. All rights reserved worldwide.
Dashboard
Object Store Image Computer Storage Network
Identity
OpenStack API
Legacy Network
Servers ServersServers
Storage Racks
Applications, Infrastructure Checks
SSH/SNMP/port get
Unified View
Standardized Data Collection
14
Unified View
Silo Tools• Impossible to correlate• Naming mismatch• Over-monitoring• Encourages “It’s not my problem,
check your system”• No Big Picture dashboards
www.gwos.com ©2016 Groundwork Open Source, Inc. All rights reserved worldwide.
Need For• Stack monitoring bottom to top• Aliasing of names• Combine best of breed collectors• Correlated metrics across all
infrastructure• Dashboard for each customer/client
15
Stack monitoring for OpenStack
www.gwos.com ©2016 Groundwork Open Source, Inc. All rights reserved worldwide.
Function Source Host name Alias
Network, Storage
NetflowSNMP
Dsw1-422 os-eng-h1
Hardware IPMISNMP
Drac-server1 os-eng-h1
Identity server check-mysqlcheck-port
server1:port os-eng-h1
Operating System
check-proccheck-memcheck-load
server1 os-eng-h1
OpenStack-API Hypervisor CPU/Mem metrics, Networkstorage
10.10.0.1 os-eng-h1
OpenStack API VM CPU/mem, Network, storage 10.10.10.123 eng-slicer-1
Operating system
check-proccheck-memcheck-load
eng-cent6-actg eng-slicer-1
DockerHost Memory, CPU eng-cent6-actg eng-slicer-1
Container Memory, CPU, Procs FAC3443DA77 eng-slicer-1
Application Check_https 172.28.102.51 eng-slicer-1
HY
PE
RV
ISO
RV
MA
PP
16
Automation/Continuous configuration discovery
Adding virtual machines, containers is automatic…
Monitoring is like a flight recorder—collecting all active data
“Continuous configuration discovery” by…Synchronization of configuration over APIsAuto-registration of agents/Download of pluginsCollection of data availability and performancewww.gwos.com ©2016 Groundwork Open Source, Inc. All rights reserved worldwide.
V C
HYPERVISOR
ManagementMonitoring System
Configuration collector
Datacollector
Plugins
DashboardsDATA
C C V V V V V C
so is the monitoring
17
OpenStack Monitoring: Tools Selection
• Fuel—OpenStack deployment• Murano—Application deployment• Fuel Plugin—Configuration generate• Fuel Plugin—Configure monitoring
www.gwos.com ©2016 Groundwork Open Source, Inc. All rights reserved worldwide.
OpenStack API Icinga2 setup
OpenStack Linux OS Platform
Hypervisor Hardware IPMIPort
Agent
OpenStack ControllerKeystone
OpenStack StorageCinder
Unified Monitoring View
Data ManagementCollectorCollectorCollector
Fuel
Murano Application Software Catalog
Icinga2
Docker
Node2
Node3
Docker
VM 1
OPENTSDB Icinga2
VM 1 VM 2 VM 3 VM 4
OPENTSDB Docker
18
Performance Data
Monitoring Systems
www.gwos.com ©2016 Groundwork Open Source, Inc. All rights reserved worldwide.
Centralized Data Collector• Monitoring send Perf Data to single API
Expandable Storage Cluster• Expand on demand
Dashboard to visualize and drill down• Historic, raw performance data• Group alike metrics to find outliers
Grafana Dashboards
19
Reference Architecture Monitoring
www.gwos.com ©2016 Groundwork Open Source, Inc. All rights reserved worldwide.
Aliasing and data normalization in backend
API
DataCollector
Hub
Collector
Collector
Collector
Collect
Data Integration
NormalizationAliasing
API
Integrate
API
Visualize
Growing number of API’s require
integration Hub
20
Benefits
Automation• Reduced cost of maintenance• Dynamic configuration• Streamlined operations
Personnel• System Administrator and DevOps
functions merging
www.gwos.com ©2016 Groundwork Open Source, Inc. All rights reserved worldwide.
Capacity & Resource Planning• Complete Bottom-to-Top
(Network-to-Application) data collection
• Allocate resources at the right level
Hardware• No vendor lock-in• Transparency through API
21
Lessons learned…
• Virtualization and containerization require new monitoring techniques• Legacy hardware/software will be around for awhile, so don’t throw tools away• Everything API needs integration• Aliasing is hard, but may be easier than cross silo cooperation• Use the best tool for the task• Don’t over-monitor to cover a gap
www.gwos.com ©2016 Groundwork Open Source, Inc. All rights reserved worldwide.
22
Conclusion
• Open source tools are the driving force for innovation• DevOps just select the best tools to do the job• A single monitoring tool to “rule them all” doesn’t exist• Integration is a complex task—don’t expect teams to agree on naming, process,
and workflow• Automation and pragmatism will prevail as DevOps has demonstrated• “Virtualize everything” will continue• Automation and continuous discovery is necessary for rapid scale out• OpenStack is the open source virtualization platform, but monitoring coverage
needs major improvements
www.gwos.com ©2016 Groundwork Open Source, Inc. All rights reserved worldwide.
23
Thank you.
Thomas [email protected]
www.gwos.com ©2016 Groundwork Open Source, Inc. All rights reserved worldwide.