LISA Cap Plan Free

Performance Management with Free and Bundled ToolsAdrian Cockcroft Netflix Inc. [email protected] (Co-authored with Mario Jauvin MFJ Associates [email protected]) 18 January, 2012

The Performance People

Agenda

Overview of Capacity Planning Requirements and Data Sources Performance Data Collection Free Network Monitoring Tools Free System Monitoring Tools Free Load Generation and Modelling Tools Licences and ReferencesJanuary 18, 2012 Adrian Cockcroft and Mario Jauvin

What are we talking about?QA Load generation with Grinder or SLAMD, modelling with PDQ and R Network monitoring with WireShark, MRTG, BigSister, Cacti, Nagios, OpenNMS, Zenoss, Openxtra, ntop Application Tier monitoring with Orca, Cacti, BigSister, Ganglia, XEtoolkit Database Tier monitoring With SEtoolkit, Orca, XEtoolkitJanuary 18, 2012 Adrian Cockcroft and Mario Jauvin

Capacity Planning Requirements and Data Sources

January 18, 2012

Adrian Cockcroft and Mario Jauvin

Definitions

Capacity Resource utilization and headroom Planning Predicting future needs by analyzing historical data and modeling future scenarios Performance Monitoring Collecting and reporting on performance data Free Tools Bundled with the OS or available for no $$$

January 18, 2012


Capacity Planning Requirements

We care about CPU, Memory, Network and Disk resources, and Application response times We need to know how much of each resource we are using now, and will use in the future We need to know how much headroom we have to handle higher loads We want to understand how headroom varies, and how it relates to application response times and throughputJanuary 18, 2012 Adrian Cockcroft and Mario Jauvin

CPU Capacity Measurements

CPU Capacity is defined by CPU type and clock rate, or a benchmark rating like SPECrateInt2000 CPU utilization is defined as busy time divided by elapsed time for each CPU CPU load average measures the average number of jobs running and ready to run

January 18, 2012


CPU Measurement Issues

Biased sample CPU measurements Microstate measurements are accurate, but are platform and tool specific Hyperthreading non-linearities Platform specific details, e.g. are interrupts included in system time?

http://perfcap.blogspot.com/2005/10/how-busy-is-your-cpu-really.html

January 18, 2012


Memory Capacity Measurements

Physical Memory Capacity Utilization and Limits

Kernel memory Shared Memory segment Executable code, stack and heap File system cache usage Unused free memory

Virtual Memory Capacity - Swap Space Memory Throughput

Page in and page out rates

January 18, 2012


Network Capacity Measurements

Network Interface Throughput

Byte and packet rates input and output TCP connection count and connection rates TCP byte rates input and output Byte rates read and write NFS/SMB service response times HTTP operation rates Get and put payload byte rates and size distributionJanuary 18, 2012 Adrian Cockcroft and Mario Jauvin

TCP Protocol Specific Throughput

NFS/SMB Protocol Specific Throughput

HTTP Protocol Specific Throughput

Disk Capacity Measurements

Detailed metrics vary by platform Easy for the simple disk cases Hard for cached RAID subsystems Almost Impossible for shared disk subsystems and SANs

Another system or volume can be sharing a backend spindle, when it gets busy your own volume can saturate, even though you did not change your own workloadJanuary 18, 2012 Adrian Cockcroft and Mario Jauvin

Capacity Planning Challenges

Constantly changing infrastructure Limited attention span from staff Horizontally scaled commodity systems Per node software licencing costs too much Too many tools, too many agents per node Too much data, not enough analysis Non-linear and non-intuitive scalability Lack of tools and metrics for virtualized resources

January 18, 2012


Observability

Four different viewpoints

Management Engineering QA Testing Operations

Each needs very different information Ideal would be different views of the same performance database Reality is a mess of disjoint tools

January 18, 2012


Management Viewpoint

Daily summary of status and problems Business oriented metrics Future scenario planning Marketing and management input Concise report with dashboard style status indicators Free tools: R, Spreadsheet and Web based displays, no good summarization toolsJanuary 18, 2012 Adrian Cockcroft and Mario Jauvin

Engineering Viewpoint

Large volumes of detailed data at several different time scales Input to tuning, reconfiguring and future product development Low level problem diagnosis Detailed reports with drill down and correlation analysis Free tools: XE/SE Toolkit, Orca, Ganglia, Cacti, R

January 18, 2012


QA Test Viewpoint

Workload specification tools Load generation frameworks Testing for functionality and performance Regression tools to compare releases Modelling difference between test configuration and production configuration Free Tools: The Grinder, SLAMD, R, PDQ

January 18, 2012


Operations Viewpoint

Immediate timeframe Real time display, updated in seconds Alert based monitoring High level problem diagnosis Simple high level graphs and views Free tools: BigSister, Nagios, OpenNMS, MRTG, Cacti, Ganglia, WireShark, ntop

January 18, 2012


Measurement Data Interfaces

Several generic raw access methods

Read the kernel directly (not a good idea) Structured system data (Solaris kstat, Linux /proc) Process data Network data Accounting data Application data Scrape data from vmstat, iostat, netstat, sar, ps Higher overhead, lower resolution, missing metrics

Command based data interfaces

Data available is platform specific either way Much more detail on this topic in the Solaris/Linux Performance Measurement and Tuning Class

January 18, 2012


Process based data

Used by ps, top, proctool and debuggers, pea.se, Solaris proc tools Solaris and Linux both have /proc/pid/metric hierarchy Linux also includes system information in /proc rather than kstat Advantages The recommended and supported process access API Metric data structures reasonably stable over releases Consistent data using locking Solaris microstate data provides accurate process state timers Disadvantages High overhead for open/read/close for every process Linux reports data as ascii text, Solaris as binary structures

January 18, 2012


Kernel Trace - TNF, Dtrace, ktrace

Solaris, Linux, Windows and other Unixes have similar features Solaris has TNF probes and prex command to control them User level probe library for hires tracepoints allows instrumentation of multithreaded applications Kernel level probes allow disk I/O and scheduler tracing Advantages Low overhead, microsecond resolution I/O trace capability is extremely useful Disadvantages Too much data to process with simple tracing capabilities Trace buffer can overflow or cause locking issues Solaris 10 Dtrace is a quite different beast! Much more flexible

January 18, 2012


Dtrace Dynamic Tracing

One of the most exiting new features in Solaris 10, rave reviews New book: "Solaris Performance and Tools" by Richard McDougall Advantages No overhead when it is not in use Low overhead probes can be put anywhere/everywhere Trace data is correlated and filtered at source, get exactly the data you want, very sophisticated data providers included Bundled, supported, designed to be safe for production systems Disadvantages Solaris specific, but being ported to BSD/Linux No high level tools support yet Yet another scripting language to learn somewhat similar to awk

January 18, 2012


Accounting Records

Standard Unix System V Accounting - acct

Tiny, incomplete (no process id!) low resolution, no overhead!

Solaris Extended System and Network Accounting - exacct

Flexible, Overly complex, Detailed data, Interval support No overhead! 100% capture ratio for infrequent samples!

January 18, 2012


Extracct for Solaris

I wrote extracct tool to get extended acct data out in a useful form See http://perfcap.blogspot.com/search?q=accounting for details and get code from http://www.orcaware.com/orca/pub/extracct Pre-compiled code for Solaris SPARC and x86. Solaris 8 to 10.

Useful data is logged in regular columns for easy import Includes low overhead network accounting config file for TCP flows Interval accounting option to force all processes to cut records Automatic log filename generation and clean switching Designed to run directly as a cron job, useful today

More work needed to interface output to SE toolkit and Orca

January 18, 2012


Example Extracct Output# ./extracct Usage: extracct [-vwr] [ file | -a dir ] -v: verbose -w: wracct all processes first -r: rotate logs -a dir: use acctadm.conf to get input logs, and write output files to dir The usual way to run the command will be from cron as shown 0 * * * * /opt/exdump/extracct -war /var/tmp/exacct > /dev/null 2>&1 2 * * * * /bin/find /var/adm/exacct -ctime +7 -exec rm {} \; This also shows how to clean up old log files, I only delete the binary files in this example, and I created /var/tmp/exacct to hold the text files. The process data in the text file looks like this:timestamp locltime duration 1114734370 17:26:10 0.0027 1114734370 17:26:10 0.0045 1114734370 17:26:10 0.0114 1109786959 10:09:19 -1.0000 1109786959 10:09:19 -1.0000 procid ppid 16527 16526 16526 16525 16525 8020 1 0 2 0 uid usr 0 0.000 0 0.000 0 0.001 0 4.311 0 0.000 sys majf 0.002 0 0.001 0 0.005 0 3.066 96 0.000 0 rwKB vcxK icxK 0.53 0.00 0.00 0.00 0.00 0.00 1.71 0.00 0.00 47504.69 49.85 0.00 0.00 0.00 sigK sycK arMB mrMB command 0.00 0.1 0.7 28.9 acctadm 0.00 0.1 1.1 28.9 sh 0.00 0.3 1.0 28.9 exdump 0.18 0.34 456.2 0.9 1.0 init 0.00 0.0 0.0 0.0 pageout

January 18, 2012


Configuration information

Configuration data comes from too many sources! Solaris device tree displayed by prtconf and prtdiag Linux configuration data can be found in /proc Solaris 8 adds dynamic configuration notification device picld SunVTS component test system has vtsprobe to get config SCSI device info using iostat -E in Solaris Logical volume data from vxprint and metastat HW RAID info from device specific tools Critical storage config info must be accessed over ethernet It is very hard to combine all this data

January 18, 2012


Free Network Monitoring Tools

January 18, 2012


SNMP

Simple network management protocol UDP protocol based on port 161 Client/server like

Client is called management application entity Server is called an agent entity

Agent entity is designed to be implemented on network hardware, router, switches, etcJanuary 18, 2012 Adrian Cockcroft and Mario Jauvin

SNMP MIBs

Management information base Defines the structure and the semantic of the information that can be reported on Most commonly used is MIB-II which defines a set of standard networking attributes

Interface tables System level information Routing tables

Specified using ASN.1 (abstract syntax notation 1)January 18, 2012 Adrian Cockcroft and Mario Jauvin

SNMP commands

Called PDU (protocol data units) GET GETNEXT GETBULK SET Encoded using BER (basic encoding rules)

January 18, 2012


Versions

Version 1, original version done in May 1991 Version 2, around 1993. Failed because the IETF credo of rough consensus and running code could not be met on securing SNMP Turned into V2c for community string security (like V1) Version 3, added security and complexity in 1998January 18, 2012 Adrian Cockcroft and Mario Jauvin

SNMP tools

Too numerous to name all but OpenNMS Nagios Cacti MRTG Net-snmp

See www.snmplink.orgJanuary 18, 2012 Adrian Cockcroft and Mario Jauvin

SNMP tools

Snmpwalk will report all data in a specified MIB getIf will report data about interfaces and includes built-in MIB browser Snmptable will report tabular data from MIB tables

January 18, 2012


OpenNMS

Well. its not that portable

95% java is not 100% java Requires about 20-30 different platform specific packages (PostgreSQL, Perl, RRD tool, Tomcat 4 etc) Difficult to install Easy auto discovery Web-based interface

January 18, 2012


OpenNMS

Main screen shot

January 18, 2012


OpenNMS

Node screen shot

January 18, 2012


Nagios

Easy to build/compile (on Solaris 10) Easy to install Quick response from CGI Configuration is manual and a pain

13 configuration files with all kinds of interrelated entries Tedious and error prone

Requires plugins to do anythingJanuary 18, 2012 Adrian Cockcroft and Mario Jauvin

Nagios

Main screen shot

January 18, 2012


Nagios

Host detail screen shot

January 18, 2012


January 18, 2012


ntop

Similar to familiar UNIX top tool for processes but used for network Provide huge selection of real-time data Can be found at http://www.openxtra.co.uk/

January 18, 2012


ntop Active Sessions

January 18, 2012


ntop Hosts

January 18, 2012


ntop Network Load

January 18, 2012


ntop_Network_Thruput

January 18, 2012


ntop Port Dist

January 18, 2012


ntop_Protocol_Dist

January 18, 2012


ntop Protocols

January 18, 2012


Zenoss

Open source monitoring and management of IT infrastructure Zenoss core is free Other editions are for a fee Get it from http://www.zenoss.com/download/

January 18, 2012


zenoss Architecture

January 18, 2012


zenoss Dash Config

January 18, 2012


zenoss Google

January 18, 2012


zenoss Google Alerts

January 18, 2012


Zenoss Graphs

January 18, 2012


zenoss Topology

January 18, 2012


MRTG

Really simple to install and configure Require manual config file creation Only for MIB-II interface plotting out of the box Graphing not flexible, axis, time etc

January 18, 2012


MRTG

Interface screen shot

January 18, 2012


MRTG

Other CPU screen shot

January 18, 2012


RRD tool

Software to store, retrieve and graph numerical time series data Use a round robin algorithm Data files are a fixed size

Dont grow Dont require maintenance

January 18, 2012


RRD tool

Compiles on most platforms Used by many SNMP based tools

OpenNMS Cacti BigSister WeatherMap4RRD MailGraph

January 18, 2012


RRD tool

14all CGI script that plots data similar to MRTG Configurable to collect data at different interval (unlike MRTG) Flexible and variable in what data can be collected

January 18, 2012


RRD tool

Sample screen shot

January 18, 2012


RRD tool

Screen shot

January 18, 2012


RRD tool

Create a RRD database rrdtool create test.rrd \ --start 920804400 \ DS:speed:COUNTER:600:U:U \ RRA:AVERAGE:0.5:1:24 \ RRA:AVERAGE:0.5:6:10

January 18, 2012


RRD toolCreate a graph rrdtool graph speed.png \ --start 920804400 --end 920808000 \ DEF:myspeed=test.rrd:speed:AVERAGE \ LINE2:myspeed#FF0000

January 18, 2012


Free Performance Data Collection and Rules Toolkits

January 18, 2012


SE toolkit Example Tools

A free performance toolkit for rapidly creating custom data sources Makes all the very extensive Solaris metrics easily available Very system specific and not enough metrics exist to port to Linux Written by Rich Pettit with contributions from Adrian Cockcroft Get SE3.4 from http://sourceforge.net/projects/setoolkit/ Open source with support for SPARC & x86 Solaris 8, 9, 10Example SE Programs monlog.se siostat.se cpu_meter.se msacct.se net.se iostat.se infotool.se anasa syslog.se mon_cm.se zoom.se xio.se vmmonitor.se pea.se tcp_monitor.se uname.se multi_meter.se dfstats cpus.se net_example kview pure_test.se nproc.se systune collisions.se kvmname.se watch uptime.se orcollator.se dumpkstats.se live_test.se xiostat.se mpvmstat.se ps-ax.se vmstat.se ps-p.se nfsstat-m.se pwatch.se perfmeter.se pw.se nx.se xload.se netmonitor.se netstatx.se nfsmonitor.se percollator.se virtual_adrian_lite.se iost.se xit.se disks.se iomonitor.se virtual_adrian.se

Function Rule Monitors cpg.se Disk Monitors CPU Monitors Process Monitors Network Monitors Clones Data browsers aw.se Contributed Code Test Programs

January 18, 2012


SE language features

SE is a 64bit interpreted dialect of C Not a new language to learn from scratch! Standard C /usr/ccs/bin/cpp used at runtime to preprocess SE scripts Main omissions - pointer types and goto Main additions - classes and string type powerful ways to handle dynamically allocated data built-in fast balanced tree routines for storing key indexed data Dynamic linking to all existing C libraries Built-in classes access kernel data Supplied class code hides details, provides the data you want Example scripts improve on basic utilities e.g. siostat.se, nx.se, pea.se Example rule based monitors e.g. virtual_adrian.se, orcallator.se

January 18, 2012


Creating Rules

Based on real experiences of all the things that go wrong Capture an approximation to intuition Test and calibrate rules on as many systems as possible Easy??

January 18, 2012


Configuring Rules

Thresholds should be configured Very application dependent Capture the operating envelope

Measure the underlying values Measure peaks in normal operation Note values during problems Set thresholds to capture the difference SE Toolkit, Cacti, Ganglia, Nagios, OpenNMS etc.January 18, 2012 Adrian Cockcroft and Mario Jauvin

This applies to any tool

Rules as Objects

Define only the input and output information Hide implementation details Make high level rule objects trivial to use and reuse SE Toolkit does it in three lines of code:

#include Declare rule object as a typed variable Read and use or print object status

January 18, 2012


"virtual adrian" rules summary

Disk Rule for all disks at once Looks for slow disks and unbalanced usage Network Rule for all networks at once Looks for slow nets and unbalanced usage Swap Rule - Looks for lack of available swap space RAM Rule - Looks for short page residence times CPU Power Rule Scales on MP systems Looks for long run queue delays Mutex Rule - Looks for kernel lock contention and high sys CPU time TCP Rule Looks for listen queue problems Reports on connection attempt failures

January 18, 2012


XE Toolkit - www.xetoolkit.com

Complete re-write of SE Toolkit by Rich Pettit

Extensible Java collector, customize with jar files Release 1.2 available April 2008 Multi-platform support Solaris, Linux/x86, Windows, BSD, OSX, HP-UX, AIX, Linux/s390, Linux/Power Free GPL version for standard use and shared derivations Open source, hosted at http://sourceforge.net/projects/xe-toolkit/ Commercial support available if needed Commercial product license for custom in-house derivations

Licencing

Addresses all the issues people had with SE toolkit !January 18, 2012 Adrian Cockcroft and Mario Jauvin

Captive Metrics / XE Toolkit Architecture

January 18, 2012


Free System Monitoring Tools

January 18, 2012


Collated Performance Data - Orca

Problems with time sync when collecting data from multiple tools No timestamp at all for vmstat, netstat, df... No timestamp by default for iostat and ps... No way to collect realtime stats from an http logfile Use SE Toolkit to generate one timestamped row containing all the data First version of percollator.se written by Adrian Cockcroft in 1996 Extended orcallator.se written by Blair Zajac a few years later Graphs generated by orca batch job feeding rrdtool based web pages Active community developing tool at http://www.orcaware.com Extended to collect much more data, including process workloads Basic data collection ported to Linux, HP-UX and Windows

Orca is basically MRTG for System metrics rather than Network See http://www.orcaware.com/orca/docs/Orca_Understanding_Performance_Data.pptJanuary 18, 2012 Adrian Cockcroft and Mario Jauvin

Orca data collections

Collected using procollator reading info from /proc on Linux[Uptime] [Average # Processes in Run Queue (Load Average)] [CPU Usage] [New Process Spawn Rate] [Number of System & Running Processes] [Context Switches & Interrupts Rate] [Interface Input Bits Per Second] [Interface Output Bits Per Second] [Interface Input Packets Per Second] [Interface Output Packets Per Second] [Interface Input Errors Per Second] [Interface Output Errors Per Second] [Interface Input Dropped Per Second] [Interface Output Dropped Per Second] [Interface Output Collisions] [Interface Output Carrier Losses] [TCP Current Connections] [IP Statistics] [TCP Statistics] [ICMP Statistics] [UDP Statistics] [Disk System Wide Reads/Writes Per Second] [Disk System Wide Transfer Rate] [Disk Reads/Writes Per Second] [Disk Transfer Rate] [Disk Space Percent Usage] [Physical Memory Usage] [Swap Usage] [Page Ins & Outs Rate] [Swap Ins & Outs Rate]

Orca on Solaris collects many more metrics than shown above Strength of Orca is lots of detailed metrics with low overhead for collection Easily customized to add more system metrics or application metrics Orca can already track HTTP traffic and parse log filesJanuary 18, 2012 Adrian Cockcroft and Mario Jauvin

All metrics are stored in round robin database format using RRDtool to generate displays over different time spans Web page is simple collection of plots with drill down by metric or by time Suitable for monitoring a relatively small number of systems in great detail, e.g. backend database servers

January 18, 2012


Cacti www.cacti.net

Web based user interface based on RRDtool More sophisticated GUI than Orca or MRTG Less sophisticated system metric collection, but more coverage of networking Better management of groups of systems and devices than Orca, useful for tens to hundreds of nodes Access control and personalization for usersJanuary 18, 2012 Adrian Cockcroft and Mario Jauvin

January 18, 2012


January 18, 2012


Ganglia www.ganglia.info

Web based RRDtool GUI somewhat similar to Cacti Better management of clusters of systems and devices than Cacti, useful for hundreds to thousands of nodes in a hierarchy of clusters Provides many summary statistic plots at cluster level and collects detailed configuration data XML based data representation Uses low overhead network protocol In common use at hundreds of large HPC Grid sites, less visibly in use at some large commercial sites

January 18, 2012


January 18, 2012


January 18, 2012


January 18, 2012


BigBrother and BigSister

Network and system dashboard alert monitor Widely used at internet sites Bigbrother is at http://www.bb4.com Bigsister is at http://bigsister.graeff.com Bigsister seems to have more features, alert logging, better portability and more efficient data collection. Compatible update to BB4.January 18, 2012 Adrian Cockcroft and Mario Jauvin

January 18, 2012


January 18, 2012


January 18, 2012


Free QA Test and Modelling Tools

January 18, 2012


QA Test Requirements

Generate test workload

SLAMD, Grinder Any of the tools already mentioned

Collect performance metrics

Report regression against baseline Predict capacity needed for production system

Use spreadsheets for simple linear prediction Use modelling tools such as PDQ for queuing models

January 18, 2012


Grinder 3 - Powerful New Features

100% Pure Java - works on any hardware platform and any operating system that supports J2SE 1.3 and above.

Java and Jython based load testing framework

Web Browsers: simulate web browsers using HTTP, and HTTPS. Web Services: test interfaces using SOAP and XML-RPC. Database: test databases using JDBC. Middleware: RPC and MOM based systems using IIOP, RMI/IIOP, RMI/JRMP, and JMS. Other Internet protocols: POP3, SMTP, FTP, and LDAP.

See http://grinder.sourceforge.net/g3/features.html J2EE Performance Testing with BEA WebLogic Server by Peter Zadrozny, Philip Aston and Ted Osborne, originally published by Expert Press and now by APress uses Grinder 2 throughout.January 18, 2012 Adrian Cockcroft and Mario Jauvin

SLAMD

Load generation framework, written in Java Originally built to test LDAP servers by Sun Extended to be very generic and published as open source. Actively being developed. Sophisticated functions and user interface See http://www.slamd.com Latest Release 2.0 has better usability focusJanuary 18, 2012 Adrian Cockcroft and Mario Jauvin

January 18, 2012


January 18, 2012


January 18, 2012


PDQ Modelling Tool

Dr Neil Gunthers toolkit at http://www.perfdynamics.com Library used from C or Perl provides MVA queueing models Use to calibrate in QA and predict in production PDQ modelling tool details:

The Practical Performance Analyst Dr. Neil Gunther McGraw-Hill, 1998 ISBN 0-07-912946-3 Analyzing Computer System Performance with Perl:PDQ 2004, ISBN 3-54-020865-8

January 18, 2012


References and Conclusion

January 18, 2012


Licences for Free Tools

Open Source Initiative

OSI Approved licences http://opensource.org/licenses/category http://zooko.com/license_quick_ref.html

Comparisons of Common Licences

January 18, 2012


Web Pages and Books

Adrians Performance and other topics blog

http://perfcap.blogspot.com http://www.mfjassociates.net/perf_links.html http://www.generalconcepts.com/resources/monitoring/ http://www.openxtra.co.uk/resource-center/open_source_network_monitor_tools.php

MFJ Associates performance tools link page More free tools compiled by John Sellens More tools compiled by Openxtra SE toolkit info: Sun Performance and Tuning - Java and the Internet - Adrian Cockcroft and Richard Pettit - Sun Press/Prentice Hall, 2nd Edition, 1998 ISBN 0-13095249-4 Solaris 8 and Linux: System Performance Tuning 2nd Edition Gian-Paolo Musumeci, OReilly 2002 ISBN: 0-596-00284-X Solaris Internals http://www.solarisinternals.com

Richard McDougall and James Mauro - new 2nd edition and new performance book by Richard McDougall and Brendan Gregg

January 18, 2012


Concluding Remarks

Many large installations depend on free tools A full suite of functionality is available Several tools are needed to cover the bases Tradeoff between function and ease of use Support may be available, but typically Google is the best support tool Functionality is increasing.January 18, 2012 Adrian Cockcroft and Mario Jauvin

[email protected] [email protected]

January 18, 2012


LISA Cap Plan Free

Documents

Transcript of LISA Cap Plan Free