Realizing Continuous Performance Management
Transcript of Realizing Continuous Performance Management
Realizing Continuous Performance ManagementSteven HainesQuest SoftwareJava Domain ExpertFebruary 14, 2008
Your logo here
Agenda
State of the Market Cost of Mistakes Test-Driven Development Continuous Integration Continuous Performance Management CPM in Practice Business Value Next Steps… Q&A
Industry Experts Say…
Forrester Research reports nearly 85 percent of companies with revenue of more than $1 billion reported incidents of significant application performance problems. Survey respondents identified the architecture and deployment as the primary causes of these problems.
-Forrester Research
Medium-sized businesses* are losing an average of 1% of their annual revenue, or $867K, to downtime. Application outages and degradations are the biggest sources of downtime, costing these companies $213K annually.
-Infonetics Research
*101 to 1000 employees
Impact of Poorly Performing Applications
Business-to-consumer
− Direct loss of customer revenue
− Loss of confidence
Business-to-business
− Loss of reputation
− Potential loss of business relationships
Intranet
− Loss of productivity
− Longer time-to-market = loss of competitive advantage
Cost of Mistakes
Performance problems found later in production cost almost 10x more to fix… Test Early, Test Often
Why is the Current State of Testing Failing?
Test suites are written to out-of-date artifacts
Test suites are not comprehensive enough
Test suites are not automated
− Not performed regularly
− No regression testing
Test-Driven Development
Test cases must be written prior to writing code
Test cases written by developers
Each code component must include a test suite
Test Suite
TestCode
TestCode
TestCode
Test-Driven Development Continuous Integration
Source codeRepository
Continuous Performance Management
Picture to show profiling
of a test: JProbe or PAS
type of screen
Profile Analyze React
Test-Driven Development
Test-Driven Development Lifecycle
Add a Test
Test Suite
Prove Failure
Add a Test
Test Suite
Test-Driven Development Lifecycle
Test-Driven Development Lifecycle
Prove Failure
Add a Test
Implement New Functionality
Test Suite
Test-Driven Development Lifecycle
Prove Failure
Add a Test
Prove Success
Implement New Functionality
Test Suite
Prove Failure
Re-factorthe Code
Add a Test
Prove Success
Implement New Functionality
Test Suite
Test-Driven Development Lifecycle
Shorter Development Cycles
Limited Debugging
Faster Triaging
Clearly Solved Business Problems
Confidence in Deployment
Business Value:
Continuous Integration
The Challenge
− With complicated applications, integration can take longer than development
The Solution
− Integrate after each code check-in
The Tool
− Continuous Integration Server
Test Suite
TestCode
TestCode
TestCode
Test-Driven Development Continuous Integration
Source codeRepository
CIServer
Build
Test
Publish
Continuous Performance Management
Picture to show profiling
of a test: JProbe or PAS
type of screen
Profile Analyze React
Test-Driven Development + Continuous Integration
Continuous Integration Lifecycle
Monitors SourceCode Repository
ContinuousIntegration
Server
Continuous Integration Lifecycle
Detects New Check-Ins
Monitors SourceCode Repository
ContinuousIntegration
Server
Continuous Integration Lifecycle
Detects New Check-Ins
Monitors SourceCode Repository
Compiles Entire Application
ContinuousIntegration
Server
Continuous Integration Lifecycle
Detects New Check-Ins
Monitors SourceCode Repository
Runs Exhaustive Test Suite
• Commit Build• Secondary Builds
Compiles Entire Application
ContinuousIntegration
Server
Continuous Integration Lifecycle
Detects New Check-Ins
Publishes Build Results
Monitors SourceCode Repository
Runs Exhaustive Test Suite
• Commit Build• Secondary Builds
Compiles Entire Application
ContinuousIntegration
Server
Shorter integration times
Integration issues resolved as they occur
Higher quality end product
Business Value:
What is Continuous Performance Management?
Challenge in Performance Testing
− Time consuming and error prone because manual
− Problems discovered at the end of development effort often affect assumptions made at the beginning of a development effort
Requires re-factoring the architecture Solution
− Performance Management in a Continuous Integration Environment
Proactive approach Execute performance tests on every
developer commit− On every code check-in
Continuous Performance Management (CPM)
• Create additional Secondary Builds that implement performance tests
1. Performance Unit Tests
2. Performance Integration Tests• Business case
• Use case scenario
3. Performance Stress/Load Tests
SecondaryBuild 2
SecondaryBuild 3
SecondaryBuild 1
CPM
Test Suite
TestCode
TestCode
TestCode
Test-Driven Development Continuous Integration
Source codeRepository
CIServer
Build
Test
Publish
Continuous Performance Management
Profile Analyze Repair
Test-Driven Development + Continuous Integration + CPM
Performance Unit Tests• Memory
– Lingering Object References (memory leaks)
– Object cycling• Performance
– Identify slow running code• Coverage
– Quantify the percentage of classes, methods, lines of code and conditions exercised by unit tests
Performance Integration Tests Performance analysis of a single user
− Identify request response times that exceed service-level agreements
− Triage to offending methods and SQL statements
− Follow requests across tiers (and across JVMs) Identify application problems prior to load
− Find problems in isolation
− Easier to pinpoint root cause
Performance Stress/Load Tests Performance analysis under load (multiple users)
− Identify request response times that exceed service-level agreements
− Triage to offending methods and SQL statements
− Follow requests across tiers (and across JVMs)
− Validate environmental configuration
− Identify resource contentions
Identify scalability problems prior to deployment
− Identify environmental configuration issues
− Resolve scalability and reliability problems
Continuous Performance Management Toolset Continuous Integration Server –
CruiseControl
Build Environment – Apache Ant
Test Bed – JUnit
Load Generator – Apache JMeter
Performance Testing:
− Unit tests: Quest JProbe
− Integration/Load tests: Quest PerformaSure
CPM – Too Arduous a Task?
It is a common misconception that CPM is too risky and time consuming of an undertaking
We’ve gone through painstaking efforts to make CPM as easy as possible to implement
Only requires a handful of build script additions
CPM In Practice – Performance Unit Tests Additions to Ant build script to profile all your JUnit tests
Easy to do – can paste on a single slide
<!-- Set location of the JProbe Enterprise Tools -->
<property name="jprobe.enterprisetools.home“ location="c:\jprobe-ent" />
<!-- Define the Pre-processor CLASSPATH -->
<path id="classpath.preprocessor">
<fileset dir="${jprobe.enterprisetools.home}" includes="lib/*.jar"/>
</path>
<!-- Execute the preprocessor -->
<java classname="com.javasrc.anttools.JProbePreprocessor">
<sysproperty key="src.dirs" value="${src};${src.test}" />
<sysproperty key="jprobe.home" value="C:\Program Files\JProbe 7.0" />
<sysproperty key="jprobe.build.dest" value="${jprobe.enterprisetools.home}\jprobe.xml" />
<classpath refid="classpath.preprocessor"/>
</java>
<!-- Import the Generated Script -->
<import file="${jprobe.enterprisetools.home}\jprobe.xml" />
CPM In Practice – Unit Test Report
CPM In Practice – Unit Test Report
Performance Unit Testing – Quest JProbe
Three analysis enablers:
− Memory
− Performance
− Coverage
CPM In Practice – Integration / Load Tests Additions to Ant build script to launch JMeter Easy to do – can paste on a single slide<!-- Import the JMeter task -->
<taskdef name="jmeter“ classname="org.programmerplanet.ant.taskdefs.jmeter.JMeterTask"/>
<!-- Execute the test script -->
<jmeter jmeterhome="C:\lib\jakarta-jmeter-2.2"
testplan="${basedir}/mytests/MyTest.jmx"
resultlog="${basedir}/mytests/MyTestResults.jtl">
<property name="jmeter.save.saveservice.output_format" value="xml" />
</jmeter>
<!-- Generate an HTML Report -->
<xslt in="${basedir}/mytests/MyTestResults.jtl"
out="${basedir}/mytests/MyTestResults.html"
style="${basedir}/mytests/jmeter-results-report.xsl" />
CPM In Practice – Integration / Load Tests Additions to Ant build script to record a performance session<property name="pas.home" location="j:\PerformaSure5.0" />
<target name="pas" description= "Start a PerformaSure recording" >
<!-- Execute nexusctl -->
<exec executable="cmd">
<arg value="/c" />
<arg value="${pas.home}/scripts/nexusctl.cmd" />
<arg value="start-recording" />
<arg value="-user" />
<arg value="user" />
<arg value="-l" />
<arg value="30m" />
<arg value="-t" />
<arg value="10s" />
<arg value="-f" />
</exec>
</target>
Performance Integration/Load Testing – Quest PerformaSure
Quest Management Server
Tag-and-Follow™
Performance Integration/Load Testing – Quest PerformaSure
Business Value Benefits
− Reduce R&D and QA time in performance testing
− Better performing applications
Results
− Faster time-to-market = more time to add features that give you a competitive advantage over your competition
− Higher quality applications = higher customer satisfaction
Next Steps…
Download one of my whitepapers from www.quest.com under Application Management, solutions for Java:
− Extending the Continuous Integration Environment with Performance Testing (Part 1)
− Achieving Continuous Performance Management (Part II)
− Best Practices in Java Environment Performance Testing
− A Formal Performance Tuning Methodology: Wait-Based Tuning
Sign up for my upcoming “Java Performance Winner’s Circle” webcast series on performance testing, tuning, and management at www.quest.com under Application Management, solutions for Java (starts in March 2008)
Watch my Continuous Integration OnSoftware Video Podcast on www.informit.com (or iTunes)
Read my forthcoming digital shortcut book, Agile Java Development with Test-Driven Development and Continuous Integration, on www.informit.com
Read my latest book, Pro Java EE 5 Performance Management and Optimization, at your favorite book store
Next Steps cont’d…
Quest Software Product Information:
JProbe – www.quest.com/jprobe
PerformaSure – www.quest.com/performasure Other Resources:
JProbe Live Product Demo – www.quest.com/events
PerformaSure Product Demand – www.quest.com/performasure Contact Quest Software Sales:
www.quest.com/company/contact-us.aspx
State of the Market Test-Driven Development Continuous Integration Continuous Performance Management CPM in Practice Business Value Next Steps…
Thank You -- Q&A
Agenda