JavaOne - Performance Focused DevOps to Improve Cont Delivery
71
1 FROM CHAOS TO CONT WEB DELIVERY THROUGH FOCUS ON PERFORMANCE Andreas Grabner http://apmblog.compuware.com @grabnerandi
-
Upload
andreas-grabner -
Category
Technology
-
view
1.140 -
download
0
description
These are the slides of my JavaOne presentation. The abstract goes like this: How do companies developing business-critical Java enterprise Web applications increase releases from 40 to 300 per year and still remain confident about a spike of 1,800 percent in traffic during key events such as Super Bowl Sunday or Cyber Monday? It takes a fundamental change in culture. Although DevOps is often seen as a mechanism for taming the chaos, adopting an agile methodology across all teams is only the first step. This session explores best practices for continuous delivery with higher quality for improving collaboration between teams by consolidating tools and for reducing overhead to fix issues. It shows how to build a performance-focused culture with tools such as Hudson, Jenkins, Chef, Puppet, Selenium, and Compuware APM/dynaTrace
Transcript of JavaOne - Performance Focused DevOps to Improve Cont Delivery
11
FROM CHAOS TO CONT WEB DELIVERY
THROUGH FOCUS ON PERFORMANCE
Andreas Grabner
http://apmblog.compuware.com
@grabnerandi
2
3
4
5
Testing is Important – and gives Confidence
6
But are we ready for “The Real” world?
7
Measure Performance during the game
Ball Possession: 40 : 60Fouls: 0 : 0Score: 0 : 0
Minute 1 - 5
8
Measure Performance during the game
Minute 6 - 35
Ball Possession: 80 : 20Fouls: 2 : 12Score: 0 : 0
9
Deep Dive Analysis
10
Options “To Fix” the situation
11
Not always a happy ending
Minute 90
Ball Possession: 80 : 20Fouls: 4 : 25Score: 3 : 0
12
FRUSTRATED FANS!!
12
13
How does that relate to
Software?
How does that relate to
Software?
1414
From Deploy to …From Deploy to …
Deploy Promotion/Event Problems Ops Playbook War Room
Timeline
1515
The “War Room” – back then
'Houston, we have a problem‘NASA Mission Control Center, Apollo 13, 1970
The “War Room” – back then
'Houston, we have a problem‘NASA Mission Control Center, Apollo 13, 1970
1616
The “War Room” – NOW
Facebook – December 2012
The “War Room” – NOW
Facebook – December 2012
1717
Problem: Unclear End User Problem DescriptionsProblem: Unclear End User Problem Descriptions
1818
Statuc Quo: Ops Runbook – System UnresponsiveStatuc Quo: Ops Runbook – System Unresponsive
1919
Problem: Unclear Ops Problem DescriptionsProblem: Unclear Ops Problem Descriptions
2020
Status Quo: Ops Runbook – High Resource UsageStatus Quo: Ops Runbook – High Resource Usage
2121Lack of data?Lack of data?
2222
23
Answers to the right questionsAnswers to the right questions
2424
What are the real questions?What are the real questions?
Individual Users? ALL users?
Is it the APP? Or Delivery Chain?
Code problem? Infrastructure?
One transaction? ALL transactions?
In AppServer? In Virtual Machine?
2525
Problem: What Devs would like to haveProblem: What Devs would like to have
2626
Problem: What Devs would like to haveProblem: What Devs would like to have
Top Contributor is related to String handling
99% of that time comes from RegEx Pattern Matching
Page Rendering is the main component
2727
Its getting this …Its getting this …Its like getting this …
28
… when you need to see this!
2929
RECAP Status Quo: We don’t like “War Rooms”RECAP Status Quo: We don’t like “War Rooms”
3030
Problem: Attitudes like this don’t help eitherProblem: Attitudes like this don’t help either
Image taken from https://www.scriptrock.com/blog/devops-whats-hype-about/
Shopzilla CIO (in 2010): “… when they get in the war room - the developers and ops teams describe the problem as the enemy, not each other”
3131
Problem: Very “expensive” to work on these issuesProblem: Very “expensive” to work on these issues
~80% of problems
caused by ~20% patterns
YES we know this
80% Dev Time in Bug Fixing
$60B Defect Costs
BUT
3232
TOP PROBLEM PATTERNS
• Focus on Web and Java
3333
Top Problem Patterns: Resource PoolsTop Problem Patterns: Resource Pools
3434
Top Problem Patterns: Resource PoolsTop Problem Patterns: Resource Pools
3535
Deployment Mistakes lead to internal ExceptionsDeployment Mistakes lead to internal Exceptions
3636
Deployment Mistakes lead to high logging overheadDeployment Mistakes lead to high logging overhead
3737
Production Deployment leads to Log SYNC IssuesProduction Deployment leads to Log SYNC Issues
3838
Long running SQL with Production DataLong running SQL with Production Data
3939
N+1 Query ProblemN+1 Query Problem
4040
Reading and processing too much data in AppReading and processing too much data in App
4141
Memory Leaks in Cache Layer with Production DataMemory Leaks in Cache Layer with Production Data
Still crashes
Problem fixed!Fixed Version Deployed
4242
Synchronization Issues under real loadSynchronization Issues under real load
4343
BLOATED Web SitesBLOATED Web Sites
17! JS Files – 1.7MB in Size
Useless Information!Even might be a security risk!
4444
Missing or incorrect configured browser cachesMissing or incorrect configured browser caches
62! Resources not cached
49! Resources with short expiration
4545
SLOW or Failing 3rd Party ContentSLOW or Failing 3rd Party Content
4646
Want MORE of these and more details?Want MORE of these and more details?
http://apmblog.compuware.com
4747
Lots of Problems that could have been avoidedLots of Problems that could have been avoided
• BUT WHY are they still making it to Production?
?
4848
Missing Focus on PerformanceMissing Focus on Performance
4949
Different Goals for Dev and OpsDifferent Goals for Dev and Ops
5050
Disconnected Teams despite “Shared Responsibility”Disconnected Teams despite “Shared Responsibility”
5151
5252
How to make the Enterprise Crew happy?How to make the Enterprise Crew happy?
?
5353
5454
Solution: DevOps + Performance FocusSolution: DevOps + Performance Focus
Culture“Shared Responsibility”
Agile Process for ALL TeamsPerformance as Key Requirement
X-Team Collaboration and Education
AutomationMeasurement, Collaboration and Deployment
Automate Performance and Architectural Problem Detection
Measurement“Visible” KPIs for each Team
Focus on Performance, Architectural and Deployment Measures
SharingExpertise, Tool and Data Sharing
“Easy” sharing of Performance, Deployment and Production Data
http://www.opscode.com/blog/2010/07/16/what-devops-means-to-me/
5555
Culture: EXTEND Requirements with …Culture: EXTEND Requirements with …
PerformancePerformance ScalabilityScalability
TestabilityTestability
DeployabilityDeployability
DeployabilityDeployability
5656
Sharing: DON’T EXCLUDE anyone from Agile ProcessSharing: DON’T EXCLUDE anyone from Agile Process
Stand-UpsStand-Ups Sharing ToolsSharing Tools
FeedbackFeedback
5757
Measurement: Define KPIs accepted by all teamsMeasurement: Define KPIs accepted by all teams
# of SQL Executions
# of Log Lines
MBs / Uses
Time for Deployment
Time for Rollback
Response TimesPerf Test Code Coverage
5858
AUTOMATION, AUTOMATION, AUTOMATIONAUTOMATION, AUTOMATION, AUTOMATION
PerformancePerformance ScalabilityScalability
Shared ToolsShared Tools Automatic FeedbackAutomatic Feedback
5959
DevOps Collaboration – TODO LIST FOR YOU!!DevOps Collaboration – TODO LIST FOR YOU!!
Perf
Test
Framew
orks
Access to Production Data
Shared Reporting and Task Management
Deployment Tools
Diagnostic Tools
Shared Performance KPIs and Tooling
Diagnosti
c Tools Test in Prod and Staging
Known How Exchange
6060
Recap – Problem – Root Cause – Solution - ResultRecap – Problem – Root Cause – Solution - Result
DevOps + Performance Culture
Automation
Measurement
Collaboration
6161
TIPS FOR DEVS
6262
Performance Focus in Test AutomationPerformance Focus in Test Automation
12 0 120ms
3 1 68ms
Build 20 testPurchase OK
testSearch OK
Build 17 testPurchase OK
testSearch OK
Build 18 testPurchase FAILED
testSearch OK
Build 19 testPurchase OK
testSearch OK
Build # Test Case Status # SQL # Excep CPU
12 0 120ms
3 1 68ms
12 5 60ms
3 1 68ms
75 0 230ms
3 1 68ms
Test Framework Results Architectural Data
We identified a regresesion
Problem solved
Lets look behind the scenes
Exceptions probably reason for failed tests
Problem fixed but now we have an architectural regression
Problem fixed but now we have an architectural regression
Now we have the functional and architectural confidence
6363
Performance Focus in Test AutomationPerformance Focus in Test Automation
Analyzing All Unit / Performance Tests
Analyzing Metrics such as DB Exec
Count
Jump in DB Callsfrom one Build to the next
6464
Performance Focus in Test AutomationPerformance Focus in Test Automation
Cross Impact of KPIs
6565
Performance Focus in Test AutomationPerformance Focus in Test Automation
Embed your Architectural Results in Jenkins
6666
Performance Focus in Test AutomationPerformance Focus in Test Automation
Here is the difference!
Compare Build that shows BAD Behavior! With Build that shows GOOD Behavior!
6767
Performance Focus in Test AutomationPerformance Focus in Test Automation
CalculateUserStats is the new Plugin that causes problems
6868
Remember – DevOps requires Cultural ChangeRemember – DevOps requires Cultural Change
ShareShare IntegrateIntegrate
CollaborateCollaborate PerformancePerformance
6969
Elevate our DevOps Investment - REDUCEElevate our DevOps Investment - REDUCE
80% Dev Time for Bug Fixing
$60B Costs by Defects
70© 2011 Compuware Corporation — All Rights Reserved © 2011 Compuware Corporation — All Rights Reserved
70
Participate in Compuware APM Discussion Forumsapmcommunity.compuware.com
Like us on Facebookfacebook.com/CompuwareAPM
Join our LinkedIn groupCompuware APM User Group
Follow us on Twittertwitter.com/CompuwareAPM
Read our BlogAbout:Performance
Watch our Videos & product Demosyoutube.com/Compuware
www.compuware.com/APM
Thank YouThank You
71© 2011 Compuware Corporation — All Rights Reserved
Simply Smarter
