Web Application Performance Using Six Sigma
Transcript of Web Application Performance Using Six Sigma
-
8/14/2019 Web Application Performance Using Six Sigma
1/48
-
8/14/2019 Web Application Performance Using Six Sigma
2/48
2
We will talk about
What is Quality?
Types of Performance Measurement
How to Measure Web App Performance? Traps we fall into and how Six Sigma
showed us the right path
Overview of Six Sigma
Using Six sigma to Improve Performance
5 Steps to Improve Web App Performance Ensuring problems do not come back
Questions
-
8/14/2019 Web Application Performance Using Six Sigma
3/48
3
Whats the biggest problem today
Todays Market demand best of everything
Quality
Cost
Schedule
Reliable, Responsive and Secure service 24X7is the demand of the global customer
Schedule
-
8/14/2019 Web Application Performance Using Six Sigma
4/48
4
Defects are inevitable
Fact: No software can be guaranteed100% defect-free
Action: No action
Result: We make it horribly true
Ask: Why the defect happened?
Do: Analyze data and improve theprocess to prevent it
And Sustain it
Six Sigma
-
8/14/2019 Web Application Performance Using Six Sigma
5/48
5
What is Quality?
Meets expectations Serves the Purpose / Needs
Intuitive / Usability Reliable
Responsive / Performance Security / Privacy
High Quality/Low Defect
Getting it right the first time, every time
Think Global
-
8/14/2019 Web Application Performance Using Six Sigma
6/48
Why is Performance important?
More online activities
Competitors are faster
Users have more options
Attracting and retaining users
We cannot overcome speed of light
Release it and then fix it no longer
an option we may loose Mind-Shareand we may not get second chance
6
-
8/14/2019 Web Application Performance Using Six Sigma
7/48
Solving Performance Problems
Your app is having performance issues
Traditional way
Put more people
Do more performance testing
But Performance Testing tools
Outcome/Results
We often march on a journey to solve a problem withoutunderstanding the problem
If we dont know what we are improving, how would weknow if we have improved?
Structured way of solving the problem
Use Six Sigma, we can define the problem, measure theright thing, analyze root cause and then solve the right
problem the right way
7
-
8/14/2019 Web Application Performance Using Six Sigma
8/48
What is Six Sigma?
Structured Problem solving methodology
Solving the right problem the right way
Focus on finding and fixing the root cause Ensuring problem does not come back
Drive Continuous Improvement
DMAIC
8
-
8/14/2019 Web Application Performance Using Six Sigma
9/48
1.DEFINE
2.MEASURE
3.ANALYZE4.IMPROVE
5.CONTROL
(SUSTAIN)
Identify major userscenarios & Perf
Goals
Use right set of toolsto measure & baseline
current Performance
Fix root cause of perfproblems in your code/architecture/deployment
Monitor to
ensure perf donot degrade withfuture changes tothe code
Measure the followingPage Load Time (1 & 2)Time to First Byte# of round tripsAvg Bytes Download
WebPagePerformance
ImprovementDMAIC cycle
Analyze data, find reasons/root-cause for slow perfConsult Perf Experts
-
8/14/2019 Web Application Performance Using Six Sigma
10/48
Six Sigma - DMAIC
Define (D) : Zero in on specific problem with defined return on effort
Measure (M) : Determine current performance of process
Analyze (A) : Validate key drivers of performance (root cause of problem) Improve (I) : Improved performance and validated realized results
Control (C) : Implement controls to ensure continued performance
10
ControlDefine Measure Analyze Improve
Lock in the results(control plan)
Mistake proofingControl pointsMonitoring PlanPositive hand offof control plan
Financial impact of the
project
Project selection Project charter
Critical to Customer(CTQ) needs
High level process map
Key output variables(metrics or Ys)
Possible causes ofdefects (Xs)
Data collection andpresentation plan
Current Performance Internal/ external
benchmarking
Key causes (vital few)of defects (Xs)
Improvement strategy Prioritize solutions
Tested & measuredsolutions
Final solutions
Project Phases and Deliverables
-
8/14/2019 Web Application Performance Using Six Sigma
11/48
11
The Six Sigma Project - challenges
Bottom-up approach
Difficult to get buy-in
Myth: Six Sigma cannot be used for software We dont need a Six Sigma project to
understand this problem and to fix it
Its common sense we can fix it
Do it if you have time on your personal time
Measuring the right things Avoiding temptation of jumping to solution
-
8/14/2019 Web Application Performance Using Six Sigma
12/48
Six Sigma Define Phase
Project Selection The right project (Strategic, ROI, etc.)
Project Charter
Why are we doing this? Business Case
Goal, Scope, Time-line
Exec Sponsors, Team & Resources
Web App XYZ (which drives 40% of revenue to the company) haveperformance problems (based on survey). This is impacting significant
% of users and we are seeing decline in users to the site. Customer Focus
Voice of Customer (VOC) what do customer need
Critical to quality (CTQ)
Customers have repeatedly complained about the performance of theXYZ app. They are not able to complete transactions in timely manner.
Current Situation
Process Map
Current Performance
12
-
8/14/2019 Web Application Performance Using Six Sigma
13/48
13
The Define Phase: Problem Statement
Testers did not do performance testing
Performance bugs were not fixed
Performance goals/expectation not clear
Performance of any product/featureshould managed in a better way
-
8/14/2019 Web Application Performance Using Six Sigma
14/48
14
The Define Phase: Goal
Ship Zero Performance Bugs
Find twice # of performance bug
Performance goals should be clear
Improve tests ability to find 95% ofthe performance bugs before Beta1
-
8/14/2019 Web Application Performance Using Six Sigma
15/48
Define Perf User Scenarios
Understand the User (Voice of the Customer) First time visitor / Guest user / authenticated user
Returning users with cache/no cache
User on the same site/session User from other MS sites/domains (w/ passport)
User Demographics (Geo, Home/Office, machine config,consumer/info worker/social, connection speed)
Typical user transactions, Back/Forward/Refresh usage
Perf goal
Regional competitor performance
What is acceptable performance
Do not use it should be fast, try JP broadband usersshould be able to get the page in 4 seconds (75th percentile), when
they visit for the first time (PLT1), 2 seconds for PLT2
15
-
8/14/2019 Web Application Performance Using Six Sigma
16/48
16
The Define Phase: Voice of Customer
Performance at-least as good as last release
Comparable to the competitors
UI Responsiveness
Notification in case of delays
I f it appears slow it is slow , irrespective ofwhat the data says
-
8/14/2019 Web Application Performance Using Six Sigma
17/48
17
The Define Phase: Critical To Quality (CTQ)
Examples:
For common scenarios, Improve performance by
atleast 10% (compared to last release)
Any action taking more than 3 seconds should havea progress bar/notification
Similar/predictable performance in any connectionmode (speed, latency)
Handle failures/major issues gracefully
-
8/14/2019 Web Application Performance Using Six Sigma
18/48
InternationalInternetRoutes
NotesMinimum bandwidth to be seen on this map is 14 GigsDoes not report bandwidth within a country
-
8/14/2019 Web Application Performance Using Six Sigma
19/48
Source data for timings is 75th percentile for country in question from: http://msncore/performance/netsmart/Netstats.asp
RT = Round TripsRed line is 34 RT
Green line is 21 RTDotted Green is 10
RT
Blue line is 3 RT
Latency and the impact to page loadtimes based on number of round trips
19Microsoft Confidential
http://msncore/performance/netsmart/Netstats.asphttp://msncore/performance/netsmart/Netstats.asp -
8/14/2019 Web Application Performance Using Six Sigma
20/48
20Microsoft Confidential
Network Round trip delays
-
8/14/2019 Web Application Performance Using Six Sigma
21/48
Performance/Load/Stress Testing
Performance testing User-Scenarios testing (typical case & best case) Establish Baseline & perform trend analysis Detect performance issues Tools: WebRunner, WANSim, HttpWatch, etc.
Load testing (volume/longevity/endurance)
Expected MAX # of concurrent users Volume of data Very Long active sessions
Stress testing (negative testing) What happens with the load exceeds significantly OR
system goes thru resource constraints/failures Does the system gracefully recovers from failure?
(e.g. Tiwan Earthquake in Dec 2006)
-
8/14/2019 Web Application Performance Using Six Sigma
22/48
Six Sigma - Measure
Measurement Process Data Collection Plan Testing, Monitoring, Measurement system analysis, sampling
Clear, concise definition of variables, process involved
Identify Key measures/drivers of performance Y = F(X1,X2,X3,X4,)
Ishikawa (Fishbone) diagram Cause & Effect diagram
Internal / External Benchmarking
Baseline Current performance & Impact Identify and measure current performance and its impact on
customer, collect more data if requiredOn 300kbps connection & 300ms Round-Trip-Time, it takes 6 seconds to load the
page for the PLT1 case, 2.5 seconds for PLT2. 20% of our user abandon the pagebefore it loads.
Web Page X have 2 HTML files, 3 .js files, 3.css, 5 images, the web App opens 2parallel tcp ports to download them.
22
-
8/14/2019 Web Application Performance Using Six Sigma
23/48
Types of Performance Measurements
Client UI Response Time Server Response Time
Load/Stress
Bytes over wire/Throughput Availability
Latency (anywhere in the world)
Browser Processing & Rendering time
User Machine - Resource utilization
Perceived Performance
If the user feels the product is slow, your product i s slow no matter what our data says
PLT1: The user visiting the site with NO CACHE PLT2: Returning user with CACHE
23
-
8/14/2019 Web Application Performance Using Six Sigma
24/48
What is Performance
PLT1: The user visiting the site withNO CACHE
PLT2: Returning user with CACHE
24Microsoft Confidential
-
8/14/2019 Web Application Performance Using Six Sigma
25/48
Measure
Business Measurements # of unique users
# of page views
Click-Thru-Rate (CTR)
Revenue
Market Share Net Promoter
Other Measurements
% of people on PLT1 / PLT2
Errors
Abandon rate (Incomplete/Closed/Click-Away)
25
-
8/14/2019 Web Application Performance Using Six Sigma
26/48
Measure: Key variables for Performance
Factors attributing to Web Page Performance # of Files, Static/Dynamic content
Page Load Time
Bytes download DNS Lookup time
Peak hours / Load & Stress
User spread / Global? Data Centers / CDN / Redirects
Multiple versions of the app
Web Page Architecture (parallel/sequential download)
Compression
Expiry dates
Keep-Alive
PLT1 / PLT2 (Caching?)
26
-
8/14/2019 Web Application Performance Using Six Sigma
27/48
Performance Cause-Effect Diagram
27
End-UserPage Load Time
Server/ Infrastructure
Application / Web Page End-User
Latency
Bandwidth
Static/Dynamic content
Total Bytes/ Compression
GeoLocation DataCenter/CDN
# of Servers
Machine config# of Files
JS/CSS processing
Load/Stress Peak Hours
DNS Setup
Caching/Keep-Alive
# of TCP ports
Page Architecture
App versioning
Best Practices @ http://MSNPerf
http://msnperf/http://msnperf/ -
8/14/2019 Web Application Performance Using Six Sigma
28/48
Six Sigma - Analyze
Find Root Cause of the problems Analyze the data from Measure phase
Identify vital few variables (x)
Perform correlation and regression analysis Data stratification
Use 5 Why techniques
Hypothesis testing Sources of variation
Use Cause-Effect diagram
Plot data on graphs (trend, releases)
Special Cause / Common Cause# of files on the site is 10, some of these files can be combined, 2 files are notcompressed. Majority of the users who abandon the site are from UK (Latency)and Dial-up users from US (slow connection). The problem started to happen fromMarch 01 (when release 2.2 went live)
28
-
8/14/2019 Web Application Performance Using Six Sigma
29/48
AnalysisFrom the data it looks like 17% of the users are in PLT1 and 83% in PLT2There are several users in a middle category (16K to 21K)
PLT1/PLT2 User Distribution
29
-
8/14/2019 Web Application Performance Using Six Sigma
30/48
Impact of slow performance on user
As it takes more time to display page users STOP the pageTypically after 5 seconds of wait, 15% user stops the page from loading and the % grows with thetime 30Microsoft Confidential
-
8/14/2019 Web Application Performance Using Six Sigma
31/48
User count & Performance across states
31
-
8/14/2019 Web Application Performance Using Six Sigma
32/48
Bytes & Page Load Time distribution
32
-
8/14/2019 Web Application Performance Using Six Sigma
33/48
-
8/14/2019 Web Application Performance Using Six Sigma
34/48
Six Sigma - Improve
Improvement strategy & Plan
Improvement solution selection
Generate ideas Involve diverse team Identify and rank solution alternatives
Pilot solutions and select final solution
FMEA Design of experiments
Test and implement final solution
Communication plan
ToBe Process Map
Track improvement, monitor trend Share success/failure stories Best Practices
34
-
8/14/2019 Web Application Performance Using Six Sigma
35/48
Six Sigma Control (Sustain improvements)
Process Monitoring/Control Plan Mistake Proofing (Poka-Yoke)
Control Chart
Response Plan
Document standard process/procedure
Train resources Share Learning
Project closure Positive Handoff
Measure Benefits (Financial / Customer Sat)
Celebrate35
-
8/14/2019 Web Application Performance Using Six Sigma
36/48
36
Sample Control Plan
We will get user feedback on Beta1 build and evaluate if weneed any more improvements
We will continue to have meeting (every other week) to
discuss any perf test related issues
Person ABC will do the measurements and monitor the appperformance until Beta1 and analyze/report user feedback
A monthly status mail will be sent to (10th of each month) theProject team, Product Manager and VP, reporting the statsrelated to the project (Perf for various category of users,
improvement over time, survey/feedback from users, etc.)
If we find that we are deviating from our goal, we will call ameeting (within 2 days) and analyze problem and develop
solution
-
8/14/2019 Web Application Performance Using Six Sigma
37/48
37
Lessons Learned / Suggestions
Lessons Learned:
Six Sigma Process Measurements helped us uncover
problems in a structured way and come up withsolutions to eliminate them or minimize its impact.
Implementing improvements requires involvement
from everyone at all levels and all disciplines
Suggestions:
By getting all disciplines (dev/pm/test) involved wecan focus on preventing the problems rather thanrelying on finding and fixing them.
Integrate new practices into the development cycleonce the improvements have been validated
-
8/14/2019 Web Application Performance Using Six Sigma
38/48
Results
Note: Specific % & # are purposely removed from this presentation
Product benefits:
Page size drastically reduced Site performance improved
User satisfaction improved
Increased click-thru-rate
Overall impact:
Increased focus on Performance More investment in performance
(expanding the performance group)
38
-
8/14/2019 Web Application Performance Using Six Sigma
39/48
Visual Round Trip Analyzer
39
Vi l R d T i A l
-
8/14/2019 Web Application Performance Using Six Sigma
40/48
40
Visual Round Trip Analyzer
-
8/14/2019 Web Application Performance Using Six Sigma
41/48
Visual Round-Trip Analyzer (VRTA)
http://msncore/performance/netsmart/VRTA_animation/sample1/main.html
File Transfer Duration
Client Port Browser toServer Connection
Time in Seconds
BandwidthUtilization
Servers
Color codedBy File Type
http://msncore/performance/netsmart/VRTA_animation/sample1/main.htmlhttp://msncore/performance/netsmart/VRTA_animation/sample1/main.html -
8/14/2019 Web Application Performance Using Six Sigma
42/48
Bad Good
42
Use more parallel TCP ports
-
8/14/2019 Web Application Performance Using Six Sigma
43/48
Unblock Java Script
43
Standard JS downloads in Serial andcreates bandwidth bottlenecks
Use a binding methodology to getaround this issue
bl k
-
8/14/2019 Web Application Performance Using Six Sigma
44/48
Unblock JS
44
Solution IThis method has been
successfully used by the
Windows Live Hotmail Team==================function AsyncLoad(){var l = arguments.length;for (var i=0;i
-
8/14/2019 Web Application Performance Using Six Sigma
45/48
Expiration dates
45
Max-Age: Supersedes Expiration
RelativeTime
URI
ContentLen Status Code
0.00http://groups.msn.com/people/ 27659 200 -- OK
0.70http://c.msn.com/c.gif 42 200 -- OK
4.53http://groups.msn.com/global/css.htm 0 304 -- Not Modified
4.88http://groups.msn.com/spacer.gif 0 304 -- Not Modified
4.89http://groups.msn.com/home_icons_chat_48x40.gif 0 304 -- Not Modified
4.90http://www.match.com/msnprofile/profile.aspx 2481 200 -- OK5.22http://groups.msn.com/home_icons_IM_48x40.gif 0 304 -- Not Modified
5.55http://groups.msn.com/msnmess_themes_65x60.gif 0 304 -- Not Modified
5.57http://groups.msn.com/home_icons_heart_42x39.gif 0 304 -- Not Modified
5.81http://www.match.com/lib.msnprofiles.master.style.css 383 200 -- OK
5.81http://www.match.com/lib.msnprofiles.style1.css 214 200 -- OK
5.89http://groups.msn.com/home_icons_MD_48x40.gif 0 304 -- Not Modified
5.91http://groups.msn.com/home_icons_groups_48x40.gif 0 304 -- Not Modified
6.20http://www.match.com/libraries/lib.template.globaljs.js 0 304 -- Not Modified6.22http://groups.msn.com/msn_phone3n_48x40.gif 0 304 -- Not Modified
6.24http://groups.msn.com/whitepages/msnwhitepages.htm 0 304 -- Not Modified
6.57http://groups.msn.com/home_icons_msnbutterfly_48x40_c.gif 0 304 -- Not Modified
6.61http://xml.eshop.msn.com/xmlbuddy/eShopOffer.aspx 1227 200 -- OK
6.86https://reader010.{domain}/reader010/html5/0625/5b30eaf68f5ad/5b30eb0ce01a1.jpg 8708 200 -- OK
7.35http://groups.msn.com/match_com_header_blue_matte.gif 0 304 -- Not Modified
7.35http://groups.msn.com/spacer.gif 0 304 -- Not Modified
7.36http://view.atdmt.com/MSN/iview/msnnkhac001300x250xWBCK4000109msn/direct;wi.300;hi.250/01 320 200 -- OK
7.43http://rad.msn.com/ADSAdClient31.dll 489 200 -- OK
7.69http://groups.msn.com/1195B_goBtn.gif 0 304 -- Not Modified
7.71http://groups.msn.com/gfol_180x150_survey_express_jan03_2.gif 0 304 -- Not Modified
8.91http://att.atdmt.com/b/MSMSNMATCVON/Harmonics_WorstNightmare_2499_300x250.gif 14852 200 -- OK
10.58 56375
304 = Bad!
U C i
http://groups.msn.com/people/http://c.msn.com/c.gifhttp://groups.msn.com/Editorial/en-us/Content/global/css.htmhttp://groups.msn.com/Editorial/en-us/images/spacer.gifhttp://groups.msn.com/Editorial/en-us/images/home_icons_chat_48x40.gifhttp://www.match.com/msnprofile/profile.aspxhttp://groups.msn.com/Editorial/en-us/images/home_icons_IM_48x40.gifhttp://groups.msn.com/editorial/en-us/images/msnmess_themes_65x60.gifhttp://groups.msn.com/Editorial/en-us/images/home_icons_heart_42x39.gifhttp://www.match.com/stylesheets/msnprofiles/lib.msnprofiles.master.style.csshttp://www.match.com/stylesheets/msnprofiles/lib.msnprofiles.style1.csshttp://groups.msn.com/Editorial/en-us/images/home_icons_MD_48x40.gifhttp://groups.msn.com/Editorial/en-us/images/home_icons_groups_48x40.gifhttp://www.match.com/libraries/lib.template.globaljs.jshttp://groups.msn.com/Editorial/en-us/images/msn_phone3n_48x40.gifhttp://groups.msn.com/Editorial/en-us/Content/whitepages/msnwhitepages.htmhttp://groups.msn.com/Editorial/en-us/images/home_icons_msnbutterfly_48x40_c.gifhttp://xml.eshop.msn.com/xmlbuddy/eShopOffer.aspxhttp://images.match.com/match/matchscene/articles/spotlight1617.jpghttp://groups.msn.com/Editorial/en-us/images/match_com_header_blue_matte.gifhttp://groups.msn.com/editorial/en-us/images/spacer.gifhttp://view.atdmt.com/MSN/iview/msnnkhac001300x250xWBCK4000109msn/direct;wi.300;hi.250/01http://rad.msn.com/ADSAdClient31.dllhttp://groups.msn.com/editorial/en-us/images/1195B_goBtn.gifhttp://groups.msn.com/editorial/en-us/images/gfol_180x150_survey_express_jan03_2.gifhttp://att.atdmt.com/b/MSMSNMATCVON/Harmonics_WorstNightmare_2499_300x250.gifhttp://att.atdmt.com/b/MSMSNMATCVON/Harmonics_WorstNightmare_2499_300x250.gifhttp://groups.msn.com/editorial/en-us/images/gfol_180x150_survey_express_jan03_2.gifhttp://groups.msn.com/editorial/en-us/images/1195B_goBtn.gifhttp://rad.msn.com/ADSAdClient31.dllhttp://view.atdmt.com/MSN/iview/msnnkhac001300x250xWBCK4000109msn/direct;wi.300;hi.250/01http://groups.msn.com/editorial/en-us/images/spacer.gifhttp://groups.msn.com/Editorial/en-us/images/match_com_header_blue_matte.gifhttp://images.match.com/match/matchscene/articles/spotlight1617.jpghttp://xml.eshop.msn.com/xmlbuddy/eShopOffer.aspxhttp://groups.msn.com/Editorial/en-us/images/home_icons_msnbutterfly_48x40_c.gifhttp://groups.msn.com/Editorial/en-us/Content/whitepages/msnwhitepages.htmhttp://groups.msn.com/Editorial/en-us/images/msn_phone3n_48x40.gifhttp://www.match.com/libraries/lib.template.globaljs.jshttp://groups.msn.com/Editorial/en-us/images/home_icons_groups_48x40.gifhttp://groups.msn.com/Editorial/en-us/images/home_icons_MD_48x40.gifhttp://www.match.com/stylesheets/msnprofiles/lib.msnprofiles.style1.csshttp://www.match.com/stylesheets/msnprofiles/lib.msnprofiles.master.style.csshttp://groups.msn.com/Editorial/en-us/images/home_icons_heart_42x39.gifhttp://groups.msn.com/editorial/en-us/images/msnmess_themes_65x60.gifhttp://groups.msn.com/Editorial/en-us/images/home_icons_IM_48x40.gifhttp://www.match.com/msnprofile/profile.aspxhttp://groups.msn.com/Editorial/en-us/images/home_icons_chat_48x40.gifhttp://groups.msn.com/Editorial/en-us/images/spacer.gifhttp://groups.msn.com/Editorial/en-us/Content/global/css.htmhttp://c.msn.com/c.gifhttp://groups.msn.com/people/ -
8/14/2019 Web Application Performance Using Six Sigma
46/48
Use Compression
46
Types of Compression: Static, Dynamic
Levels 0 through 10
26KB/5.5 = 5KB; 20KB BW savings
K Ali TCP t
-
8/14/2019 Web Application Performance Using Six Sigma
47/48
Keep-Alive TCP ports
47
-
8/14/2019 Web Application Performance Using Six Sigma
48/48
Questions?
Mukesh [email protected]
mailto:[email protected]:[email protected]