Performance Quality Metrics for Mobile Web and Mobile Native - Agile Testing Days 2014
-
Upload
andreas-grabner -
Category
Software
-
view
334 -
download
0
description
Transcript of Performance Quality Metrics for Mobile Web and Mobile Native - Agile Testing Days 2014
![Page 1: Performance Quality Metrics for Mobile Web and Mobile Native - Agile Testing Days 2014](https://reader033.fdocuments.net/reader033/viewer/2022060121/559452bf1a28abce4f8b46cd/html5/thumbnails/1.jpg)
Performance Quality Metrics for Mobile Web and Mobile Native
http://bit.ly/atd2014challenge
@grabnerandi
![Page 2: Performance Quality Metrics for Mobile Web and Mobile Native - Agile Testing Days 2014](https://reader033.fdocuments.net/reader033/viewer/2022060121/559452bf1a28abce4f8b46cd/html5/thumbnails/2.jpg)
Why?
What?
![Page 3: Performance Quality Metrics for Mobile Web and Mobile Native - Agile Testing Days 2014](https://reader033.fdocuments.net/reader033/viewer/2022060121/559452bf1a28abce4f8b46cd/html5/thumbnails/3.jpg)
It allows us to Evolve from …
![Page 4: Performance Quality Metrics for Mobile Web and Mobile Native - Agile Testing Days 2014](https://reader033.fdocuments.net/reader033/viewer/2022060121/559452bf1a28abce4f8b46cd/html5/thumbnails/4.jpg)
Some Examples from Real Life
300 Deployments / Year
50-60 Deployments / Day
10+ Deployments / Day
Every 11.6 seconds
![Page 5: Performance Quality Metrics for Mobile Web and Mobile Native - Agile Testing Days 2014](https://reader033.fdocuments.net/reader033/viewer/2022060121/559452bf1a28abce4f8b46cd/html5/thumbnails/5.jpg)
More details on Amazon
75% fewer outages since 2006
90% fewer outage minutes
~0.001% of deployments cause a problem
Instantaneous automatic rollback
Deploying every 11.6s
![Page 6: Performance Quality Metrics for Mobile Web and Mobile Native - Agile Testing Days 2014](https://reader033.fdocuments.net/reader033/viewer/2022060121/559452bf1a28abce4f8b46cd/html5/thumbnails/6.jpg)
The “War Room” – back then
'Houston, we have a problem‘NASA Mission Control Center, Apollo 13, 1970
![Page 7: Performance Quality Metrics for Mobile Web and Mobile Native - Agile Testing Days 2014](https://reader033.fdocuments.net/reader033/viewer/2022060121/559452bf1a28abce4f8b46cd/html5/thumbnails/7.jpg)
The “War Room” – NOW
Facebook – December 2012
![Page 8: Performance Quality Metrics for Mobile Web and Mobile Native - Agile Testing Days 2014](https://reader033.fdocuments.net/reader033/viewer/2022060121/559452bf1a28abce4f8b46cd/html5/thumbnails/8.jpg)
5 Situations on
WHY this happens,
HOW to avoid it,
Metrics to look at
![Page 9: Performance Quality Metrics for Mobile Web and Mobile Native - Agile Testing Days 2014](https://reader033.fdocuments.net/reader033/viewer/2022060121/559452bf1a28abce4f8b46cd/html5/thumbnails/9.jpg)
![Page 10: Performance Quality Metrics for Mobile Web and Mobile Native - Agile Testing Days 2014](https://reader033.fdocuments.net/reader033/viewer/2022060121/559452bf1a28abce4f8b46cd/html5/thumbnails/10.jpg)
Don’t assume
You know the environment
![Page 11: Performance Quality Metrics for Mobile Web and Mobile Native - Agile Testing Days 2014](https://reader033.fdocuments.net/reader033/viewer/2022060121/559452bf1a28abce4f8b46cd/html5/thumbnails/11.jpg)
1.3 Million iOS Apps
$ 10 billion iOS purchases
2013
1.4 million Android Apps
$ 5 billion Google Payout within 1 year
![Page 12: Performance Quality Metrics for Mobile Web and Mobile Native - Agile Testing Days 2014](https://reader033.fdocuments.net/reader033/viewer/2022060121/559452bf1a28abce4f8b46cd/html5/thumbnails/12.jpg)
Distance Calculation Issues
480km biking in 1 hour!
Solution: Unit Test in Live App reports Geo
Calc Problems
Finding: Only happens on certain
Android versions
![Page 13: Performance Quality Metrics for Mobile Web and Mobile Native - Agile Testing Days 2014](https://reader033.fdocuments.net/reader033/viewer/2022060121/559452bf1a28abce4f8b46cd/html5/thumbnails/13.jpg)
Bluetooth 4.0 Issue on AndroidOS Issue
crashing App
![Page 14: Performance Quality Metrics for Mobile Web and Mobile Native - Agile Testing Days 2014](https://reader033.fdocuments.net/reader033/viewer/2022060121/559452bf1a28abce4f8b46cd/html5/thumbnails/14.jpg)
3rd Party Issues
Impact of bad 3rd party calls
![Page 15: Performance Quality Metrics for Mobile Web and Mobile Native - Agile Testing Days 2014](https://reader033.fdocuments.net/reader033/viewer/2022060121/559452bf1a28abce4f8b46cd/html5/thumbnails/15.jpg)
Slow Network – Bad Latency
Impact of Latency and Bandwidth
![Page 16: Performance Quality Metrics for Mobile Web and Mobile Native - Agile Testing Days 2014](https://reader033.fdocuments.net/reader033/viewer/2022060121/559452bf1a28abce4f8b46cd/html5/thumbnails/16.jpg)
Metrics: Crashes, Exceptions, # and Status of 3rd Party Calls, Payload of
Web Service Calls
Dev: Build for Mobile
Test: Test on Mobile and Diff. Carriers
Ops: Monitor Mobile
![Page 17: Performance Quality Metrics for Mobile Web and Mobile Native - Agile Testing Days 2014](https://reader033.fdocuments.net/reader033/viewer/2022060121/559452bf1a28abce4f8b46cd/html5/thumbnails/17.jpg)
![Page 18: Performance Quality Metrics for Mobile Web and Mobile Native - Agile Testing Days 2014](https://reader033.fdocuments.net/reader033/viewer/2022060121/559452bf1a28abce4f8b46cd/html5/thumbnails/18.jpg)
#Push without a Plan
![Page 19: Performance Quality Metrics for Mobile Web and Mobile Native - Agile Testing Days 2014](https://reader033.fdocuments.net/reader033/viewer/2022060121/559452bf1a28abce4f8b46cd/html5/thumbnails/19.jpg)
434 Resources in total on that page:230 JPEGs, 75 PNGs, 50 GIFs, …
Total size of ~ 20MB
Mobile Landing Page of Super Bowl Ad
![Page 20: Performance Quality Metrics for Mobile Web and Mobile Native - Agile Testing Days 2014](https://reader033.fdocuments.net/reader033/viewer/2022060121/559452bf1a28abce4f8b46cd/html5/thumbnails/20.jpg)
m.store.com redirects to www.store.com
ALL CSS and JS files are
redirected to the www domain
This is a lot of time “wasted” especially on high latency mobile
connections
![Page 21: Performance Quality Metrics for Mobile Web and Mobile Native - Agile Testing Days 2014](https://reader033.fdocuments.net/reader033/viewer/2022060121/559452bf1a28abce4f8b46cd/html5/thumbnails/21.jpg)
Metrics: Load Time, # Resources (Images, …),
# HTTP 3xx, 4xx, 5xx
Dev: Build for Mobile
Test: Test on Mobile
Ops: Monitor Mobile
![Page 22: Performance Quality Metrics for Mobile Web and Mobile Native - Agile Testing Days 2014](https://reader033.fdocuments.net/reader033/viewer/2022060121/559452bf1a28abce4f8b46cd/html5/thumbnails/22.jpg)
![Page 23: Performance Quality Metrics for Mobile Web and Mobile Native - Agile Testing Days 2014](https://reader033.fdocuments.net/reader033/viewer/2022060121/559452bf1a28abce4f8b46cd/html5/thumbnails/23.jpg)
Architectural Decisions gone
Bad
![Page 24: Performance Quality Metrics for Mobile Web and Mobile Native - Agile Testing Days 2014](https://reader033.fdocuments.net/reader033/viewer/2022060121/559452bf1a28abce4f8b46cd/html5/thumbnails/24.jpg)
We wanted Web 2.0 and Mobile Ready!
![Page 25: Performance Quality Metrics for Mobile Web and Mobile Native - Agile Testing Days 2014](https://reader033.fdocuments.net/reader033/viewer/2022060121/559452bf1a28abce4f8b46cd/html5/thumbnails/25.jpg)
![Page 26: Performance Quality Metrics for Mobile Web and Mobile Native - Agile Testing Days 2014](https://reader033.fdocuments.net/reader033/viewer/2022060121/559452bf1a28abce4f8b46cd/html5/thumbnails/26.jpg)
![Page 27: Performance Quality Metrics for Mobile Web and Mobile Native - Agile Testing Days 2014](https://reader033.fdocuments.net/reader033/viewer/2022060121/559452bf1a28abce4f8b46cd/html5/thumbnails/27.jpg)
Metrics: # Visitors# Requests / User
Dev: Follow Best Practices
Test: Find realistic scenarios
![Page 28: Performance Quality Metrics for Mobile Web and Mobile Native - Agile Testing Days 2014](https://reader033.fdocuments.net/reader033/viewer/2022060121/559452bf1a28abce4f8b46cd/html5/thumbnails/28.jpg)
![Page 29: Performance Quality Metrics for Mobile Web and Mobile Native - Agile Testing Days 2014](https://reader033.fdocuments.net/reader033/viewer/2022060121/559452bf1a28abce4f8b46cd/html5/thumbnails/29.jpg)
“Blindly” (Re)use Existing
Components
![Page 30: Performance Quality Metrics for Mobile Web and Mobile Native - Agile Testing Days 2014](https://reader033.fdocuments.net/reader033/viewer/2022060121/559452bf1a28abce4f8b46cd/html5/thumbnails/30.jpg)
Requirement: We need a report
![Page 31: Performance Quality Metrics for Mobile Web and Mobile Native - Agile Testing Days 2014](https://reader033.fdocuments.net/reader033/viewer/2022060121/559452bf1a28abce4f8b46cd/html5/thumbnails/31.jpg)
Using Hibernate results in 4k+ SQL Statements to display 3 items!
Hibernate Executes 4k+ Statements
Individual Execution VERY FAST
But Total SUM takes 6s
![Page 32: Performance Quality Metrics for Mobile Web and Mobile Native - Agile Testing Days 2014](https://reader033.fdocuments.net/reader033/viewer/2022060121/559452bf1a28abce4f8b46cd/html5/thumbnails/32.jpg)
Requirement: We need a fancy UI
![Page 33: Performance Quality Metrics for Mobile Web and Mobile Native - Agile Testing Days 2014](https://reader033.fdocuments.net/reader033/viewer/2022060121/559452bf1a28abce4f8b46cd/html5/thumbnails/33.jpg)
Using Telerik Controls Results in 9s for Data-Binding of UI Controls
#1: Slow Stored ProcedureDepending on Request execution
time of this SP varies between 1 and 7.5s
#2: 240! Similar SQL StatementsMost of these 240! Statements are not
prepared and just differ in things like Column Names
![Page 34: Performance Quality Metrics for Mobile Web and Mobile Native - Agile Testing Days 2014](https://reader033.fdocuments.net/reader033/viewer/2022060121/559452bf1a28abce4f8b46cd/html5/thumbnails/34.jpg)
Metrics: # Total SQLs# SQLs / Web Request# Same SQLs / Request
Transferred Rows
Test: With realistic Data
Dev: “Learn” Frameworks
![Page 35: Performance Quality Metrics for Mobile Web and Mobile Native - Agile Testing Days 2014](https://reader033.fdocuments.net/reader033/viewer/2022060121/559452bf1a28abce4f8b46cd/html5/thumbnails/35.jpg)
![Page 36: Performance Quality Metrics for Mobile Web and Mobile Native - Agile Testing Days 2014](https://reader033.fdocuments.net/reader033/viewer/2022060121/559452bf1a28abce4f8b46cd/html5/thumbnails/36.jpg)
#No “Agile” Deployment
![Page 37: Performance Quality Metrics for Mobile Web and Mobile Native - Agile Testing Days 2014](https://reader033.fdocuments.net/reader033/viewer/2022060121/559452bf1a28abce4f8b46cd/html5/thumbnails/37.jpg)
Load Spike resulted in UnavailabilityAd
on
air
![Page 38: Performance Quality Metrics for Mobile Web and Mobile Native - Agile Testing Days 2014](https://reader033.fdocuments.net/reader033/viewer/2022060121/559452bf1a28abce4f8b46cd/html5/thumbnails/38.jpg)
Alternative: “GoDaddy goes DevOps”
1h before SuperBowl KickOff
1h after Game ended
![Page 39: Performance Quality Metrics for Mobile Web and Mobile Native - Agile Testing Days 2014](https://reader033.fdocuments.net/reader033/viewer/2022060121/559452bf1a28abce4f8b46cd/html5/thumbnails/39.jpg)
Behind the Scenes
![Page 40: Performance Quality Metrics for Mobile Web and Mobile Native - Agile Testing Days 2014](https://reader033.fdocuments.net/reader033/viewer/2022060121/559452bf1a28abce4f8b46cd/html5/thumbnails/40.jpg)
Metrics: AvailabilityPage Size, # Objects
# Hosts, # Connections
DevOps: “Feature” Switches
![Page 41: Performance Quality Metrics for Mobile Web and Mobile Native - Agile Testing Days 2014](https://reader033.fdocuments.net/reader033/viewer/2022060121/559452bf1a28abce4f8b46cd/html5/thumbnails/41.jpg)
![Page 42: Performance Quality Metrics for Mobile Web and Mobile Native - Agile Testing Days 2014](https://reader033.fdocuments.net/reader033/viewer/2022060121/559452bf1a28abce4f8b46cd/html5/thumbnails/42.jpg)
# of Requests / User
# of Log Messages
# of Crashes
# Objects Allocated
# Objects In Cache
Cache Hit Ratio
# of Images
# of SQLs
# SQLs per RequestAvailability
# HTTP 3xx, 4xx
Page Size
![Page 43: Performance Quality Metrics for Mobile Web and Mobile Native - Agile Testing Days 2014](https://reader033.fdocuments.net/reader033/viewer/2022060121/559452bf1a28abce4f8b46cd/html5/thumbnails/43.jpg)
A final thought …
![Page 44: Performance Quality Metrics for Mobile Web and Mobile Native - Agile Testing Days 2014](https://reader033.fdocuments.net/reader033/viewer/2022060121/559452bf1a28abce4f8b46cd/html5/thumbnails/44.jpg)
![Page 45: Performance Quality Metrics for Mobile Web and Mobile Native - Agile Testing Days 2014](https://reader033.fdocuments.net/reader033/viewer/2022060121/559452bf1a28abce4f8b46cd/html5/thumbnails/45.jpg)
How about this idea?
12 0 120ms
3 1 68ms
Build 20 testPurchase OK
testSearch OK
Build 17 testPurchase OK
testSearch OK
Build 18 testPurchase FAILED
testSearch OK
Build 19 testPurchase OK
testSearch OK
Build # Test Case Status # SQL # Excep CPU
12 0 120ms
3 1 68ms
12 5 60ms
3 1 68ms
75 0 230ms
3 1 68ms
Test Framework Results Architectural Data
We identified a regresesion
Problem solved
Let’s look behind the scenes
Exceptions probably reason for
failed testsProblem fixed but now we have an
architectural regression
Problem fixed but now we have an
architectural regressionNow we have the functional and
architectural confidence
![Page 46: Performance Quality Metrics for Mobile Web and Mobile Native - Agile Testing Days 2014](https://reader033.fdocuments.net/reader033/viewer/2022060121/559452bf1a28abce4f8b46cd/html5/thumbnails/46.jpg)
How? Performance Focus in Test AutomationAnalyzing All Unit / Performance Tests
Analyze PerfMetrics
Identify Regressions
![Page 47: Performance Quality Metrics for Mobile Web and Mobile Native - Agile Testing Days 2014](https://reader033.fdocuments.net/reader033/viewer/2022060121/559452bf1a28abce4f8b46cd/html5/thumbnails/47.jpg)
![Page 48: Performance Quality Metrics for Mobile Web and Mobile Native - Agile Testing Days 2014](https://reader033.fdocuments.net/reader033/viewer/2022060121/559452bf1a28abce4f8b46cd/html5/thumbnails/48.jpg)
![Page 49: Performance Quality Metrics for Mobile Web and Mobile Native - Agile Testing Days 2014](https://reader033.fdocuments.net/reader033/viewer/2022060121/559452bf1a28abce4f8b46cd/html5/thumbnails/49.jpg)
If we do all that
![Page 51: Performance Quality Metrics for Mobile Web and Mobile Native - Agile Testing Days 2014](https://reader033.fdocuments.net/reader033/viewer/2022060121/559452bf1a28abce4f8b46cd/html5/thumbnails/51.jpg)
FREE Products & More Info
• Dynatrace Free Trial – http://bit.ly/dtrial– Full End-to-End Visibility in your Java, .NET, PHP Apps
– Sign up for a 30 Days (option for Lifetime) Free Trial on http://bit.ly/atd2014challenge
• Our Blog: http://blog.dynatrace.com
• My contact: @grabnerandi, [email protected]
![Page 52: Performance Quality Metrics for Mobile Web and Mobile Native - Agile Testing Days 2014](https://reader033.fdocuments.net/reader033/viewer/2022060121/559452bf1a28abce4f8b46cd/html5/thumbnails/52.jpg)
Bonus Bite(s)
![Page 53: Performance Quality Metrics for Mobile Web and Mobile Native - Agile Testing Days 2014](https://reader033.fdocuments.net/reader033/viewer/2022060121/559452bf1a28abce4f8b46cd/html5/thumbnails/53.jpg)
![Page 54: Performance Quality Metrics for Mobile Web and Mobile Native - Agile Testing Days 2014](https://reader033.fdocuments.net/reader033/viewer/2022060121/559452bf1a28abce4f8b46cd/html5/thumbnails/54.jpg)
Don’t ruin your reputation
![Page 55: Performance Quality Metrics for Mobile Web and Mobile Native - Agile Testing Days 2014](https://reader033.fdocuments.net/reader033/viewer/2022060121/559452bf1a28abce4f8b46cd/html5/thumbnails/55.jpg)
![Page 56: Performance Quality Metrics for Mobile Web and Mobile Native - Agile Testing Days 2014](https://reader033.fdocuments.net/reader033/viewer/2022060121/559452bf1a28abce4f8b46cd/html5/thumbnails/56.jpg)
![Page 57: Performance Quality Metrics for Mobile Web and Mobile Native - Agile Testing Days 2014](https://reader033.fdocuments.net/reader033/viewer/2022060121/559452bf1a28abce4f8b46cd/html5/thumbnails/57.jpg)
Outdated Library causing Issues
![Page 58: Performance Quality Metrics for Mobile Web and Mobile Native - Agile Testing Days 2014](https://reader033.fdocuments.net/reader033/viewer/2022060121/559452bf1a28abce4f8b46cd/html5/thumbnails/58.jpg)
![Page 59: Performance Quality Metrics for Mobile Web and Mobile Native - Agile Testing Days 2014](https://reader033.fdocuments.net/reader033/viewer/2022060121/559452bf1a28abce4f8b46cd/html5/thumbnails/59.jpg)
Implementation Flaws
![Page 60: Performance Quality Metrics for Mobile Web and Mobile Native - Agile Testing Days 2014](https://reader033.fdocuments.net/reader033/viewer/2022060121/559452bf1a28abce4f8b46cd/html5/thumbnails/60.jpg)
Business Impact requires Action!
![Page 61: Performance Quality Metrics for Mobile Web and Mobile Native - Agile Testing Days 2014](https://reader033.fdocuments.net/reader033/viewer/2022060121/559452bf1a28abce4f8b46cd/html5/thumbnails/61.jpg)
Solution: Cache to the RESCUE!!
![Page 62: Performance Quality Metrics for Mobile Web and Mobile Native - Agile Testing Days 2014](https://reader033.fdocuments.net/reader033/viewer/2022060121/559452bf1a28abce4f8b46cd/html5/thumbnails/62.jpg)
Implementation and Rollout
Implemented InMemory Cache
Worked well in Load Testing
![Page 63: Performance Quality Metrics for Mobile Web and Mobile Native - Agile Testing Days 2014](https://reader033.fdocuments.net/reader033/viewer/2022060121/559452bf1a28abce4f8b46cd/html5/thumbnails/63.jpg)
Result: Out of Memory Crashes!!
Still crashes
Problem fixed!Fixed Version Deployed
![Page 64: Performance Quality Metrics for Mobile Web and Mobile Native - Agile Testing Days 2014](https://reader033.fdocuments.net/reader033/viewer/2022060121/559452bf1a28abce4f8b46cd/html5/thumbnails/64.jpg)
Metrics: Heap Size, # Objects Allocated,# Objects in Cache
Cache Hit Ratio
Test: With realistic Data