ArcGIS Server Performance and Scalability - Testing...
Transcript of ArcGIS Server Performance and Scalability - Testing...
![Page 1: ArcGIS Server Performance and Scalability - Testing ...proceedings.esri.com/library/userconf/devsummit11/papers/tech/2011ar_1.pdf · Presentation outline • Test Objectives • Preparing](https://reader033.fdocuments.net/reader033/viewer/2022050715/5e053508b7d4f15e3615afdc/html5/thumbnails/1.jpg)
ArcGIS Server Performance and Scalability - Testing
MethodologiesAndrew Sakowicz
Frank Pizzi
![Page 2: ArcGIS Server Performance and Scalability - Testing ...proceedings.esri.com/library/userconf/devsummit11/papers/tech/2011ar_1.pdf · Presentation outline • Test Objectives • Preparing](https://reader033.fdocuments.net/reader033/viewer/2022050715/5e053508b7d4f15e3615afdc/html5/thumbnails/2.jpg)
Introductions
•• Who are we?Who are we?-- Professional ServicesProfessional Services
-- Enterprise Implementation Services TeamEnterprise Implementation Services Team
•• Who are you?Who are you?
![Page 3: ArcGIS Server Performance and Scalability - Testing ...proceedings.esri.com/library/userconf/devsummit11/papers/tech/2011ar_1.pdf · Presentation outline • Test Objectives • Preparing](https://reader033.fdocuments.net/reader033/viewer/2022050715/5e053508b7d4f15e3615afdc/html5/thumbnails/3.jpg)
Audience
• Testers• Developers• Managers• System Architects
![Page 4: ArcGIS Server Performance and Scalability - Testing ...proceedings.esri.com/library/userconf/devsummit11/papers/tech/2011ar_1.pdf · Presentation outline • Test Objectives • Preparing](https://reader033.fdocuments.net/reader033/viewer/2022050715/5e053508b7d4f15e3615afdc/html5/thumbnails/4.jpg)
Presentation outline
• Test Objectives• Preparing test environment • Test data• Test scripts• Load test• Analysis• Reporting• Capacity planning
![Page 5: ArcGIS Server Performance and Scalability - Testing ...proceedings.esri.com/library/userconf/devsummit11/papers/tech/2011ar_1.pdf · Presentation outline • Test Objectives • Preparing](https://reader033.fdocuments.net/reader033/viewer/2022050715/5e053508b7d4f15e3615afdc/html5/thumbnails/5.jpg)
Test Objectives
![Page 6: ArcGIS Server Performance and Scalability - Testing ...proceedings.esri.com/library/userconf/devsummit11/papers/tech/2011ar_1.pdf · Presentation outline • Test Objectives • Preparing](https://reader033.fdocuments.net/reader033/viewer/2022050715/5e053508b7d4f15e3615afdc/html5/thumbnails/6.jpg)
Test Objectives
- Contractual Service Level Agreement- Bottlenecks- Capacity- Benchmark
![Page 7: ArcGIS Server Performance and Scalability - Testing ...proceedings.esri.com/library/userconf/devsummit11/papers/tech/2011ar_1.pdf · Presentation outline • Test Objectives • Preparing](https://reader033.fdocuments.net/reader033/viewer/2022050715/5e053508b7d4f15e3615afdc/html5/thumbnails/7.jpg)
Preparing Test Environment
![Page 8: ArcGIS Server Performance and Scalability - Testing ...proceedings.esri.com/library/userconf/devsummit11/papers/tech/2011ar_1.pdf · Presentation outline • Test Objectives • Preparing](https://reader033.fdocuments.net/reader033/viewer/2022050715/5e053508b7d4f15e3615afdc/html5/thumbnails/8.jpg)
Preparing Test Environment
Tuning• Critical step in successful GIS deployments
- Profile individual operations- Tuned systems optimize the use of critical resources- Map service performance directly linked to underlying
MXD document
• Tools- Mxdperfstat- Map Service Publishing Toolbar- Fiddler- NeXpert
![Page 9: ArcGIS Server Performance and Scalability - Testing ...proceedings.esri.com/library/userconf/devsummit11/papers/tech/2011ar_1.pdf · Presentation outline • Test Objectives • Preparing](https://reader033.fdocuments.net/reader033/viewer/2022050715/5e053508b7d4f15e3615afdc/html5/thumbnails/9.jpg)
Browser
Web Server
SOM
SOC
Total Response Time (t1-
t2)
Wait Time
Search & Retrieval
Time
Usage Time
SDE/DBMS
A test is executed at the web browser.
It measures web browser call’s elapsed time
(roundtrip between browser and data source)
t1 t2
Preparing Test Environment
Tuning Measure at web browser, e.g. stop watch or programmatically using API events
![Page 10: ArcGIS Server Performance and Scalability - Testing ...proceedings.esri.com/library/userconf/devsummit11/papers/tech/2011ar_1.pdf · Presentation outline • Test Objectives • Preparing](https://reader033.fdocuments.net/reader033/viewer/2022050715/5e053508b7d4f15e3615afdc/html5/thumbnails/10.jpg)
Preparing Test Environment
Tuning - Fiddler
HTTP RequestHTTP Request
HTTP ResponseHTTP Response
Inspector TabInspector Tab
Fiddler HTTP Debugging
Proxy
![Page 11: ArcGIS Server Performance and Scalability - Testing ...proceedings.esri.com/library/userconf/devsummit11/papers/tech/2011ar_1.pdf · Presentation outline • Test Objectives • Preparing](https://reader033.fdocuments.net/reader033/viewer/2022050715/5e053508b7d4f15e3615afdc/html5/thumbnails/11.jpg)
Preparing Test Environment
Tuning - Fiddler FiddlerFiddler’’s Timeline measuress Timeline measuresTransaction timeTransaction time
![Page 12: ArcGIS Server Performance and Scalability - Testing ...proceedings.esri.com/library/userconf/devsummit11/papers/tech/2011ar_1.pdf · Presentation outline • Test Objectives • Preparing](https://reader033.fdocuments.net/reader033/viewer/2022050715/5e053508b7d4f15e3615afdc/html5/thumbnails/12.jpg)
Preparing Test Environment
Tuning - NeXpert
Transaction Transaction MarkerMarker““StepStep””
Fiddler PlugFiddler Plug--ininNeXpertNeXpert provides provides
Transaction marker and Transaction marker and performance reportperformance report
![Page 13: ArcGIS Server Performance and Scalability - Testing ...proceedings.esri.com/library/userconf/devsummit11/papers/tech/2011ar_1.pdf · Presentation outline • Test Objectives • Preparing](https://reader033.fdocuments.net/reader033/viewer/2022050715/5e053508b7d4f15e3615afdc/html5/thumbnails/13.jpg)
Preparing Test Environment
Tuning – NeXpert ReportNeXpertNeXpert ReportReport
![Page 14: ArcGIS Server Performance and Scalability - Testing ...proceedings.esri.com/library/userconf/devsummit11/papers/tech/2011ar_1.pdf · Presentation outline • Test Objectives • Preparing](https://reader033.fdocuments.net/reader033/viewer/2022050715/5e053508b7d4f15e3615afdc/html5/thumbnails/14.jpg)
Preparing the test environment
Fiddler Add-on
![Page 15: ArcGIS Server Performance and Scalability - Testing ...proceedings.esri.com/library/userconf/devsummit11/papers/tech/2011ar_1.pdf · Presentation outline • Test Objectives • Preparing](https://reader033.fdocuments.net/reader033/viewer/2022050715/5e053508b7d4f15e3615afdc/html5/thumbnails/15.jpg)
Profile Web Requests with Fiddler
![Page 16: ArcGIS Server Performance and Scalability - Testing ...proceedings.esri.com/library/userconf/devsummit11/papers/tech/2011ar_1.pdf · Presentation outline • Test Objectives • Preparing](https://reader033.fdocuments.net/reader033/viewer/2022050715/5e053508b7d4f15e3615afdc/html5/thumbnails/16.jpg)
Preparing Test Environment
Tuning• Optimize using map service publishing toolbar
- Analyze Map- Preview Map- Save Map Service Definition- Publish to ArcGIS Server
![Page 17: ArcGIS Server Performance and Scalability - Testing ...proceedings.esri.com/library/userconf/devsummit11/papers/tech/2011ar_1.pdf · Presentation outline • Test Objectives • Preparing](https://reader033.fdocuments.net/reader033/viewer/2022050715/5e053508b7d4f15e3615afdc/html5/thumbnails/17.jpg)
Preparing Test Environment
• Tuning• Optimize mxd with mxdperfstat
http://resources.arcgis.com/gallery/file/enterprise- gis/details?entryID=6391E988-1422-2418-88DE- 3E052E78213C
Item At Scale Layer NameRefresh
Time
(sec)Recommendations
Feature
sVertices
Labelin
g
Geogra
phy
Phase
(sec)
Graphic
s Phase
(sec)
Cursor
Phase
(sec)
DBMS
CPUDBM
S LIO
41 1,000TaxlotDenseLabel 1.93
Simplify labeling, symbology:
GraphicsPhase=1.42; simplify
geometry and/or set label
scale; convert polygon to
polyline: vertices
fetched=200001; simplify
geometry and/or set label
scale: vertices fetched=200001; 1 200,001 TRUE 0.45 1.42 1.04 0.02 266
42 1,000TaxlotDenseNoLa
bel 0.53simplify geometry: vertices
fetched=200001; 1 200,001 FALSE 0.45 0.02 0.9 0.02 140
![Page 18: ArcGIS Server Performance and Scalability - Testing ...proceedings.esri.com/library/userconf/devsummit11/papers/tech/2011ar_1.pdf · Presentation outline • Test Objectives • Preparing](https://reader033.fdocuments.net/reader033/viewer/2022050715/5e053508b7d4f15e3615afdc/html5/thumbnails/18.jpg)
Preparing Test Environment
Concurrent UsersConcurrent Users
TunedTuned
Not TunedNot Tuned
![Page 19: ArcGIS Server Performance and Scalability - Testing ...proceedings.esri.com/library/userconf/devsummit11/papers/tech/2011ar_1.pdf · Presentation outline • Test Objectives • Preparing](https://reader033.fdocuments.net/reader033/viewer/2022050715/5e053508b7d4f15e3615afdc/html5/thumbnails/19.jpg)
Preparing Test Environment
Configuration• Critical step in successful GIS deployments
- Incorrect settings can limit access to hardware resources
• Configure ArcGIS Server Instances
![Page 20: ArcGIS Server Performance and Scalability - Testing ...proceedings.esri.com/library/userconf/devsummit11/papers/tech/2011ar_1.pdf · Presentation outline • Test Objectives • Preparing](https://reader033.fdocuments.net/reader033/viewer/2022050715/5e053508b7d4f15e3615afdc/html5/thumbnails/20.jpg)
Preparing Test Environment
• Optimize ArcMap document and configure correct number of ArcGIS Server instances
Corrected(Unloaded TT: 0.34 sec)
(2.1 Instances/core)
Uncorrected(Unloaded TT: 11.97 sec)
(1.6 Instances/core)
![Page 21: ArcGIS Server Performance and Scalability - Testing ...proceedings.esri.com/library/userconf/devsummit11/papers/tech/2011ar_1.pdf · Presentation outline • Test Objectives • Preparing](https://reader033.fdocuments.net/reader033/viewer/2022050715/5e053508b7d4f15e3615afdc/html5/thumbnails/21.jpg)
Test Data
![Page 22: ArcGIS Server Performance and Scalability - Testing ...proceedings.esri.com/library/userconf/devsummit11/papers/tech/2011ar_1.pdf · Presentation outline • Test Objectives • Preparing](https://reader033.fdocuments.net/reader033/viewer/2022050715/5e053508b7d4f15e3615afdc/html5/thumbnails/22.jpg)
Test Data
Bbox (Using Fiddler) Area of InterestArea of Interest
Selected ExtentSelected ExtentFrom HTTPFrom HTTPDebuggingDebugging
ProxyProxy
![Page 23: ArcGIS Server Performance and Scalability - Testing ...proceedings.esri.com/library/userconf/devsummit11/papers/tech/2011ar_1.pdf · Presentation outline • Test Objectives • Preparing](https://reader033.fdocuments.net/reader033/viewer/2022050715/5e053508b7d4f15e3615afdc/html5/thumbnails/23.jpg)
Test Data
Attribute Data
![Page 24: ArcGIS Server Performance and Scalability - Testing ...proceedings.esri.com/library/userconf/devsummit11/papers/tech/2011ar_1.pdf · Presentation outline • Test Objectives • Preparing](https://reader033.fdocuments.net/reader033/viewer/2022050715/5e053508b7d4f15e3615afdc/html5/thumbnails/24.jpg)
Test Data
Generate Bboxes
One simple example of Python One simple example of Python script to generate script to generate BboxesBboxes
![Page 25: ArcGIS Server Performance and Scalability - Testing ...proceedings.esri.com/library/userconf/devsummit11/papers/tech/2011ar_1.pdf · Presentation outline • Test Objectives • Preparing](https://reader033.fdocuments.net/reader033/viewer/2022050715/5e053508b7d4f15e3615afdc/html5/thumbnails/25.jpg)
Test Data
Heat Map based on Heat Map based on response times response times
from from ArcGISArcGIS ServerServer
![Page 26: ArcGIS Server Performance and Scalability - Testing ...proceedings.esri.com/library/userconf/devsummit11/papers/tech/2011ar_1.pdf · Presentation outline • Test Objectives • Preparing](https://reader033.fdocuments.net/reader033/viewer/2022050715/5e053508b7d4f15e3615afdc/html5/thumbnails/26.jpg)
Test Data Observe correlation between featureObserve correlation between feature
density and performancedensity and performance
![Page 27: ArcGIS Server Performance and Scalability - Testing ...proceedings.esri.com/library/userconf/devsummit11/papers/tech/2011ar_1.pdf · Presentation outline • Test Objectives • Preparing](https://reader033.fdocuments.net/reader033/viewer/2022050715/5e053508b7d4f15e3615afdc/html5/thumbnails/27.jpg)
Test Data - GIS Test Tool
• How to generate appropriate test data?
![Page 28: ArcGIS Server Performance and Scalability - Testing ...proceedings.esri.com/library/userconf/devsummit11/papers/tech/2011ar_1.pdf · Presentation outline • Test Objectives • Preparing](https://reader033.fdocuments.net/reader033/viewer/2022050715/5e053508b7d4f15e3615afdc/html5/thumbnails/28.jpg)
GIS Test Tool
• Test Data Generation by Area of Interest and Selected Scales
- Bounding Boxes- Point- Lines- Routes
- Start, Stop, Distance constraints- Attribute Data
![Page 29: ArcGIS Server Performance and Scalability - Testing ...proceedings.esri.com/library/userconf/devsummit11/papers/tech/2011ar_1.pdf · Presentation outline • Test Objectives • Preparing](https://reader033.fdocuments.net/reader033/viewer/2022050715/5e053508b7d4f15e3615afdc/html5/thumbnails/29.jpg)
Generating good test data
Demo GIS Test Tool
![Page 30: ArcGIS Server Performance and Scalability - Testing ...proceedings.esri.com/library/userconf/devsummit11/papers/tech/2011ar_1.pdf · Presentation outline • Test Objectives • Preparing](https://reader033.fdocuments.net/reader033/viewer/2022050715/5e053508b7d4f15e3615afdc/html5/thumbnails/30.jpg)
Test Scripts
![Page 31: ArcGIS Server Performance and Scalability - Testing ...proceedings.esri.com/library/userconf/devsummit11/papers/tech/2011ar_1.pdf · Presentation outline • Test Objectives • Preparing](https://reader033.fdocuments.net/reader033/viewer/2022050715/5e053508b7d4f15e3615afdc/html5/thumbnails/31.jpg)
Test Scripts
• Record user workflow based on application user requirements
• Create single user web test - Define transactions- Set think time and pacing based on application user
requirements- Parameterize transaction inputs- Verify test script with single user
![Page 32: ArcGIS Server Performance and Scalability - Testing ...proceedings.esri.com/library/userconf/devsummit11/papers/tech/2011ar_1.pdf · Presentation outline • Test Objectives • Preparing](https://reader033.fdocuments.net/reader033/viewer/2022050715/5e053508b7d4f15e3615afdc/html5/thumbnails/32.jpg)
Visual Studio Quick Introduction - WebTest
TransactionTransaction
HTTP RequestHTTP Request
Data sourceData source
Query String parameter referencinQuery String parameter referencindata sourcedata source
![Page 33: ArcGIS Server Performance and Scalability - Testing ...proceedings.esri.com/library/userconf/devsummit11/papers/tech/2011ar_1.pdf · Presentation outline • Test Objectives • Preparing](https://reader033.fdocuments.net/reader033/viewer/2022050715/5e053508b7d4f15e3615afdc/html5/thumbnails/33.jpg)
Testing with JMeter Build HTTP RequesBuild HTTP Reques
![Page 34: ArcGIS Server Performance and Scalability - Testing ...proceedings.esri.com/library/userconf/devsummit11/papers/tech/2011ar_1.pdf · Presentation outline • Test Objectives • Preparing](https://reader033.fdocuments.net/reader033/viewer/2022050715/5e053508b7d4f15e3615afdc/html5/thumbnails/34.jpg)
Testing with JMeter– Test Data from CSV
Set Test DataSet Test Data
![Page 35: ArcGIS Server Performance and Scalability - Testing ...proceedings.esri.com/library/userconf/devsummit11/papers/tech/2011ar_1.pdf · Presentation outline • Test Objectives • Preparing](https://reader033.fdocuments.net/reader033/viewer/2022050715/5e053508b7d4f15e3615afdc/html5/thumbnails/35.jpg)
Testing with JMeter
![Page 36: ArcGIS Server Performance and Scalability - Testing ...proceedings.esri.com/library/userconf/devsummit11/papers/tech/2011ar_1.pdf · Presentation outline • Test Objectives • Preparing](https://reader033.fdocuments.net/reader033/viewer/2022050715/5e053508b7d4f15e3615afdc/html5/thumbnails/36.jpg)
JMeter Plug-in
• http://code.google.com/p/jmeter-plugins/- Many new plug-in enhancements for JMeter
![Page 37: ArcGIS Server Performance and Scalability - Testing ...proceedings.esri.com/library/userconf/devsummit11/papers/tech/2011ar_1.pdf · Presentation outline • Test Objectives • Preparing](https://reader033.fdocuments.net/reader033/viewer/2022050715/5e053508b7d4f15e3615afdc/html5/thumbnails/37.jpg)
Testing with GIS Test Tool
![Page 38: ArcGIS Server Performance and Scalability - Testing ...proceedings.esri.com/library/userconf/devsummit11/papers/tech/2011ar_1.pdf · Presentation outline • Test Objectives • Preparing](https://reader033.fdocuments.net/reader033/viewer/2022050715/5e053508b7d4f15e3615afdc/html5/thumbnails/38.jpg)
Load Test
![Page 39: ArcGIS Server Performance and Scalability - Testing ...proceedings.esri.com/library/userconf/devsummit11/papers/tech/2011ar_1.pdf · Presentation outline • Test Objectives • Preparing](https://reader033.fdocuments.net/reader033/viewer/2022050715/5e053508b7d4f15e3615afdc/html5/thumbnails/39.jpg)
Load Test
• Create load test- Define user load- Create machine counters to gather raw data for analysis
• Execute
![Page 40: ArcGIS Server Performance and Scalability - Testing ...proceedings.esri.com/library/userconf/devsummit11/papers/tech/2011ar_1.pdf · Presentation outline • Test Objectives • Preparing](https://reader033.fdocuments.net/reader033/viewer/2022050715/5e053508b7d4f15e3615afdc/html5/thumbnails/40.jpg)
Visual Studio Quick Introduction – Load TestScenarios:Scenarios:
Test Mix (Test Mix (WebTestWebTest or Unit Test), or Unit Test), Browser Mix,Browser Mix,Network Mix, Network Mix, Step LoadsStep Loads
PerfmonPerfmon Counter Sets:Counter Sets:Available categories that may be Available categories that may be
mapped to a machine in mapped to a machine in the deploymentthe deployment
Run Settings:Run Settings:Counter Set Mappings Counter Set Mappings –– Machine metricsMachine metrics
Test durationTest duration
![Page 41: ArcGIS Server Performance and Scalability - Testing ...proceedings.esri.com/library/userconf/devsummit11/papers/tech/2011ar_1.pdf · Presentation outline • Test Objectives • Preparing](https://reader033.fdocuments.net/reader033/viewer/2022050715/5e053508b7d4f15e3615afdc/html5/thumbnails/41.jpg)
Demo Visual Studio - Load Test Run
Threshold Threshold rules rules
violatedviolated
![Page 42: ArcGIS Server Performance and Scalability - Testing ...proceedings.esri.com/library/userconf/devsummit11/papers/tech/2011ar_1.pdf · Presentation outline • Test Objectives • Preparing](https://reader033.fdocuments.net/reader033/viewer/2022050715/5e053508b7d4f15e3615afdc/html5/thumbnails/42.jpg)
Testing with JMeter – Create Step Load
![Page 43: ArcGIS Server Performance and Scalability - Testing ...proceedings.esri.com/library/userconf/devsummit11/papers/tech/2011ar_1.pdf · Presentation outline • Test Objectives • Preparing](https://reader033.fdocuments.net/reader033/viewer/2022050715/5e053508b7d4f15e3615afdc/html5/thumbnails/43.jpg)
Testing with JMeter – Set Thread Properties
![Page 44: ArcGIS Server Performance and Scalability - Testing ...proceedings.esri.com/library/userconf/devsummit11/papers/tech/2011ar_1.pdf · Presentation outline • Test Objectives • Preparing](https://reader033.fdocuments.net/reader033/viewer/2022050715/5e053508b7d4f15e3615afdc/html5/thumbnails/44.jpg)
Testing with JMeter - Results
![Page 45: ArcGIS Server Performance and Scalability - Testing ...proceedings.esri.com/library/userconf/devsummit11/papers/tech/2011ar_1.pdf · Presentation outline • Test Objectives • Preparing](https://reader033.fdocuments.net/reader033/viewer/2022050715/5e053508b7d4f15e3615afdc/html5/thumbnails/45.jpg)
GIS Test Tool
Testing with GIS Test Tool
![Page 46: ArcGIS Server Performance and Scalability - Testing ...proceedings.esri.com/library/userconf/devsummit11/papers/tech/2011ar_1.pdf · Presentation outline • Test Objectives • Preparing](https://reader033.fdocuments.net/reader033/viewer/2022050715/5e053508b7d4f15e3615afdc/html5/thumbnails/46.jpg)
GIS Test Tool– Performance Test Capabilities
• Define web tests including QA step for verification• Define transactions including think time• Define load tests• Execute load tests• Capture system metrics for multiple machines• View results and export to Excel
![Page 47: ArcGIS Server Performance and Scalability - Testing ...proceedings.esri.com/library/userconf/devsummit11/papers/tech/2011ar_1.pdf · Presentation outline • Test Objectives • Preparing](https://reader033.fdocuments.net/reader033/viewer/2022050715/5e053508b7d4f15e3615afdc/html5/thumbnails/47.jpg)
Analysis
![Page 48: ArcGIS Server Performance and Scalability - Testing ...proceedings.esri.com/library/userconf/devsummit11/papers/tech/2011ar_1.pdf · Presentation outline • Test Objectives • Preparing](https://reader033.fdocuments.net/reader033/viewer/2022050715/5e053508b7d4f15e3615afdc/html5/thumbnails/48.jpg)
Tips and Tricks
Analyze Results - Validation• Compare and correlate key measurements
- Response Time (increasing, higher than initially profiled for single user)
- Throughput- CPU on all tiers- Network on all tiers- Disk on all tiers- Passed tests- Failed test
![Page 49: ArcGIS Server Performance and Scalability - Testing ...proceedings.esri.com/library/userconf/devsummit11/papers/tech/2011ar_1.pdf · Presentation outline • Test Objectives • Preparing](https://reader033.fdocuments.net/reader033/viewer/2022050715/5e053508b7d4f15e3615afdc/html5/thumbnails/49.jpg)
Tips and Tricks
Analyze Results - Validation• Lack of errors does not validate a test
- Requests may succeed but return zero size image- Spot check request response content size
![Page 50: ArcGIS Server Performance and Scalability - Testing ...proceedings.esri.com/library/userconf/devsummit11/papers/tech/2011ar_1.pdf · Presentation outline • Test Objectives • Preparing](https://reader033.fdocuments.net/reader033/viewer/2022050715/5e053508b7d4f15e3615afdc/html5/thumbnails/50.jpg)
Tips and Tricks
Analyze Results – Reporting and Analysis• Exclude failure range, e.g. failure rate > 5% from the
analysis• Exclude excessive resource utilization range
![Page 51: ArcGIS Server Performance and Scalability - Testing ...proceedings.esri.com/library/userconf/devsummit11/papers/tech/2011ar_1.pdf · Presentation outline • Test Objectives • Preparing](https://reader033.fdocuments.net/reader033/viewer/2022050715/5e053508b7d4f15e3615afdc/html5/thumbnails/51.jpg)
Tips and Tricks
Analyze Results Validation Example – Unexpected response time decrease under heavy load
Unexpected curve shape:Unexpected curve shape:Response time should be increasing.Response time should be increasing.
Likely root cause: Likely root cause: failed or 0 size image requests.failed or 0 size image requests.
![Page 52: ArcGIS Server Performance and Scalability - Testing ...proceedings.esri.com/library/userconf/devsummit11/papers/tech/2011ar_1.pdf · Presentation outline • Test Objectives • Preparing](https://reader033.fdocuments.net/reader033/viewer/2022050715/5e053508b7d4f15e3615afdc/html5/thumbnails/52.jpg)
Tips and Tricks
Validation Example – Expected CPU and Response Time Correlation
Max CPUMax CPUutilizationutilization
Response Time increase Response Time increase correlates with:correlates with:
••User loadUser load••CPU utilizationCPU utilization
![Page 53: ArcGIS Server Performance and Scalability - Testing ...proceedings.esri.com/library/userconf/devsummit11/papers/tech/2011ar_1.pdf · Presentation outline • Test Objectives • Preparing](https://reader033.fdocuments.net/reader033/viewer/2022050715/5e053508b7d4f15e3615afdc/html5/thumbnails/53.jpg)
Tips and Tricks Validation Example – Test failure due to w3wp memory bottleneck
Root cause:Root cause:System memory bottleneck at System memory bottleneck at
Web Server processWeb Server processSymptom:Symptom:
System available memory System available memory is decreasingis decreasing
![Page 54: ArcGIS Server Performance and Scalability - Testing ...proceedings.esri.com/library/userconf/devsummit11/papers/tech/2011ar_1.pdf · Presentation outline • Test Objectives • Preparing](https://reader033.fdocuments.net/reader033/viewer/2022050715/5e053508b7d4f15e3615afdc/html5/thumbnails/54.jpg)
Tips and Tricks Validation Example – Unexpected CPU utilization while accessing cache services
Unexpected bottleneck in Unexpected bottleneck in cache application :cache application :
100% CPU utilization .100% CPU utilization .How to determined root cause?How to determined root cause?
![Page 55: ArcGIS Server Performance and Scalability - Testing ...proceedings.esri.com/library/userconf/devsummit11/papers/tech/2011ar_1.pdf · Presentation outline • Test Objectives • Preparing](https://reader033.fdocuments.net/reader033/viewer/2022050715/5e053508b7d4f15e3615afdc/html5/thumbnails/55.jpg)
Tips and Tricks
Determining System Capacity• Maximum number of concurrent users
corresponding to, e.g.:- Maximum acceptable response time- First failure or 5%- Resource utilization greater than 85%, for example CPU
• Different ways of defining acceptance criteria (performance level of service), e.g.
- 95% of requests under 3 sec- Max request under 10 sec
![Page 56: ArcGIS Server Performance and Scalability - Testing ...proceedings.esri.com/library/userconf/devsummit11/papers/tech/2011ar_1.pdf · Presentation outline • Test Objectives • Preparing](https://reader033.fdocuments.net/reader033/viewer/2022050715/5e053508b7d4f15e3615afdc/html5/thumbnails/56.jpg)
Tips and Tricks - Execute
• Ensure - Only target applications are running- Application data is in the same state for every test- Good configuration management is critical to getting
consistent load test results
5 6
![Page 57: ArcGIS Server Performance and Scalability - Testing ...proceedings.esri.com/library/userconf/devsummit11/papers/tech/2011ar_1.pdf · Presentation outline • Test Objectives • Preparing](https://reader033.fdocuments.net/reader033/viewer/2022050715/5e053508b7d4f15e3615afdc/html5/thumbnails/57.jpg)
Tips and Tricks - Common pitfalls
• Application stack is not properly configured and tuned
• Single error in a test can lead to a cascade of failing requests
• Test client machine is bottleneck- CPU- Memory- Networks (cache test in particular)
5
![Page 58: ArcGIS Server Performance and Scalability - Testing ...proceedings.esri.com/library/userconf/devsummit11/papers/tech/2011ar_1.pdf · Presentation outline • Test Objectives • Preparing](https://reader033.fdocuments.net/reader033/viewer/2022050715/5e053508b7d4f15e3615afdc/html5/thumbnails/58.jpg)
Tips and Tricks - Common pitfalls
Poor Configuration Example – Unexpected high CPU utilization due to lsass authentication not configured correctly
High CPU utilization (187%)High CPU utilization (187%)of Web server due toof Web server due to
authentication process authentication process ((lsass.exelsass.exe))
![Page 59: ArcGIS Server Performance and Scalability - Testing ...proceedings.esri.com/library/userconf/devsummit11/papers/tech/2011ar_1.pdf · Presentation outline • Test Objectives • Preparing](https://reader033.fdocuments.net/reader033/viewer/2022050715/5e053508b7d4f15e3615afdc/html5/thumbnails/59.jpg)
Report
![Page 60: ArcGIS Server Performance and Scalability - Testing ...proceedings.esri.com/library/userconf/devsummit11/papers/tech/2011ar_1.pdf · Presentation outline • Test Objectives • Preparing](https://reader033.fdocuments.net/reader033/viewer/2022050715/5e053508b7d4f15e3615afdc/html5/thumbnails/60.jpg)
Report
- Executive Summary- Test Plan
- Workflows- Work load
- Deployment documentation- Results and Charts
- Key Indicators, e.g. Response Time, Throughput- System Metrics, e.g. CPU %- Errors
- Summary and Conclusions- Provide management recommendations for improvements
- Appendix
![Page 61: ArcGIS Server Performance and Scalability - Testing ...proceedings.esri.com/library/userconf/devsummit11/papers/tech/2011ar_1.pdf · Presentation outline • Test Objectives • Preparing](https://reader033.fdocuments.net/reader033/viewer/2022050715/5e053508b7d4f15e3615afdc/html5/thumbnails/61.jpg)
Test Tools
![Page 62: ArcGIS Server Performance and Scalability - Testing ...proceedings.esri.com/library/userconf/devsummit11/papers/tech/2011ar_1.pdf · Presentation outline • Test Objectives • Preparing](https://reader033.fdocuments.net/reader033/viewer/2022050715/5e053508b7d4f15e3615afdc/html5/thumbnails/62.jpg)
Testing - Selecting Load Test Tool
Test Tools Open Source
Pros Cons
LoadRunner No •Industry Leader•Automatic negative correlations identified with service level agreements•Http Web Testing•Click and Script•Very good tools for testing SOA• Test results stored in database•Thick Client Testing•Can be used for bottleneck analysis
•High Cost•Test Development in in C programming language•Test metrics difficult to manage and correlate•Poor user community with few available examples
Silk PerformerNo •Good solution for testing Citrix
•Wizard driven interface guides the user•Can be used for bottleneck analysis
•Moderate to High Cost•Test metrics are poor•Test Development uses proprietary language•Test metrics difficult to manage and correlate•Poor user community with few available examples
Visual Studio Test Team No •Low to moderate cost•Excellent Test Metric reporting•Test Scripting in C# or VB .NET•Unit and Web Testing available•Blog support with good examples•Very good for bottleneck analysis
•No built in support for AMF•No Thick Client options•Moderate user community
JMeter Yes •Free•Tool
•Provides only response times•Poor User community with few available examples
6 2
![Page 63: ArcGIS Server Performance and Scalability - Testing ...proceedings.esri.com/library/userconf/devsummit11/papers/tech/2011ar_1.pdf · Presentation outline • Test Objectives • Preparing](https://reader033.fdocuments.net/reader033/viewer/2022050715/5e053508b7d4f15e3615afdc/html5/thumbnails/63.jpg)
Testing - Selecting Load Test Tool
• Tool selection depends on objective- Commercial tools all have system metrics and
correlation tools- Free tools typically provide response times and
throughput, but leave system metrics to the tester to gather and report on
![Page 64: ArcGIS Server Performance and Scalability - Testing ...proceedings.esri.com/library/userconf/devsummit11/papers/tech/2011ar_1.pdf · Presentation outline • Test Objectives • Preparing](https://reader033.fdocuments.net/reader033/viewer/2022050715/5e053508b7d4f15e3615afdc/html5/thumbnails/64.jpg)
Capacity Planning
![Page 65: ArcGIS Server Performance and Scalability - Testing ...proceedings.esri.com/library/userconf/devsummit11/papers/tech/2011ar_1.pdf · Presentation outline • Test Objectives • Preparing](https://reader033.fdocuments.net/reader033/viewer/2022050715/5e053508b7d4f15e3615afdc/html5/thumbnails/65.jpg)
Input Capacity Planning
• Find Input for Capacity Planning- Test Report
- Includes Throughput (Transactions per hour) - Includes System Metrics – %CPU Utilization, #Cores - Spec Rate from the machines tested
- Use this information to calculate Service Time for Transactions
![Page 66: ArcGIS Server Performance and Scalability - Testing ...proceedings.esri.com/library/userconf/devsummit11/papers/tech/2011ar_1.pdf · Presentation outline • Test Objectives • Preparing](https://reader033.fdocuments.net/reader033/viewer/2022050715/5e053508b7d4f15e3615afdc/html5/thumbnails/66.jpg)
Input Capacity Planning
• Capacity model expressed as Service Time
100%3600#
TH
CPUCPUST
![Page 67: ArcGIS Server Performance and Scalability - Testing ...proceedings.esri.com/library/userconf/devsummit11/papers/tech/2011ar_1.pdf · Presentation outline • Test Objectives • Preparing](https://reader033.fdocuments.net/reader033/viewer/2022050715/5e053508b7d4f15e3615afdc/html5/thumbnails/67.jpg)
Capacity Planning
• Estimate capacity for a different hardware platforms- Find your target (t) server on
http://www.spec.org/cpu2006/results/rint2006.html
t
b
t
tbt rCPUSpecRatePe
rCPUSpecRatePeCPU
THSTCPU
%3600100#
![Page 68: ArcGIS Server Performance and Scalability - Testing ...proceedings.esri.com/library/userconf/devsummit11/papers/tech/2011ar_1.pdf · Presentation outline • Test Objectives • Preparing](https://reader033.fdocuments.net/reader033/viewer/2022050715/5e053508b7d4f15e3615afdc/html5/thumbnails/68.jpg)
Capacity Planning
• SPEC.org- http://www.spec.org/cpu2006/results/rint2006.html- Find your server from the results
![Page 69: ArcGIS Server Performance and Scalability - Testing ...proceedings.esri.com/library/userconf/devsummit11/papers/tech/2011ar_1.pdf · Presentation outline • Test Objectives • Preparing](https://reader033.fdocuments.net/reader033/viewer/2022050715/5e053508b7d4f15e3615afdc/html5/thumbnails/69.jpg)
Capacity Planning
![Page 70: ArcGIS Server Performance and Scalability - Testing ...proceedings.esri.com/library/userconf/devsummit11/papers/tech/2011ar_1.pdf · Presentation outline • Test Objectives • Preparing](https://reader033.fdocuments.net/reader033/viewer/2022050715/5e053508b7d4f15e3615afdc/html5/thumbnails/70.jpg)
Capacity Planning
• Additional examples on capacity planning can be found here:
- http://resources.arcgis.com/gallery/file/enterprise- gis/details?entryID=6367F821-1422-2418-886F- FCC43C8C8E22
![Page 71: ArcGIS Server Performance and Scalability - Testing ...proceedings.esri.com/library/userconf/devsummit11/papers/tech/2011ar_1.pdf · Presentation outline • Test Objectives • Preparing](https://reader033.fdocuments.net/reader033/viewer/2022050715/5e053508b7d4f15e3615afdc/html5/thumbnails/71.jpg)
System Designer
![Page 72: ArcGIS Server Performance and Scalability - Testing ...proceedings.esri.com/library/userconf/devsummit11/papers/tech/2011ar_1.pdf · Presentation outline • Test Objectives • Preparing](https://reader033.fdocuments.net/reader033/viewer/2022050715/5e053508b7d4f15e3615afdc/html5/thumbnails/72.jpg)
Questions?
![Page 73: ArcGIS Server Performance and Scalability - Testing ...proceedings.esri.com/library/userconf/devsummit11/papers/tech/2011ar_1.pdf · Presentation outline • Test Objectives • Preparing](https://reader033.fdocuments.net/reader033/viewer/2022050715/5e053508b7d4f15e3615afdc/html5/thumbnails/73.jpg)
Contact us
• Professional Services- Andrew Sakowicz, [email protected] Frank Pizzi, [email protected]