Performance testing of a Road Tolling system - Siegfried Goeschl - EuroSTAR 2012
-
Upload
eurostar-conference -
Category
Technology
-
view
24 -
download
1
description
Transcript of Performance testing of a Road Tolling system - Siegfried Goeschl - EuroSTAR 2012
Performance Testing of a Road Tolling System
Oliver BaendlSiegfried Goeschl
1Donnerstag, 27. September 12
Gauteng Open Road TollingSouth Africa
2Donnerstag, 27. September 12
42 Tolling Points3Donnerstag, 27. September 12
1.500.000 registered road-user4Donnerstag, 27. September 12
11 Satellite Centers5Donnerstag, 27. September 12
41 E-Toll Shops
6Donnerstag, 27. September 12
20 Mobile Payment Units7Donnerstag, 27. September 12
Central Operation Center8Donnerstag, 27. September 12
750 Concurrent CRM Client Users
9Donnerstag, 27. September 12
162 vehicle passges per second10Donnerstag, 27. September 12
400 concurrent web users11Donnerstag, 27. September 12
Two Data Centers
12Donnerstag, 27. September 12
Master Control Program
13Donnerstag, 27. September 12
Project Overview
• Vehicles are passing through a tolling point
• The vehicle is identified either using the license plate number (via OCR) or a tag
• Vehicle passage are transmitted to Open Road Tolling Back Office (ORTBO)
• 90% video passages with three images
• 10% tag passages
14Donnerstag, 27. September 12
Project Overview
• Vehicle passages are collected and assigned to road-user accounts in the Transaction Clearing House (TCH)
• TCH responsible for billing the customer
• Passages of unregistered or black-listed cars are processed by the Violation Processing Center (VPC)
15Donnerstag, 27. September 12
Project Overview
• Overall system contains multiple Oracle RACs and MSSQL clusters
• Interfaces between the subsystems mostly web services
• Backup data center replicates the “Central Operation Center” over dark fiber
16Donnerstag, 27. September 12
Schedule Phase One
• Performance testing of CRM Server simulating 771 concurrent & active CRM client sessions
• Performance testing of public-facing web portal simulating 400 concurrent & active web user sessions
• On success the CRM subsystem goes live and has real user accounts
17Donnerstag, 27. September 12
Schedule Phase Two
• Simulating up to 162 vehicle passage per second
• Validating the end-to-end processing under load (passage processing, customer billing and violation processing)
• On success the overall system goes live
18Donnerstag, 27. September 12
Performance Test Scope
• End to end performance acceptance test
• Mandatory and client-witnessed test
• Strict acceptance criteria regarding
‣ throughput
‣ number and severity of errors
‣ validation of test runs
19Donnerstag, 27. September 12
Performance Test Types
Baseline Test One hour average load
Stress Test One hour peak load
Endurance Test 24 hours maximum load
20Donnerstag, 27. September 12
Performance Test Hardware
• Six dedicated load injectors
‣ Windows 2008 Server
‣ 3 physical and 3 virtual boxes
‣ Dual quad-cores with 8 GB RAM
• No direct access outside the data center
‣ Citrix over RDP over RDP
21Donnerstag, 27. September 12
Performance Test Tools• All performance test tools have issues
‣ Make sure that they work for you
• We prefer JVM-based test tools
‣ Easy to migrate between different OS
‣ We are Java developers
• We prefer command-line invocation
‣ RDP over slow network is really slow
22Donnerstag, 27. September 12
Performance Test Tools
• We use a non-distributed installation
‣ Complex network and routing
‣ Firewall not under our control
• Check the license and price tags
‣ Virtual users can be expensive
‣ Connectivity to license server?
23Donnerstag, 27. September 12
The Test Tools We Used
Subsystem Interface Planned Delivered
CRM Server WebService
Public Web Portal HTML
Vehicle PassageProcessing WebService
24Donnerstag, 27. September 12
Apache JMeter
• Open Source and free
• Implemented a new reporting backend to scope with huge result files
• Rock solid - only minor issues
• Extensible using scripting and Java libraries
• Rough GUI compared to commercial tools
• Unsuitable for complex ASP.NET web sites
25Donnerstag, 27. September 12
WAPT Pro
• Excellent value for money
• Good support
• ASP.NET module saved my day
• Extensible through Javascript
• One major issue with generation of test values under high load (requires restart)
26Donnerstag, 27. September 12
soapUI Pro
• Excellent for prototyping and functional testing of web services
• WebService mocking saved my day
• Extensible using Groovy and Java libraries
• Various issues during load testing
• soapUI was replaced with JMeter for performance tests
27Donnerstag, 27. September 12
• No software installation
• Take two (or more)
• Automation is your friend
• Performance test for everyone
• Performance tests are a valuable asset
Some More Thoughts
28Donnerstag, 27. September 12
No Software Installation
• Portable Apps on USB Stick
‣ Run tests directly from USB Stick
‣ Clone USB Stick to hard disk
• Have all your tools on the USB stick
‣ Java, JMeter, Editor, Git, ...
29Donnerstag, 27. September 12
Take Two (or more)
• A minimum of two share-nothing load injectors are required to prove that the production servers is causing the performance bottleneck (and not your test tool)
• We used a maximum of five share-nothing load injectors simultaneously
30Donnerstag, 27. September 12
• Performance test were run mostly at night and over the weekend
• Test execution managed by Hudson
• Test failures triggers email notification
• Test protocols are copied to FTP server
Automation Is Your Friend
31Donnerstag, 27. September 12
Performance Tests for Everyone
• All test scenarios are configured as Hudson jobs
‣ Baseline, stress & endurance test
‣ Some other internal tests
• Everyone can start and monitor a performance test scenario over the web browser
• Even developers can run performance tests
32Donnerstag, 27. September 12
• Performance tests can be used in creative and un-planned ways
‣ Smoke test for new deployment
‣ Testing database failover
‣ Testing data center replication & failover
‣ Testing different database setups
Performance Tests as Asset
33Donnerstag, 27. September 12
The few things you should take home
34Donnerstag, 27. September 12
Lessons Learned
• Tests tools are buggy
• Client-witnessed test are hard
• Performance tests are a valuable asset
• Creation and maintenance of complex performance test suite is a project on its own
35Donnerstag, 27. September 12
Questions & Answers
36Donnerstag, 27. September 12
Resources
JMeter SLA Report https://github.com/sgoeschl/jmeter-sla-report
JAMon API http://jamonapi.sourceforge.net/
JMeter ANT Tasks http://www.programmerplanet.org/pages/projects/jmeter-ant-task.php
WAPT Pro http://www.loadtestingtool.com
soapUI http://www.soapui.org/
37Donnerstag, 27. September 12