Post on 14-Jan-2022
Simulating Production Beforehand, Mining Statistics After
Mario AndreoliSolution & Infrastructure Architect
Performance Analysis , Testing & Tuning
Simulating Production BeforehandMining Statistics Afterward
May 28 2013
TASSQ Presentation
@MarioAndreoli
LinkedIn.com/in/MarioAndreoli
Mario@acs1.com
Simulating Production Beforehand, Mining Statistics After
Mario AndreoliSolution & Infrastructure Architect
Performance Analysis , Testing & Tuning
28 May 2013| 2
� Introduction
� Challenge - Certifying performance on a mission critical financial application
� Solution to the “Performance Certification” challenge
� Inception – understand what needs to be done
� Elaboration – explain how the solution will work
� Construction – build the test components and a data factory
� Test Execution and Test Results
� Additional Benefits
� Where are we today?
Agenda
Simulating Production Beforehand, Mining Statistics After
Mario AndreoliSolution & Infrastructure Architect
Performance Analysis , Testing & Tuning
28 May 2013| 3
� Historically a Developer, Solution Architect
� Strengths (certifications) in RUP, OOAD, J2EE
� “Reverse Engineering” strengths
� Created a bridge between Technical & Testing
� Teams consist of T/BSAs, Developers, Testers
� Certified the performance of a $250 million financial application
— not in the negative “Gartner Statistic” group
Introduction
Simulating Production Beforehand, Mining Statistics After
Mario AndreoliSolution & Infrastructure Architect
Performance Analysis , Testing & Tuning
28 May 2013| 4
Questions:
� How will it perform in Production?
— Is the Hardware sized appropriately?
— Is the Software tuned efficiently?
—Will the solution handle the anticipated volume?
—Are we confident transaction processing will have a significant percentage of straight through processing?
—Can the communication infrastructure handle the load?
—Will the end users have an acceptable experience?
—We can’t wait for Production to catch issues!
� How do I build confidence for the Stakeholders?
— Top down support is crucial. How do we get it?
— How can we build a plan that Stakeholders buy into?
General Challenges
Simulating Production Beforehand, Mining Statistics After
Mario AndreoliSolution & Infrastructure Architect
Performance Analysis , Testing & Tuning
Answers:
� How will it perform in Production?
—Design and create a working model thataccurately simulates the solution before it is in Production
� How do I build confidence for the Stakeholders?
—Provide tools Stakeholders can use to be part of the process and monitor business healththroughout and after
28 May 2013| 5
We Can Address It
Simulating Production Beforehand, Mining Statistics After
Mario AndreoliSolution & Infrastructure Architect
Performance Analysis , Testing & Tuning
28 May 2013| 6
Technical Challenges:
� Vendor Application
— no source access, little architectural information, no data model
� Complex Workflow Engine
— Senior RBT resource estimated an extremely high possible number of paths (test cases)
� 40+ interfaces / 250+ concurrent users
� Multiple functional streams
— Each with their own critical and complex requirements
� Global access (interfaces & users)
� Millions of varying types of messages daily
— exponentially more transactions
Inception
Simulating Production Beforehand, Mining Statistics After
Mario AndreoliSolution & Infrastructure Architect
Performance Analysis , Testing & Tuning
28 May 2013| 7
Financial Challenges:
� Market close deadlines
� Financial critical transactions
� Vital upstream and downstream SLAs
� This has to work, $000,000,000’s depend on it
— We do not want to be (negatively) in the newspaper the next day
� Manual functional testing of all production (financial) data scenarios almost impossible
� Distributed geographical markets
Inception
Simulating Production Beforehand, Mining Statistics After
Mario AndreoliSolution & Infrastructure Architect
Performance Analysis , Testing & Tuning
28 May 2013| 8
Approach:
� Senior QA resource team came to a conclusion that in addition to a good test strategy, a different approach was needed
� A strong technical resource with ‘out of the box’ thinking
� Iterative with quick and frequent deliveries
� Built in layers – with agile yet controlled processing
� Quick iterations and deployments to:
— start the testing early…learn early
— easily repetitive (building the regression suite),
— no (or minimal) manual interventiontests
statistics
learning
decisions
fixes
Resolution
Inception
Simulating Production Beforehand, Mining Statistics After
Mario AndreoliSolution & Infrastructure Architect
Performance Analysis , Testing & Tuning
28 May 2013| 9
Iterative Approach:
Inception
� Build layers of tests that can all talk to each other and be used in harmony to do full performance, load, regression testing etc.
Simulating Production Beforehand, Mining Statistics After
Mario AndreoliSolution & Infrastructure Architect
Performance Analysis , Testing & Tuning
28 May 2013| 10
White-Boxing a Black-Box:
� Understand business concepts?
� Vendor + Business Stream
� Flows, Begin-Status, End-Status, Rules
� Sample Messages (in/out)
� Understand the architecture through Reverse Engineering
� Structural Diagrams:
� Deployment, Package, Component
� Hardware, Software, Communication
� Interfaces and Message Transportation
� Behaviour Diagrams:
� Activity, State, Sequence
� Major Flows for System Behaviour
� Guide other teams/groups
� Functional coverage
� Code coverage
Elaboration
Simulating Production Beforehand, Mining Statistics After
Mario AndreoliSolution & Infrastructure Architect
Performance Analysis , Testing & Tuning
Performance Test Scope:
Elaboration
Performance
TestingTypes
Overall
Program
Components
Dependencies
Load
Endurance
Repetitive
Stress
H/W, Network,
Database
(optional)
Timing to
Volumes and SLA
(Performance NFR)
FinancialApp User
Interface
Response Time
Interface:
SI Processing Time
Batch Cyle Time
FinancialApp
User Interface
Interface
Processing
Batch
Processing
Risk
Volume
Importance
EnvironmentFunctional
Testing
Tuning
ESB/FTP Transportation
Failover & DR
Performance
External System
Performance
Automation
Tools
User Interface
Loadrunner
Interface
Harness
Batch
Schedule
Management
Database
Reset
(Flashback)
Stability over long periods
H/W Resource Management
Finding Ceiling of
Architectural Component
Realistic Production
Volumes & Timelines
Anticipated growth
Exact same execution,
repeated to determine
transaction accuracy
Stream:
Trade Processing
Cash
Entitlement
Tax
Integration
Corporate Actions
Query
User Types:
Inquiry (Detail, Index, Wildcard)
Update
40+ Interfaces
Day in the Life
of
FinancialApp
Database:
--Start
--Growth
IntradayEOD/BOD
FinancialApp version
Core version
FinancialApp Deployment Footprint
All other infrastructure components
Technical Test Team
Functional Test Team
Vendor
IT Support
SCM
Environment Coordinator
LoadRunner
Infrastructure Resources (DBA,
Capacity & Planning, Websphere etc.)
Geographic:
Europe users
BI
Entitlement &
Position
Generation
Out of Scope
Supporting
Resources
“A Day In
The Life”
All Overall Program
components running
together over a 24 hour
period
Measurements
Prioritization
Statistics
Monitoring
Simulating Production Beforehand, Mining Statistics After
Mario AndreoliSolution & Infrastructure Architect
Performance Analysis , Testing & Tuning
28 May 2013| 12
Business Concepts: Prioritize & Understand� Select your high volume, critical business flows to understand and build
harnesses for
� The scenario below is 1of 4 built (there were 10 scenarios in total, but the 4 selected consisted of 90% of the overall volume)
Elaboration
Simulating Production Beforehand, Mining Statistics After
Mario AndreoliSolution & Infrastructure Architect
Performance Analysis , Testing & Tuning
28 May 2013| 13
Architecture � We take each scenario and perform a volume analysis
� Map volume per time period
� Analyze peaks
� Confirm your findings with business resources
Elaboration
Simulating Production Beforehand, Mining Statistics After
Mario AndreoliSolution & Infrastructure Architect
Performance Analysis , Testing & Tuning
28 May 2013| 14
Hardware: Technical Understanding & Architecture
Elaboration
Reverse engineer the system and create the diagrams to help you understand the application:
Deployment: The physical layout of servers/nodes and their roles
It may also depict any load balancing strategies
Communication amongst servers/nodes/components should be depicted as well
Simulating Production Beforehand, Mining Statistics After
Mario AndreoliSolution & Infrastructure Architect
Performance Analysis , Testing & Tuning
28 May 2013| 15
Software: Technical Understanding & Architecture
Elaboration
Reverse engineer the system and create the diagrams to help you understand the application:
Component: Determine the ‘pieces’ of the system and map the relationships between these different ‘pieces’
Simulating Production Beforehand, Mining Statistics After
Mario AndreoliSolution & Infrastructure Architect
Performance Analysis , Testing & Tuning
28 May 2013| 16
Interface Transportation & Processing� Understand the major points and paths whereby messages will flow in and out
of the application
� Understand scope, which caveats you need to state in test results
� Point of binding for test harness depends on purpose, capability, constraints
Elaboration
Simulating Production Beforehand, Mining Statistics After
Mario AndreoliSolution & Infrastructure Architect
Performance Analysis , Testing & Tuning
28 May 2013| 17
Tying It All Together:A Day in the Life (DIL) of the Financial Application:
� Map all business layers on a 24 hour clock
� Each layer can have many components running to execute the pertinent tests
Elaboration
24 Hour Clock (EST)
Activity 0:00 1:00 2:00 3:00 4:00 5:00 6:00 7:00 8:00 9:00 10:00 11:00 12:00 13:00 14:00 15:00 16:00 17:00 18:00 19:00 20:00 21:00 22:00 23:00
Batch Processing Intraday Batches EOD Batches
Online - Europe
Online - North America
Trade Transactions
Annoucement Interfaces
Other Interfaces
Entitlements Processing
BI Data Replication
Application
Layer
Test
Component
1..* *
DIL Profile
1..*1
Simulating Production Beforehand, Mining Statistics After
Mario AndreoliSolution & Infrastructure Architect
Performance Analysis , Testing & Tuning
28 May 2013| 18
Tying It All Together:A Day in the Life of the Financial Application:
� At this point we have a good understanding of critical:
— functions, flows, volumes, processesand timing
� We now need a profile that unifies all our data and understanding
� Circulate this profile for agreement and sign off
� This will be our contract starting point for our physical design, prototyping and construction
� If we can emulate a 24 hour period, we now have a baseline to test with
Elaboration
Simulating Production Beforehand, Mining Statistics After
Mario AndreoliSolution & Infrastructure Architect
Performance Analysis , Testing & Tuning
28 May 2013| 19
Tying It All Together:A Day in the Life of the Financial Application:
� A component:
— can be a user scenario, interface feed, business process etc.
— is its own “thread” in the final simulator
— can be adjusted:
� On/Off
� Volume increase/decrease
� Duration time
� Period spike/lull
� Volume over Duration
—will be recordingstatistics and pertinent data
Elaboration
CC11
CC22
CC33
CCxx
DIL Profile
Simulating Production Beforehand, Mining Statistics After
Mario AndreoliSolution & Infrastructure Architect
Performance Analysis , Testing & Tuning
28 May 2013| 20
Day in the Life - Profile:
Elaboration
24 hour clock, broken up in 30 minute intervals
Interfaces, Business Processes,
GUI Lo
ad Profile
24 Hour Clock 0:00 1:00 2:00 3:00 4:00 5:00 6:00 7:00 8:00 9:00 10:00 11:00 12:00 13:00 14:00 15:00 16:00 17:00GUI Load Time Line 0:00 1:00 2:00 3:00 4:00 5:00 6:00 7:00 8:00 9:00 10:00 11:00 12:00 13:00 14:00 15:00GUI Load - Europe 15 15 15 15 15 15 15 15 15 15
GUI Load - North America
20 20 20 20 20 20 20 20 20 20
GUI Load - North America 2
200 200 200 200 200 200 200 200 200 200
Entitlement S
Entitlement S-V 2750 2750 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100
Simulating Production Beforehand, Mining Statistics After
Mario AndreoliSolution & Infrastructure Architect
Performance Analysis , Testing & Tuning
28 May 2013| 21
Day in the Life - Profile:
Elaboration
Map your volume pattern over a 24 hour clock
Simulating Production Beforehand, Mining Statistics After
Mario AndreoliSolution & Infrastructure Architect
Performance Analysis , Testing & Tuning
28 May 2013| 22
Test Harness Component Pattern� With our requirements solidified in the Day in Life profile, we now can start the
development of the Test Harness Components
� Each component:
— will do one thing and will do it well
— will follow the same general patternand specialize where needed
— Is responsible for managing its behaviour and parameters but will communicate with others when necessary
— will take the DIL profile as its input
— will persist statistics and data for analysis as its output
Construction
Application
Layer
Test
Component
1..* *
DIL Profile
1..*1
Simulating Production Beforehand, Mining Statistics After
Mario AndreoliSolution & Infrastructure Architect
Performance Analysis , Testing & Tuning
28 May 2013| 23
Test Harness Component� Physical implementation of a component depends on its role
� The example below shows an interface processing Test Harness component alongside a user load scenario
Construction
Simulating Production Beforehand, Mining Statistics After
Mario AndreoliSolution & Infrastructure Architect
Performance Analysis , Testing & Tuning
28 May 2013| 24
Data Factory� Messages used for testing are provided via one of three methods:
— Existing Production Data
— Data on any of the other systems on the network
— Custom message production based on: Rules & Templates
� Used by business users and functional testers to review data and use for their own tests and investigation
Construction
Simulating Production Beforehand, Mining Statistics After
Mario AndreoliSolution & Infrastructure Architect
Performance Analysis , Testing & Tuning
28 May 2013| 25
Statistics Monitor� During a test, statistics of the execution are critical in determining results and
effectiveness of each component and overall test success
� Transactions may be processed, but proper test execution and code coverage can only be confirmed by end state of the transactions
� Throughout the test, ongoing variances in expected results need to be proactively pushed to the respective parties so as to not compromise the overall test.
— Cost of running a test vs. a restart dictates level of granularity for notification
Construction
Simulating Production Beforehand, Mining Statistics After
Mario AndreoliSolution & Infrastructure Architect
Performance Analysis , Testing & Tuning
28 May 2013| 26
Statistics Monitor – Application Health� Application health comes from two distinct locations:
— Internal Application Health
— External Infrastructure
� Internal Application Health
— Transaction status (per business stream), user load, messages in/out etc.
— Anything critical to predict variances in business trends
— Watermarks can be set up for proactive notifications
� External Infrastructure
— CPU, disk, memory, network, queues, database health
— Prognosis, Dynatrace, Groundworks etc.
Test Execution
Simulating Production Beforehand, Mining Statistics After
Mario AndreoliSolution & Infrastructure Architect
Performance Analysis , Testing & Tuning
28 May 2013| 27
Test Execution
Statistics Monitor – Internal Health� Dashboard with all statistics
� historical comparisons and trends
Functional A
reas
Search CriteriaVersion,Feedback
Daily breakdown of data
History by hour
Graphical History
Simulating Production Beforehand, Mining Statistics After
Mario AndreoliSolution & Infrastructure Architect
Performance Analysis , Testing & Tuning
28 May 2013| 28
Success is Difficult, Iterations Make it Achievable
Test Results
In this scenario, a test harness component was injecting Trade messages
The application settles the trades based on business rules and sends a confirmation message when complete
Above: “message injected” (blue line) versus “messages processed” (pink line) had a significant delay due to poor processing performance.
Right: eight iterations later (with fine tuning and vendor fixes) Trade settlement met NFR SLA times
Simulating Production Beforehand, Mining Statistics After
Mario AndreoliSolution & Infrastructure Architect
Performance Analysis , Testing & Tuning
28 May 2013| 29
Other Significant Benefits
Organizational & Stream Readiness� Organizational Readiness wanted to give the operational support team an
opportunity to
— experience a “production day” before going live
— review the system capabilities, identify key risk areas and an indication of STP rates
— The business unit can carry out their day-to -ay work in an accurate and timely fashion
� Certain business streams wanted to understand what rate of “Straight Through Entry and Processing” (STE/P) would be achieved for the new system and compare that to the existing system
— A specific date was used as the baseline
— Flows and message types were chosen and collected from production for that date
— Once the test was executed statistics were compared and led to rule tweaks (in the new system) and re-runs to match STE/P rates
� Production Support
— Able to recreate production scenarios using the Test Harness
— Step through transactions as they happened in Production to debug issues
Simulating Production Beforehand, Mining Statistics After
Mario AndreoliSolution & Infrastructure Architect
Performance Analysis , Testing & Tuning
Production
� Application has been in production since April 1 2013
� Hardware, environment, user load and infrastructure all operating as predicted
� All NFR SLAs being met
� Final piece being added is a reporting module on the Statistics database
— Custom reports driven by users
� Utilities and tools promoted to Production and used by the executive team and production support to monitor application health
� A completely reusable set of processes and components adaptable to a varying number of systems (small to large)
28 May 2013| 30
Where are we today?
Simulating Production Beforehand, Mining Statistics After
Mario AndreoliSolution & Infrastructure Architect
Performance Analysis , Testing & Tuning
28 May 2013| 31
Thank You
Contact Information
� Twitter: @MarioAndreoli
� LinkedIn.com/in/MarioAndreoli
� Mario@acs1.com