Simulating Production Beforehand Mining Statistics Afterward
Transcript of Simulating Production Beforehand Mining Statistics Afterward
![Page 1: Simulating Production Beforehand Mining Statistics Afterward](https://reader031.fdocuments.net/reader031/viewer/2022012223/61e08bf88865e6602d74d393/html5/thumbnails/1.jpg)
Simulating Production Beforehand, Mining Statistics After
Mario AndreoliSolution & Infrastructure Architect
Performance Analysis , Testing & Tuning
Simulating Production BeforehandMining Statistics Afterward
May 28 2013
TASSQ Presentation
@MarioAndreoli
LinkedIn.com/in/MarioAndreoli
![Page 2: Simulating Production Beforehand Mining Statistics Afterward](https://reader031.fdocuments.net/reader031/viewer/2022012223/61e08bf88865e6602d74d393/html5/thumbnails/2.jpg)
Simulating Production Beforehand, Mining Statistics After
Mario AndreoliSolution & Infrastructure Architect
Performance Analysis , Testing & Tuning
28 May 2013| 2
� Introduction
� Challenge - Certifying performance on a mission critical financial application
� Solution to the “Performance Certification” challenge
� Inception – understand what needs to be done
� Elaboration – explain how the solution will work
� Construction – build the test components and a data factory
� Test Execution and Test Results
� Additional Benefits
� Where are we today?
Agenda
![Page 3: Simulating Production Beforehand Mining Statistics Afterward](https://reader031.fdocuments.net/reader031/viewer/2022012223/61e08bf88865e6602d74d393/html5/thumbnails/3.jpg)
Simulating Production Beforehand, Mining Statistics After
Mario AndreoliSolution & Infrastructure Architect
Performance Analysis , Testing & Tuning
28 May 2013| 3
� Historically a Developer, Solution Architect
� Strengths (certifications) in RUP, OOAD, J2EE
� “Reverse Engineering” strengths
� Created a bridge between Technical & Testing
� Teams consist of T/BSAs, Developers, Testers
� Certified the performance of a $250 million financial application
— not in the negative “Gartner Statistic” group
Introduction
![Page 4: Simulating Production Beforehand Mining Statistics Afterward](https://reader031.fdocuments.net/reader031/viewer/2022012223/61e08bf88865e6602d74d393/html5/thumbnails/4.jpg)
Simulating Production Beforehand, Mining Statistics After
Mario AndreoliSolution & Infrastructure Architect
Performance Analysis , Testing & Tuning
28 May 2013| 4
Questions:
� How will it perform in Production?
— Is the Hardware sized appropriately?
— Is the Software tuned efficiently?
—Will the solution handle the anticipated volume?
—Are we confident transaction processing will have a significant percentage of straight through processing?
—Can the communication infrastructure handle the load?
—Will the end users have an acceptable experience?
—We can’t wait for Production to catch issues!
� How do I build confidence for the Stakeholders?
— Top down support is crucial. How do we get it?
— How can we build a plan that Stakeholders buy into?
General Challenges
![Page 5: Simulating Production Beforehand Mining Statistics Afterward](https://reader031.fdocuments.net/reader031/viewer/2022012223/61e08bf88865e6602d74d393/html5/thumbnails/5.jpg)
Simulating Production Beforehand, Mining Statistics After
Mario AndreoliSolution & Infrastructure Architect
Performance Analysis , Testing & Tuning
Answers:
� How will it perform in Production?
—Design and create a working model thataccurately simulates the solution before it is in Production
� How do I build confidence for the Stakeholders?
—Provide tools Stakeholders can use to be part of the process and monitor business healththroughout and after
28 May 2013| 5
We Can Address It
![Page 6: Simulating Production Beforehand Mining Statistics Afterward](https://reader031.fdocuments.net/reader031/viewer/2022012223/61e08bf88865e6602d74d393/html5/thumbnails/6.jpg)
Simulating Production Beforehand, Mining Statistics After
Mario AndreoliSolution & Infrastructure Architect
Performance Analysis , Testing & Tuning
28 May 2013| 6
Technical Challenges:
� Vendor Application
— no source access, little architectural information, no data model
� Complex Workflow Engine
— Senior RBT resource estimated an extremely high possible number of paths (test cases)
� 40+ interfaces / 250+ concurrent users
� Multiple functional streams
— Each with their own critical and complex requirements
� Global access (interfaces & users)
� Millions of varying types of messages daily
— exponentially more transactions
Inception
![Page 7: Simulating Production Beforehand Mining Statistics Afterward](https://reader031.fdocuments.net/reader031/viewer/2022012223/61e08bf88865e6602d74d393/html5/thumbnails/7.jpg)
Simulating Production Beforehand, Mining Statistics After
Mario AndreoliSolution & Infrastructure Architect
Performance Analysis , Testing & Tuning
28 May 2013| 7
Financial Challenges:
� Market close deadlines
� Financial critical transactions
� Vital upstream and downstream SLAs
� This has to work, $000,000,000’s depend on it
— We do not want to be (negatively) in the newspaper the next day
� Manual functional testing of all production (financial) data scenarios almost impossible
� Distributed geographical markets
Inception
![Page 8: Simulating Production Beforehand Mining Statistics Afterward](https://reader031.fdocuments.net/reader031/viewer/2022012223/61e08bf88865e6602d74d393/html5/thumbnails/8.jpg)
Simulating Production Beforehand, Mining Statistics After
Mario AndreoliSolution & Infrastructure Architect
Performance Analysis , Testing & Tuning
28 May 2013| 8
Approach:
� Senior QA resource team came to a conclusion that in addition to a good test strategy, a different approach was needed
� A strong technical resource with ‘out of the box’ thinking
� Iterative with quick and frequent deliveries
� Built in layers – with agile yet controlled processing
� Quick iterations and deployments to:
— start the testing early…learn early
— easily repetitive (building the regression suite),
— no (or minimal) manual interventiontests
statistics
learning
decisions
fixes
Resolution
Inception
![Page 9: Simulating Production Beforehand Mining Statistics Afterward](https://reader031.fdocuments.net/reader031/viewer/2022012223/61e08bf88865e6602d74d393/html5/thumbnails/9.jpg)
Simulating Production Beforehand, Mining Statistics After
Mario AndreoliSolution & Infrastructure Architect
Performance Analysis , Testing & Tuning
28 May 2013| 9
Iterative Approach:
Inception
� Build layers of tests that can all talk to each other and be used in harmony to do full performance, load, regression testing etc.
![Page 10: Simulating Production Beforehand Mining Statistics Afterward](https://reader031.fdocuments.net/reader031/viewer/2022012223/61e08bf88865e6602d74d393/html5/thumbnails/10.jpg)
Simulating Production Beforehand, Mining Statistics After
Mario AndreoliSolution & Infrastructure Architect
Performance Analysis , Testing & Tuning
28 May 2013| 10
White-Boxing a Black-Box:
� Understand business concepts?
� Vendor + Business Stream
� Flows, Begin-Status, End-Status, Rules
� Sample Messages (in/out)
� Understand the architecture through Reverse Engineering
� Structural Diagrams:
� Deployment, Package, Component
� Hardware, Software, Communication
� Interfaces and Message Transportation
� Behaviour Diagrams:
� Activity, State, Sequence
� Major Flows for System Behaviour
� Guide other teams/groups
� Functional coverage
� Code coverage
Elaboration
![Page 11: Simulating Production Beforehand Mining Statistics Afterward](https://reader031.fdocuments.net/reader031/viewer/2022012223/61e08bf88865e6602d74d393/html5/thumbnails/11.jpg)
Simulating Production Beforehand, Mining Statistics After
Mario AndreoliSolution & Infrastructure Architect
Performance Analysis , Testing & Tuning
Performance Test Scope:
Elaboration
Performance
TestingTypes
Overall
Program
Components
Dependencies
Load
Endurance
Repetitive
Stress
H/W, Network,
Database
(optional)
Timing to
Volumes and SLA
(Performance NFR)
FinancialApp User
Interface
Response Time
Interface:
SI Processing Time
Batch Cyle Time
FinancialApp
User Interface
Interface
Processing
Batch
Processing
Risk
Volume
Importance
EnvironmentFunctional
Testing
Tuning
ESB/FTP Transportation
Failover & DR
Performance
External System
Performance
Automation
Tools
User Interface
Loadrunner
Interface
Harness
Batch
Schedule
Management
Database
Reset
(Flashback)
Stability over long periods
H/W Resource Management
Finding Ceiling of
Architectural Component
Realistic Production
Volumes & Timelines
Anticipated growth
Exact same execution,
repeated to determine
transaction accuracy
Stream:
Trade Processing
Cash
Entitlement
Tax
Integration
Corporate Actions
Query
User Types:
Inquiry (Detail, Index, Wildcard)
Update
40+ Interfaces
Day in the Life
of
FinancialApp
Database:
--Start
--Growth
IntradayEOD/BOD
FinancialApp version
Core version
FinancialApp Deployment Footprint
All other infrastructure components
Technical Test Team
Functional Test Team
Vendor
IT Support
SCM
Environment Coordinator
LoadRunner
Infrastructure Resources (DBA,
Capacity & Planning, Websphere etc.)
Geographic:
Europe users
BI
Entitlement &
Position
Generation
Out of Scope
Supporting
Resources
“A Day In
The Life”
All Overall Program
components running
together over a 24 hour
period
Measurements
Prioritization
Statistics
Monitoring
![Page 12: Simulating Production Beforehand Mining Statistics Afterward](https://reader031.fdocuments.net/reader031/viewer/2022012223/61e08bf88865e6602d74d393/html5/thumbnails/12.jpg)
Simulating Production Beforehand, Mining Statistics After
Mario AndreoliSolution & Infrastructure Architect
Performance Analysis , Testing & Tuning
28 May 2013| 12
Business Concepts: Prioritize & Understand� Select your high volume, critical business flows to understand and build
harnesses for
� The scenario below is 1of 4 built (there were 10 scenarios in total, but the 4 selected consisted of 90% of the overall volume)
Elaboration
![Page 13: Simulating Production Beforehand Mining Statistics Afterward](https://reader031.fdocuments.net/reader031/viewer/2022012223/61e08bf88865e6602d74d393/html5/thumbnails/13.jpg)
Simulating Production Beforehand, Mining Statistics After
Mario AndreoliSolution & Infrastructure Architect
Performance Analysis , Testing & Tuning
28 May 2013| 13
Architecture � We take each scenario and perform a volume analysis
� Map volume per time period
� Analyze peaks
� Confirm your findings with business resources
Elaboration
![Page 14: Simulating Production Beforehand Mining Statistics Afterward](https://reader031.fdocuments.net/reader031/viewer/2022012223/61e08bf88865e6602d74d393/html5/thumbnails/14.jpg)
Simulating Production Beforehand, Mining Statistics After
Mario AndreoliSolution & Infrastructure Architect
Performance Analysis , Testing & Tuning
28 May 2013| 14
Hardware: Technical Understanding & Architecture
Elaboration
Reverse engineer the system and create the diagrams to help you understand the application:
Deployment: The physical layout of servers/nodes and their roles
It may also depict any load balancing strategies
Communication amongst servers/nodes/components should be depicted as well
![Page 15: Simulating Production Beforehand Mining Statistics Afterward](https://reader031.fdocuments.net/reader031/viewer/2022012223/61e08bf88865e6602d74d393/html5/thumbnails/15.jpg)
Simulating Production Beforehand, Mining Statistics After
Mario AndreoliSolution & Infrastructure Architect
Performance Analysis , Testing & Tuning
28 May 2013| 15
Software: Technical Understanding & Architecture
Elaboration
Reverse engineer the system and create the diagrams to help you understand the application:
Component: Determine the ‘pieces’ of the system and map the relationships between these different ‘pieces’
![Page 16: Simulating Production Beforehand Mining Statistics Afterward](https://reader031.fdocuments.net/reader031/viewer/2022012223/61e08bf88865e6602d74d393/html5/thumbnails/16.jpg)
Simulating Production Beforehand, Mining Statistics After
Mario AndreoliSolution & Infrastructure Architect
Performance Analysis , Testing & Tuning
28 May 2013| 16
Interface Transportation & Processing� Understand the major points and paths whereby messages will flow in and out
of the application
� Understand scope, which caveats you need to state in test results
� Point of binding for test harness depends on purpose, capability, constraints
Elaboration
![Page 17: Simulating Production Beforehand Mining Statistics Afterward](https://reader031.fdocuments.net/reader031/viewer/2022012223/61e08bf88865e6602d74d393/html5/thumbnails/17.jpg)
Simulating Production Beforehand, Mining Statistics After
Mario AndreoliSolution & Infrastructure Architect
Performance Analysis , Testing & Tuning
28 May 2013| 17
Tying It All Together:A Day in the Life (DIL) of the Financial Application:
� Map all business layers on a 24 hour clock
� Each layer can have many components running to execute the pertinent tests
Elaboration
24 Hour Clock (EST)
Activity 0:00 1:00 2:00 3:00 4:00 5:00 6:00 7:00 8:00 9:00 10:00 11:00 12:00 13:00 14:00 15:00 16:00 17:00 18:00 19:00 20:00 21:00 22:00 23:00
Batch Processing Intraday Batches EOD Batches
Online - Europe
Online - North America
Trade Transactions
Annoucement Interfaces
Other Interfaces
Entitlements Processing
BI Data Replication
Application
Layer
Test
Component
1..* *
DIL Profile
1..*1
![Page 18: Simulating Production Beforehand Mining Statistics Afterward](https://reader031.fdocuments.net/reader031/viewer/2022012223/61e08bf88865e6602d74d393/html5/thumbnails/18.jpg)
Simulating Production Beforehand, Mining Statistics After
Mario AndreoliSolution & Infrastructure Architect
Performance Analysis , Testing & Tuning
28 May 2013| 18
Tying It All Together:A Day in the Life of the Financial Application:
� At this point we have a good understanding of critical:
— functions, flows, volumes, processesand timing
� We now need a profile that unifies all our data and understanding
� Circulate this profile for agreement and sign off
� This will be our contract starting point for our physical design, prototyping and construction
� If we can emulate a 24 hour period, we now have a baseline to test with
Elaboration
![Page 19: Simulating Production Beforehand Mining Statistics Afterward](https://reader031.fdocuments.net/reader031/viewer/2022012223/61e08bf88865e6602d74d393/html5/thumbnails/19.jpg)
Simulating Production Beforehand, Mining Statistics After
Mario AndreoliSolution & Infrastructure Architect
Performance Analysis , Testing & Tuning
28 May 2013| 19
Tying It All Together:A Day in the Life of the Financial Application:
� A component:
— can be a user scenario, interface feed, business process etc.
— is its own “thread” in the final simulator
— can be adjusted:
� On/Off
� Volume increase/decrease
� Duration time
� Period spike/lull
� Volume over Duration
—will be recordingstatistics and pertinent data
Elaboration
CC11
CC22
CC33
CCxx
DIL Profile
![Page 20: Simulating Production Beforehand Mining Statistics Afterward](https://reader031.fdocuments.net/reader031/viewer/2022012223/61e08bf88865e6602d74d393/html5/thumbnails/20.jpg)
Simulating Production Beforehand, Mining Statistics After
Mario AndreoliSolution & Infrastructure Architect
Performance Analysis , Testing & Tuning
28 May 2013| 20
Day in the Life - Profile:
Elaboration
24 hour clock, broken up in 30 minute intervals
Interfaces, Business Processes,
GUI Lo
ad Profile
24 Hour Clock 0:00 1:00 2:00 3:00 4:00 5:00 6:00 7:00 8:00 9:00 10:00 11:00 12:00 13:00 14:00 15:00 16:00 17:00GUI Load Time Line 0:00 1:00 2:00 3:00 4:00 5:00 6:00 7:00 8:00 9:00 10:00 11:00 12:00 13:00 14:00 15:00GUI Load - Europe 15 15 15 15 15 15 15 15 15 15
GUI Load - North America
20 20 20 20 20 20 20 20 20 20
GUI Load - North America 2
200 200 200 200 200 200 200 200 200 200
Entitlement S
Entitlement S-V 2750 2750 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100
![Page 21: Simulating Production Beforehand Mining Statistics Afterward](https://reader031.fdocuments.net/reader031/viewer/2022012223/61e08bf88865e6602d74d393/html5/thumbnails/21.jpg)
Simulating Production Beforehand, Mining Statistics After
Mario AndreoliSolution & Infrastructure Architect
Performance Analysis , Testing & Tuning
28 May 2013| 21
Day in the Life - Profile:
Elaboration
Map your volume pattern over a 24 hour clock
![Page 22: Simulating Production Beforehand Mining Statistics Afterward](https://reader031.fdocuments.net/reader031/viewer/2022012223/61e08bf88865e6602d74d393/html5/thumbnails/22.jpg)
Simulating Production Beforehand, Mining Statistics After
Mario AndreoliSolution & Infrastructure Architect
Performance Analysis , Testing & Tuning
28 May 2013| 22
Test Harness Component Pattern� With our requirements solidified in the Day in Life profile, we now can start the
development of the Test Harness Components
� Each component:
— will do one thing and will do it well
— will follow the same general patternand specialize where needed
— Is responsible for managing its behaviour and parameters but will communicate with others when necessary
— will take the DIL profile as its input
— will persist statistics and data for analysis as its output
Construction
Application
Layer
Test
Component
1..* *
DIL Profile
1..*1
![Page 23: Simulating Production Beforehand Mining Statistics Afterward](https://reader031.fdocuments.net/reader031/viewer/2022012223/61e08bf88865e6602d74d393/html5/thumbnails/23.jpg)
Simulating Production Beforehand, Mining Statistics After
Mario AndreoliSolution & Infrastructure Architect
Performance Analysis , Testing & Tuning
28 May 2013| 23
Test Harness Component� Physical implementation of a component depends on its role
� The example below shows an interface processing Test Harness component alongside a user load scenario
Construction
![Page 24: Simulating Production Beforehand Mining Statistics Afterward](https://reader031.fdocuments.net/reader031/viewer/2022012223/61e08bf88865e6602d74d393/html5/thumbnails/24.jpg)
Simulating Production Beforehand, Mining Statistics After
Mario AndreoliSolution & Infrastructure Architect
Performance Analysis , Testing & Tuning
28 May 2013| 24
Data Factory� Messages used for testing are provided via one of three methods:
— Existing Production Data
— Data on any of the other systems on the network
— Custom message production based on: Rules & Templates
� Used by business users and functional testers to review data and use for their own tests and investigation
Construction
![Page 25: Simulating Production Beforehand Mining Statistics Afterward](https://reader031.fdocuments.net/reader031/viewer/2022012223/61e08bf88865e6602d74d393/html5/thumbnails/25.jpg)
Simulating Production Beforehand, Mining Statistics After
Mario AndreoliSolution & Infrastructure Architect
Performance Analysis , Testing & Tuning
28 May 2013| 25
Statistics Monitor� During a test, statistics of the execution are critical in determining results and
effectiveness of each component and overall test success
� Transactions may be processed, but proper test execution and code coverage can only be confirmed by end state of the transactions
� Throughout the test, ongoing variances in expected results need to be proactively pushed to the respective parties so as to not compromise the overall test.
— Cost of running a test vs. a restart dictates level of granularity for notification
Construction
![Page 26: Simulating Production Beforehand Mining Statistics Afterward](https://reader031.fdocuments.net/reader031/viewer/2022012223/61e08bf88865e6602d74d393/html5/thumbnails/26.jpg)
Simulating Production Beforehand, Mining Statistics After
Mario AndreoliSolution & Infrastructure Architect
Performance Analysis , Testing & Tuning
28 May 2013| 26
Statistics Monitor – Application Health� Application health comes from two distinct locations:
— Internal Application Health
— External Infrastructure
� Internal Application Health
— Transaction status (per business stream), user load, messages in/out etc.
— Anything critical to predict variances in business trends
— Watermarks can be set up for proactive notifications
� External Infrastructure
— CPU, disk, memory, network, queues, database health
— Prognosis, Dynatrace, Groundworks etc.
Test Execution
![Page 27: Simulating Production Beforehand Mining Statistics Afterward](https://reader031.fdocuments.net/reader031/viewer/2022012223/61e08bf88865e6602d74d393/html5/thumbnails/27.jpg)
Simulating Production Beforehand, Mining Statistics After
Mario AndreoliSolution & Infrastructure Architect
Performance Analysis , Testing & Tuning
28 May 2013| 27
Test Execution
Statistics Monitor – Internal Health� Dashboard with all statistics
� historical comparisons and trends
Functional A
reas
Search CriteriaVersion,Feedback
Daily breakdown of data
History by hour
Graphical History
![Page 28: Simulating Production Beforehand Mining Statistics Afterward](https://reader031.fdocuments.net/reader031/viewer/2022012223/61e08bf88865e6602d74d393/html5/thumbnails/28.jpg)
Simulating Production Beforehand, Mining Statistics After
Mario AndreoliSolution & Infrastructure Architect
Performance Analysis , Testing & Tuning
28 May 2013| 28
Success is Difficult, Iterations Make it Achievable
Test Results
In this scenario, a test harness component was injecting Trade messages
The application settles the trades based on business rules and sends a confirmation message when complete
Above: “message injected” (blue line) versus “messages processed” (pink line) had a significant delay due to poor processing performance.
Right: eight iterations later (with fine tuning and vendor fixes) Trade settlement met NFR SLA times
![Page 29: Simulating Production Beforehand Mining Statistics Afterward](https://reader031.fdocuments.net/reader031/viewer/2022012223/61e08bf88865e6602d74d393/html5/thumbnails/29.jpg)
Simulating Production Beforehand, Mining Statistics After
Mario AndreoliSolution & Infrastructure Architect
Performance Analysis , Testing & Tuning
28 May 2013| 29
Other Significant Benefits
Organizational & Stream Readiness� Organizational Readiness wanted to give the operational support team an
opportunity to
— experience a “production day” before going live
— review the system capabilities, identify key risk areas and an indication of STP rates
— The business unit can carry out their day-to -ay work in an accurate and timely fashion
� Certain business streams wanted to understand what rate of “Straight Through Entry and Processing” (STE/P) would be achieved for the new system and compare that to the existing system
— A specific date was used as the baseline
— Flows and message types were chosen and collected from production for that date
— Once the test was executed statistics were compared and led to rule tweaks (in the new system) and re-runs to match STE/P rates
� Production Support
— Able to recreate production scenarios using the Test Harness
— Step through transactions as they happened in Production to debug issues
![Page 30: Simulating Production Beforehand Mining Statistics Afterward](https://reader031.fdocuments.net/reader031/viewer/2022012223/61e08bf88865e6602d74d393/html5/thumbnails/30.jpg)
Simulating Production Beforehand, Mining Statistics After
Mario AndreoliSolution & Infrastructure Architect
Performance Analysis , Testing & Tuning
Production
� Application has been in production since April 1 2013
� Hardware, environment, user load and infrastructure all operating as predicted
� All NFR SLAs being met
� Final piece being added is a reporting module on the Statistics database
— Custom reports driven by users
� Utilities and tools promoted to Production and used by the executive team and production support to monitor application health
� A completely reusable set of processes and components adaptable to a varying number of systems (small to large)
28 May 2013| 30
Where are we today?
![Page 31: Simulating Production Beforehand Mining Statistics Afterward](https://reader031.fdocuments.net/reader031/viewer/2022012223/61e08bf88865e6602d74d393/html5/thumbnails/31.jpg)
Simulating Production Beforehand, Mining Statistics After
Mario AndreoliSolution & Infrastructure Architect
Performance Analysis , Testing & Tuning
28 May 2013| 31
Thank You
Contact Information
� Twitter: @MarioAndreoli
� LinkedIn.com/in/MarioAndreoli