BFS-US SSDU Research Best Strategies to avoid worst performance failures in Investment Banking...
-
Upload
charlene-chandler -
Category
Documents
-
view
214 -
download
0
Transcript of BFS-US SSDU Research Best Strategies to avoid worst performance failures in Investment Banking...
BFS-US SSDU ResearchBFS-US SSDU Research
Best Strategies to avoid worst performance failures in Investment Banking
Sujatha Badri NarayananKarthik Authoor Venkata
Infosys Limited (NASDAQ: INFY)
BFS-US SSDU ResearchBFS-US SSDU Research
Abstract
• Investment banking functionalities are grouped into front office, middle office and back office.
• Performance testing specifically related to trading applications within investment banking plays a vital role.
• Era has shifted from traditional manual trading in stock exchanges like NASDAQ, Globex to electronic trading.
• People trade millions of shares in micro seconds with various exchanges these days using electronic trading.
2
BFS-US SSDU ResearchBFS-US SSDU Research
Why this Paper?“Every large system that works started as a small system that worked.” – Anonymous
“Fast, good, cheap: pick any two.” – Anonymous
“Lots of methodologies have come and gone, paradigms have changed but the requirements are always the same; Make it good and make it fast.” – Anonymous
In Electronic trading, complex heterogeneous systems are spread over diverse technologies with various platforms ranging from legacy mainframe to today’s web based technologies are used.
Today’s systems and interfaces are built with their own protocols both at application layer and network layer.
Even minor performance improvement in these systems will yield mammoth benefits leading to savings in millions of dollars.
In this whitepaper we will attempt to describe some of the good performance testing strategies that can be followed to immunize ourselves from disasters in the Investment banking domain specifically in the trading business.
3
BFS-US SSDU ResearchBFS-US SSDU Research
Few Performance experiences as seen by User
4
Sl.No User experience Scenarios in the Trading Business
1 Session login failed.2 Market related data reaching the end user late due to network traffic rendering the info
useless3 Market data graphs failed to pull the real time data from the server
4 Viewing the detailed transaction statements on the web application giving distorted or half information due to delay in response from server
5 CMP(Current Market price) dashboard showing old market price for the stocks as the real time market update is slow
6 Placing a trade order from Stock trading web application during market mayhem (lot of trading activities happening at the same time). Server timed out.
7 Placed a trade order from web application during heavy traffic. Trade got executed but the confirmation never came back leaving the transaction in dilemma stage
8 Making a critical payment from bank web app to a third party using secure payment network. The money got debited from my account but due to heavy traffic on the third party server, the transaction failed.
9 Downloading “My” bank statement on the desktop which failed10 Making a mobile payment from network switches from 3G to Wireless and thus failing the
transaction
BFS-US SSDU ResearchBFS-US SSDU Research
Proposed Strategies for better user experiences
Following are the key strategies proposed, that might help to provide solutions to some of the
performance issues that have been experienced by users as listed in the previous slide:
1) Performance Testing from an Architect’s View – Performance test driven
architecture
2) Infrastructure Optimization like Server, Database, Network etc -
3) Optimal combination and usage of tools based on technology used – Fiddler,
Web Load, VSTS, Jmeter, LR, Wily Introscope, Dynatrace, Shunra, Strobe, Xcap
etc.
4) Component based testing – Stressing the system at component level
5) Seamless Failover Testing – Time taken for switch over during disaster5
BFS-US SSDU ResearchBFS-US SSDU Research
Strategy 5:
Seamless Failover Testing
Strategy 2:Infrastructure Optimization
Stra
tegy 4
:
Com
pon
ent b
ased
testing
Performance Strategy layers – An End-to-End View
6
Strategy 1 : Architect’s View
Strategy 3 : Optimal combination and usage of tools based on technology
Load Balanc
er
Web
Serv
er
Web
Serv
er
Ap
p S
erv
er
Ap
p S
erv
er
MF
DB
BFS-US SSDU ResearchBFS-US SSDU Research
Strategy 1: Performance Testing from an Architect’s View
7
Life Cycle Phases Performance Strategy Activities
Requirements gathering / analysis / validation
System understanding NFR gathering Understanding NFR & validating them Modeling
o NFRo Workload
Architecture Analysis & Design Scalability assessment of the architecture Validation of the design from a Performance
stand point Capacity and sizing planning of the
infrastructure Performance Predictive Modeling Designing the performance Test strategy
BFS-US SSDU ResearchBFS-US SSDU Research
Strategy 1: Performance Testing from an Architect’s View.. ContdFollowing are the performance test strategy activities (as mentioned against each
lifecycle phases in the previous slide) that can be leveraged
1) Performance Testing from an Architect’s View – Performance test driven
architecture
2) Infrastructure Optimization like Server, Database, Network etc -
3) Optimal combination and usage of tools based on technology used – Fiddler,
WebLoad, VSTS, Jmeter, LR, Wily Introscope, Dynatrace, Shunra, Xcap etc.
4) Component based testing – Stressing the system at component level
5) Seamless Failover Testing – Time taken for switch over during disaster
8
BFS-US SSDU ResearchBFS-US SSDU Research
Strategy 2: Infrastructure Optimization
Why Infrastructure Optimization ?• Major component in an organization’s CAPEX• Level of optimization is a key goal for many organizations• Superior user experience which is the core business should also not get
affected
Most Commonly used servers in Investment Banking
Web Server : Tomcat, IISApp Server : Tomcat, IIS, IBM WebSphereDatabase : Oracle 10g, 11g, Microsoft SQL Server
9
BFS-US SSDU ResearchBFS-US SSDU Research
Suggested optimizations on TomCat:
Have the latest OS patches, Service packs and JDKs installed as higher versions always have better performance and better security.
Have windows server instead of a non server version as non server version will have various limitations like number of users etc.
Connector tuning will yield good results – Generally Client creates the connection and once server sends response to the request connection is closed. One may land up opening 100s of connection to render a single page – Modify “MaxKeepAliveRequests” to a higher value option to avoid this.
Tomcat also has a thread pool. Make the best use of the same by adjusting minProcessors and maxProcessors values.10
Strategy 2: Infrastructure Optimization - Web Server
BFS-US SSDU ResearchBFS-US SSDU Research
Suggested optimizations on IIS:
Configure the following parameters like Shutdown worker processes when idle for good performance
Managing log files will result in good performance in almost all the servers. Make sure that only required values are selected for logging and old log files are archived to avoid space issue. 11
Strategy 2: Infrastructure Optimization - Web Server … Contd
BFS-US SSDU ResearchBFS-US SSDU Research 12
Strategy 2: Infrastructure Optimization - Application Server
Suggested optimizations on WebSphere Application Server for better performance :
Modify Initial Heap Size and Maximum Heap size to 3096
Have Loglevel severity to “error” or “fatal”
“show application print statements” can be disabled
IBM Service Logs can be disabled (Process Defnitions -> Logging & Tracking -> IBM Service Logs)
Have a strategy around connection pool management in WAS
BFS-US SSDU ResearchBFS-US SSDU Research 13
Strategy 2: Infrastructure Optimization – Database Server
Suggested optimizations on Oracle DB Server for better performance :
Ensure that the database is loaded to the required volume before Load testing.As in any other server system resources monitoring plays a vital role in DB server also.
Besides CPU, Memory, ensure that Disk I/ O is monitored. Ensure that all the disks are properly loaded.
Other major aspects of Oracle tuning are :- Denormalizing a database – If your database is doing too many joins then you
can consider this option to improve performance Indexing (B-Tree Concept)
- is used in databases to speed up retrieval process- is one of the most time tested technique that can be used to
improve performance Full Table Scan - it is better to go for full table scan for smaller tables
BFS-US SSDU ResearchBFS-US SSDU Research 14
Strategy 2: Infrastructure Optimization – Database Server .. Contd
Tuning SQL itself is another interesting topic :
You can make use of AWR logs and other monitoring tools to know what queries take more time and what queries are called often and then select the queries that should be
tuned.
Following are few samples of SQL tuning : - If you are retrieving from multiple tables, the sequence of the tables matters
a lot for performance. Optimizers (if tuned properly – cost optimization)does help in making such decisions.
If it is a read only table read once and cache them so that we can avoid reading multiple times
Do not read the entire table when only few columns are required. DB filters can be handy here.
BFS-US SSDU ResearchBFS-US SSDU Research 15
Strategy 3: Optimal combination and usage of tools
Selecting an optimal combination of tools plays a vital role in identifying bottlenecks.
While selecting load generation tools we may need to take into the account the following key factors:
Protocol Support required for your application like : Http, Win socket, SOAP, WAP, IMAP, POP3, EJB, RMI-Java, COM/DCOM, Tuxedo, Seibel, SAP R/3, Terminal Emulation, SSL, XML, PeopleSoft, VT100-520, ODBC, FTP, Streaming Media, LDAP, TCP/IP, UDP, MMS, Microsoft .NET SOAP Stack, CORBA, Oracle OCI, Ajax
Ease of Parameterization and Correlation Network address management User ramp up / down facility Scheduling tests License details Customer Support
BFS-US SSDU ResearchBFS-US SSDU Research 16
Strategy 3: Optimal combination and usage of tools… Contd
While selecting monitoring tools we may need to take into the account the following key factors:
Technology support required for your application - .Net, J2EE, Mainframe etc Supported system resources monitoring Depth in analysis For e.g. does it help in identifying bottlenecks at code level ?
Query level ? License details Customer SupportThere are many Performance tools in the market but based on our experience in
Investment banking we find that the following tools are most commonly used.
Strategic Areas / Technology
.Net J2EE Mainframe
Load Generation Tools VSTS HP LR , Radview’s Web load, Compuware QALoad, Borland ‘s Silk Performer, Microsoft Web Application Stress Tool (WAS)
HP LR using RTE protocol
Monitoring Tools IIS Debug Diagnostics
HP Diagnostics, Wily Introscope, Dynatrace, Site scope
Strobe
BFS-US SSDU ResearchBFS-US SSDU Research 17
Strategy 3: Optimal combination and usage of tools… Contd
There are two types of monitoring tools Agent based tools like HP Diagnostics ,Wily Introscope Agent less tools like Site Scope
Sample Agent based Monitoring Tool output – Wily Introscope
Detailed analysis up to code / query level can be done in agent based monitoring.
BFS-US SSDU ResearchBFS-US SSDU Research
SLAs should be broken down during architecture phase for each of the components.
Components should be individually load/volume tested to make sure that they meet required SLA.
This will ease bottleneck identification and help in addressing the same at an earlier stage rather than struggling during final phase of the project.
18
Strategy 4: Component based Testing
BFS-US SSDU ResearchBFS-US SSDU Research
In investment banking as availability at the right time matters most, failover testing is essential.
In failover testing , the behavior of the system will be tested by bringing down the servers in a logical way – web, app etc.
In many instances CPU utilization was high and later during analysis we realized that Load Balancers were not working as expected.
Make sure that your strategy covers testing load balancers also.
19
Strategy 5: Failover Testing
BFS-US SSDU ResearchBFS-US SSDU Research
Designing for the right performance and ensuring that performance is engineered into the product / application is key.
Specifically in investment banking business , organizations cannot afford to incur higher expenditure on application failures and higher opex on unnecessary infrastructure, hence performance testing becomes an indispensable activity for optimizing the quality of the application
We recommend organizations to view performance quality from the bigger picture and adopt the best strategies which were discussed during these sessions so that business at large is benefited thereby making organizations less vulnerable to competitive pressures.
20
The Way ahead
BFS-US SSDU ResearchBFS-US SSDU Research 21
Conclusion and Results
Performance testers should also focus on performance engineering standpoint in addition to load testing / volume testing strategies.
User experience Scenarios Strategy1
Session login failed.Load beyond expected may be the reason. Capacity Modeling could have avoided this situation – Strategy 1
2 Market related data reaching the end user late due to network traffic rendering the info useless
Response time not meeting SLAs – Strategy 2 & 3
3 Market data graphs failed to pull the real time data from the server
May be the server responsible for pulling the data is down. Strategy 5
4 Viewing the detailed transaction statements on the web application giving distorted or half information due to delay in response from server
Database Server response may be slow. Strategy 3
5 CMP(Current Market price) dashboard showing old market price for the stocks as the real time market update is slow
Possible reason could be bottleneck in one of the components – Strategy 4
BFS-US SSDU ResearchBFS-US SSDU Research 22
Conclusion and Results … Contd
User experience Scenarios Strategy6 Placing a trade order from Stock trading
web application during market mayhem (lot of trading activities happening at the same time). Server timed out.
Load beyond expected may be the reason. Capacity Modeling / Proper Workload modeling could have avoided this situation – Strategy 1
7 Placed a trade order from web application during heavy traffic. Trade got executed but the confirmation never came back leaving the transaction in dilemma stage
Incomplete transaction due to poor response time. Possible reason could be bottleneck in one of the layers / components – Strategy 4
8 Making a critical payment from bank web app to a third party using secure payment network. The money got debited from my account but due to heavy traffic on the third party server the transaction failed.
Possible reason could be bottleneck in one of the layers / components – Strategy 4
BFS-US SSDU ResearchBFS-US SSDU Research
References and Acknowledgements
References
http://msdn.microsoft.com
Tomcat – The definite guide by Jason Brittain & Ian F. Darwin
Performance Tuning for Apache Tomcat by Mark Thomas
Oracle Performance Tuning by Peter Corrigan and Mark Gurry
Performance Testing Advanced Certification material from Infosys
Acknowledgements
Vijay from Infosys Infrastructure Management Services
Sanjeeb Kumar from Financial Services team for sharing the user experiences
Finacle SPE team
BFS-US SSDU ResearchBFS-US SSDU Research
24