Symantec I3 - Inquire & Cluster

56
Symantec i 3 AVAILABILITY & PERFORMANCE

description

Performance Management form synthetic transactions, Web, J2EE, OS and Cluster HW

Transcript of Symantec I3 - Inquire & Cluster

Page 1: Symantec I3 - Inquire & Cluster

Symantec i3

AVAILABILITY & PERFORMANCE

Page 2: Symantec I3 - Inquire & Cluster

2 Symantec Corporation, All Rights Reserved 2005

The Web Applications Environment

Multi-tier applications that can include:– Client (Web browser)

– Web Server

– J2EE Application Server

– Interfaces to external systems

– Database

Worldwide, around-the-clock user access

Usage patterns are not controlled

Management requires the expertise of multiple IT teams

Page 3: Symantec I3 - Inquire & Cluster

3 Symantec Corporation, All Rights Reserved 2005

Architecture

WebClients

WebClients

Web Servers

J2EE Server

Oracle Databases Legacy System

ExternalSystem

Page 4: Symantec I3 - Inquire & Cluster

4 Symantec Corporation, All Rights Reserved 2005

Symantec i3

Symantec iSymantec i33 is an is an integratedintegrated solution that is solution that is end-user focusedend-user focused. .

Symantec iSymantec i33 provides a built-in provides a built-in methodologymethodology that enables that enables you to effectively manage the you to effectively manage the AvailabilityAvailability and and PerformancePerformance

of business applications.of business applications.

Page 5: Symantec I3 - Inquire & Cluster

5 Symantec Corporation, All Rights Reserved 2005

Why an Integrated Solution?

With a multi-tier architecture, IT must:

Align IT to business priorities– Monitor the availability and performance of “business processes”– Monitor “business processes” 24x7 from multiple geographies

– Constantly update the “business processes” by adjusting to real usage patterns initiated by real users

Easily & rapidly detect problems, & analyze in context

Correlate application flow activities among technologies

Pinpoint & analyze their causes

Implement proactive tuning of your performance and availability

Page 6: Symantec I3 - Inquire & Cluster

6 Symantec Corporation, All Rights Reserved 2005

AVAILABILITY & PERFORMANCE

Page 7: Symantec I3 - Inquire & Cluster

7 Symantec Corporation, All Rights Reserved 2005

AVAILABILITY & PERFORMANCE

Page 8: Symantec I3 - Inquire & Cluster

8 Symantec Corporation, All Rights Reserved 2005

Case Study

Enterprise web application running on a 4-tier architecture

International flower ordering service– The system is critical to sales & support activities– The worldwide data center is located in NY

– The central orders database is updated at 10:00pm NY time

– Sales peak around local holidays, e.g. Mother’s Day on May 9 in Italy & USA

Page 9: Symantec I3 - Inquire & Cluster

9 Symantec Corporation, All Rights Reserved 2005

Page 10: Symantec I3 - Inquire & Cluster

10 Symantec Corporation, All Rights Reserved 2005

Page 11: Symantec I3 - Inquire & Cluster

11 Symantec Corporation, All Rights Reserved 2005

Page 12: Symantec I3 - Inquire & Cluster

12 Symantec Corporation, All Rights Reserved 2005

Status

Conclusions – symptoms detected Conclusions – symptoms detected

A problem was identified in the check_my_order business transaction from USA & Italy

A synthetic transaction identified the problem

The problem occurred after database update, at 10:05pm NY time (4:05am the next day in Italy)

Next Steps – find the sourceNext Steps – find the source

Identify the problematic page component

Page 13: Symantec I3 - Inquire & Cluster

13 Symantec Corporation, All Rights Reserved 2005

Page 14: Symantec I3 - Inquire & Cluster

14 Symantec Corporation, All Rights Reserved 2005

Page 15: Symantec I3 - Inquire & Cluster

15 Symantec Corporation, All Rights Reserved 2005

Status

Conclusions – source foundConclusions – source found

Java class /verify/orders/country/orders_list.jsp is the problematic component

Next Steps – focus on the reasonNext Steps – focus on the reason

Identify whether this problem also occurs with real transactions

Page 16: Symantec I3 - Inquire & Cluster

16 Symantec Corporation, All Rights Reserved 2005

AVAILABILITY & PERFORMANCE

Page 17: Symantec I3 - Inquire & Cluster

17 Symantec Corporation, All Rights Reserved 2005

Page 18: Symantec I3 - Inquire & Cluster

18 Symantec Corporation, All Rights Reserved 2005

Page 19: Symantec I3 - Inquire & Cluster

19 Symantec Corporation, All Rights Reserved 2005

Page 20: Symantec I3 - Inquire & Cluster

20 Symantec Corporation, All Rights Reserved 2005

Status

Interim Conclusions – focus on the reasonInterim Conclusions – focus on the reason

We have contrasted synthetic transactions vs. reality

The problem only occurs with synthetic transactions, due to timing (10:05pm NY / 4:05am Italy)

Real users will be affected once the next day starts - Mother’s Day in Italy, USA & some other countries

Next StepsNext Steps

Investigate server side to pinpoint the problematic tier

Page 21: Symantec I3 - Inquire & Cluster

21 Symantec Corporation, All Rights Reserved 2005

Page 22: Symantec I3 - Inquire & Cluster

22 Symantec Corporation, All Rights Reserved 2005

Page 23: Symantec I3 - Inquire & Cluster

23 Symantec Corporation, All Rights Reserved 2005

Status

Interim Conclusions – focus on the reasonInterim Conclusions – focus on the reason

The J2EE tier is dominating resource consumption

The problematic method is verify_orders_country_orders_list._jspService

Next StepsNext Steps

Further investigation is needed in the J2EE tier

Page 24: Symantec I3 - Inquire & Cluster

24 Symantec Corporation, All Rights Reserved 2005

AVAILABILITY & PERFORMANCE

Page 25: Symantec I3 - Inquire & Cluster

25 Symantec Corporation, All Rights Reserved 2005

Page 26: Symantec I3 - Inquire & Cluster

26 Symantec Corporation, All Rights Reserved 2005

Page 27: Symantec I3 - Inquire & Cluster

27 Symantec Corporation, All Rights Reserved 2005

Status

Interim Conclusions – focus on the reasonInterim Conclusions – focus on the reason

Smartune ranked JDBC Access Major Time as the main problem

Next StepsNext Steps

Seek more details and advice

Page 28: Symantec I3 - Inquire & Cluster

28 Symantec Corporation, All Rights Reserved 2005

Page 29: Symantec I3 - Inquire & Cluster

29 Symantec Corporation, All Rights Reserved 2005

Page 30: Symantec I3 - Inquire & Cluster

30 Symantec Corporation, All Rights Reserved 2005

Status

Conclusions Conclusions

The problem is not the database tier, but how frequently the database is accessed

Due to the unusually high number of db queries, time spent in the JSP is too long

Italy and USA behavior differs from other countries’ due to the Mother’s Day high increase in orders

Next Steps – address the problem & verify the solutionNext Steps – address the problem & verify the solution

Instead of querying each order item individually, a new single query that returns all items should be implemented

Page 31: Symantec I3 - Inquire & Cluster

31 Symantec Corporation, All Rights Reserved 2005

Page 32: Symantec I3 - Inquire & Cluster

32 Symantec Corporation, All Rights Reserved 2005

Case Study SummaryWe have successfully completed the methodology cycle:

DetectDetect

An SLA breach problem was detected by a synthetic transaction in Italy and USA, before the problem was experienced by real users

FindFindA specific JSP is responsible for the slow response time, but only in specific countries

FocusFocus The unusually long order list in Italy & USA caused an exceptional number of connections, resulting in lengthy JDBC access time

ImproveImproveRestructure queries and results pages

VerifyVerifyThe SLA problem has disappeared without affecting real users

Page 33: Symantec I3 - Inquire & Cluster

33 Symantec Corporation, All Rights Reserved 2005

Synthetic Transactions Versus Reality

Synthetic transactions indicate availability

They should systematically reflect a dynamic real-world situation

– Identify application usage changes relating to both existing and new applications

– Reflect new delivered functionality

– Reflect trends regarding usage from new locations

Page 34: Symantec I3 - Inquire & Cluster

34 Symantec Corporation, All Rights Reserved 2005

Page 35: Symantec I3 - Inquire & Cluster

35 Symantec Corporation, All Rights Reserved 2005

Page 36: Symantec I3 - Inquire & Cluster

36 Symantec Corporation, All Rights Reserved 2005

Page 37: Symantec I3 - Inquire & Cluster

37 Symantec Corporation, All Rights Reserved 2005

Synthetic Transactions Versus Reality

Summary Summary

Systematically analyzed changes in usage pattern

Identified new used functionality, change in usage patterns, and heavily used locations

Based on the findings, the user can now adjust the synthetic transactions to reflect the actual real-world situation

Page 38: Symantec I3 - Inquire & Cluster

38 Symantec Corporation, All Rights Reserved 2005

AVAILABILITY & PERFORMANCE

Page 39: Symantec I3 - Inquire & Cluster

39 Symantec Corporation, All Rights Reserved 2005

VCS Case Study

Enterprise web application running on a 4-tier architecture

Symantec Cluster Server (VCS) is activated on two machines running Oracle

Each machine switches to the other upon failure

Page 40: Symantec I3 - Inquire & Cluster

40 Symantec Corporation, All Rights Reserved 2005

Page 41: Symantec I3 - Inquire & Cluster

41 Symantec Corporation, All Rights Reserved 2005

Page 42: Symantec I3 - Inquire & Cluster

42 Symantec Corporation, All Rights Reserved 2005

Page 43: Symantec I3 - Inquire & Cluster

43 Symantec Corporation, All Rights Reserved 2005

Status

Interim ConclusionsInterim Conclusions

Upon Oracle tier failure, VCS activated failover

Next StepsNext Steps

Follow the Alerts advice and use the VRTS Cluster Manager to investigate further

Page 44: Symantec I3 - Inquire & Cluster

44 Symantec Corporation, All Rights Reserved 2005

AVAILABILITY & PERFORMANCE

Page 45: Symantec I3 - Inquire & Cluster

45 Symantec Corporation, All Rights Reserved 2005

Page 46: Symantec I3 - Inquire & Cluster

46 Symantec Corporation, All Rights Reserved 2005

Page 47: Symantec I3 - Inquire & Cluster

47 Symantec Corporation, All Rights Reserved 2005

Page 48: Symantec I3 - Inquire & Cluster

48 Symantec Corporation, All Rights Reserved 2005

Status

ConclusionsConclusions

Oracle ran on sys1

DiskGroup-oradg failed on sys1, hence VCS switched Oracle to sys2

Despite a major problem, the application remained available

But, to ensure availability over the long-term, the problematic resource must be repaired

Page 49: Symantec I3 - Inquire & Cluster

49 Symantec Corporation, All Rights Reserved 2005

Page 50: Symantec I3 - Inquire & Cluster

50 Symantec Corporation, All Rights Reserved 2005

Proactive & Preventive VCS Actions

Implement a systematic review of resource consumption overtime to forecast utilization

Identify up coming over-consumption

Establish VCS failover policies and assign system resources, accordingly

Page 51: Symantec I3 - Inquire & Cluster

51 Symantec Corporation, All Rights Reserved 2005

Page 52: Symantec I3 - Inquire & Cluster

52 Symantec Corporation, All Rights Reserved 2005

Page 53: Symantec I3 - Inquire & Cluster

53 Symantec Corporation, All Rights Reserved 2005

Page 54: Symantec I3 - Inquire & Cluster

54 Symantec Corporation, All Rights Reserved 2005

Proactive & Preventive VCS Actions

Summary Summary

Discovered peak of CPU consumption on server SrvProdD at the end of each month

In the future, this increase will exceed the overall CPU capacities

Based on the findings, the user can now establish an appropriate VCS failover system

Page 55: Symantec I3 - Inquire & Cluster

55 Symantec Corporation, All Rights Reserved 2005

AVAILABILITY & PERFORMANCE

Page 56: Symantec I3 - Inquire & Cluster

56 Symantec Corporation, All Rights Reserved 2005

&ANSWERS

QUESTIONS