Network Performance Monitoring

8/13/2019 Network Performance Monitoring

1/37


2/37

Topic

Level

Presenter(s):

ConfidentialProprietary InformationFor use by authorized NextGen clients and employees only. Not for public distribution. Copyright 2013

Technical

Client

Network Performance

Monitoring


3/37


Things our lawyers make us say...

Neither NextGen Healthcare nor any presenter at the Users

Group Meeting is engaged in rendering legal or other

professional advice and this presentation is not a substitute

for the advice of your attorney, accountant and/or other

professional advisor.


4/37


5/37


5

BJC HealthCare Hospital Locations

Broad Missouri/Illinois Geography (only 13% from St Louis City)

13 Hospital Facilities

BJC Medical Group

Largest employer in

state of Missouri

Long-time

relationship with

Washington

University School of

Medicine


6/37


National Leader Local Brand Identity


7/37ConfidentialProprietary InformationFor use by authorized NextGen clients and employees only. Not for public distribution. Copyright 2013

400+ Providers

100+ Locations

1200+ Concurrent Users

Growth Strategy

Tier 1 Physicians employed by BJC using NextGen

Tier 2 Private Physicians not employed by BJC and using

NextGen

Tier 3

Private Physicians not employed by BJC and notusing NextGen. Leverage Community Health Connection

(CHC)



How issues where handle in the past

CustomerCalls

Mobilize ISTeams

CaptureData

AccessData

DetermineRoot

Cause

ImplementFix

Challenges

Communication between teams

Ability to capture data to accessand implement fix quickly

How to reduce customer impact



Performance Monitoring Project

CustomerCalls

Or

ALERTS

Mobilize ISTeams

CaptureData

AccessData

DetermineRoot

Cause

ImplementFix

Performance Monitoring Objectives

Proactive monitoring

Identify critical data

Application

Network

Hardware Resolve problems before customer

impact

Keys to Success

Relationship with customers Relationship with vendors

Relationship with internal teams



Things our lawyers make us say...

Neither NextGen Healthcare nor any presenter at the Users

Group Meeting is engaged in rendering legal or other

professional advice and this presentation is not a substitute

for the advice of your attorney, accountant and/or other

professional advisor.



Sample Encounters

Track encounter performance at different times of the

day

Identify system hangs, slowness before input from

endusers

Compare performance from different locations in the

environment

Can be as simple or as complicated as you want



Sample Encounter



Database Monitoring

SQL Counters

SQL DMVs/Tables

SQL Notifications

SCOM Data



Page Life Expectancy

Page Life- The amount of time a page stays in memory

before being replaced

Constant low values indicate memory pressure

Track values continuously to compare over time






0

200

400

600

800

1000

1200

1400

1600

10:00

10:05

10:10

10:15

10:20

10:25

10:30

10:35

10:40

10:45

10:50

10:55

Microsoft Recomended Value

Microsoft Warning Value

BJC Value






0

5000

10000

15000

20000

25000

30000

35000

Microsoft Recomended Value

Microsoft Warning Value

BJC Value



Cache Hit Ratio

Buffer Cache Hit Ratio-Percent of pages found in cache

without having to read from disk

Recommended Value of >99%


18/37



Database Backup Completion

Real Life Example

0:00

2:24

4:48

7:12

9:36

12:00

14:24

We

d

Thur

Mo

n

Tue

s

We

d

Thur

F

ri

Mo

n

Tue

s

We

d

Thur

F

ri

Mo

n

Tue

s

We

d

Thur

F

ri

Mo

n

Tue

s

We

d

Thur

F

ri



Indexes

Missing Indexes

Impacts performance as SQL needs to read more

data than necessary to return results to user

Covering Indexes



Indexes

Unused Indexes

Unnecessary writes to the database


22/37


Expensive Queries

Find procedures/queries that can be tuned to

increase performance or scheduled to run during

non-production hours

Find recent reports run that had large datareads


23/37


Daily/Weekly Maintenance

NextGen Weekly maintenance updates defragments

indexes on a weekly basis

There are some indexes that fragment too quickly for

once a week rebuilds

Consider running weekly maintenance daily or daily

defrag on a subset of tables


24/37


Database Blocking

SQL Agent Job that sends email to support staff to alert

database blocking

Also writes blocking history (head blocker, blocked

queries) to table for review

Support response changes based on EHR, EPM,

support application blocking


25/37


26/37


Database Storage Monitoring

0

50000

100000

150000

200000

250000

6/30/2013

7/1/2013

7/2/2013

7/3/2013

7/4/2013

7/5/2013

7/6/2013

7/7/2013

7/8/2013

7/9/2013

7/10/2013

7/11/2013

7/12/2013

7/13/2013

7/14/2013

7/15/2013

7/16/2013

7/17/2013

7/18/2013

7/19/2013

7/20/2013

7/21/2013

7/22/2013

7/23/2013

7/24/2013

7/25/2013

7/26/2013

7/27/2013

7/28/2013

7/29/2013

Core

Index

LOB


27/37


Image File Storage

0

20000

40000

60000

80000

100000

120000

6/30/2013

7/1/2013

7/2/2013

7/3/2013

7/4/2013

7/5/2013

7/6/2013

7/7/2013

7/8/2013

7/9/2013

7/10/2013

7/11/2013

7/12/2013

7/13/2013

7/14/2013

7/15/2013

7/16/2013

7/17/2013

7/18/2013

7/19/2013

7/20/2013

7/21/2013

7/22/2013

7/23/2013

7/24/2013

7/25/2013

7/26/2013

7/27/2013

7/28/2013

7/29/2013

NextgenRoot

ICS Images


28/37


Server Storage


29/37


Application Server Monitoring

0

10

20

30

40

50

60

70

80

0

500

1000

1500

2000

2500

3000

3500

Citrix Server Utilization

Min Free Memory

Mem Warning

Max CPU

CPU Warning


30/37



0

10

20

30

40

50

60

70

80

90

100

41199.3

5328

41199.3

6022

41199.3

6717

41199.3

7411

41199.3

8105

41199.3

88

41199.3

9494

41199.4

0189

41199.4

0883

41199.4

1578

41199.4

2272

41199.4

2984

41199.4

3696

41199.4

439

41199.4

5085

41199.4

5779

41199.4

6474

41199.4

7168

41199.4

7862

41199.4

8557

41199.4

9251

41199.4

9946

41199.5

0658

41199.5

1352

41199.5

2046

41199.5

2741

41199.5

3435

41199.5

413

41199.5

4824

41199.5

5519

41199.5

6213

41199.5

6908

41199.5

7602

41199.5

8297

41199.5

9008

41199.5

9703

41199.6

0397

41199.6

1092

41199.6

1786

41199.6

2481

41199.6

3175

41199.6

3869

41199.6

4564

41199.6

5258

41199.6

5953

41199.6

6647

41199.6

7376

41199.6

8088

41199.6

8783

41199.6

9477

41199.7

0172

41199.7

0866

BJCNXXAP01 CPU Usage 10/17/2012


31/37



Users on each server

Load caused by new applications

Environment Changes that will change the amount of

users supported


32/37


Interface Server Monitoring

Intrf_Queue Table

Record Cleanup

Temp_Rec_Trfr


33/37


Network Monitoring

Alert Support on Node Down (Wan/Primary Switch)

Track utilization on WAN links (inbound/outbound)

Track network latency


34/37


BBP

Track completion times on time sensitive jobs

Disable Job steps that require data when data is not

present

Track completion rates

Notify users before job completes


35/37


Tools

SQL Server Management Studio

Perfmon

SCOM

Orion Report Server

Citrix Management Console

In-house development


36/37


Monthly Review with LeadershipSummary

Action Plan

We will be adding indexes suggested by the SQL Server to improve performance of database reads after removing the indexes that were not used previously. In addition we discovered that the patient images on the templates add a significant amount of time to the template loadtime when using the fat client environment. The EHR team is working on removing the image where they can. after initially believing that only hiding the image was necessary. We have migrated the disk space monitoring from SCOM to the Orion data and resumed the display of

all of the storage drives to the SQL and File servers. This was necessary due to a change made by Microsoft to the method that data is captured into SCOM.

Object Notes Impact

Workstation MonitoringWorkstation monitoring time was up slightly to 143s this month. We continue to have issues with the process completing on a consistent basis. The change in the

workflow for the new KBM caused an increase in the amount of work inside the templates which is giving us more points of f ailure within the script. We may need to revisitthe script to simplify the steps that it takes.

UsersWe saw no change in the total user capacity in the environment which was expected as there were no major changes. The amount of users continue to trend upwards at

a slight pace. The eight o'clock hour was up to 678 and the fi ve o'clock hour was up to 307 users.

Blocking

We are working with the DBA team to remove the blocking emails that are sent out when only the BBP machines are impacted, that will eliminate some of the extraneous

emails that are sent when users are not impacted. We had a major blocking issue on 11/6. We cleared the blocking by stopping the CHS service. In working with

NextGen, they have offered us updates on triggers, that will lesson the possibility of blocking caused by CHS i n the future. It is not known if CHS was the root cause ofthe block or another process, just that the interaction within CHS and the rest of the system caused blocking. We also found that information was being sent to CHS that

was not necessary and removed that. Deadlocks Deadlocks were down slightly to 25/day during business days and 27.21 overall.

Network Outages Steelville clinic experienced a three hour outage caused by a router failure. The router was replaced by the network team. The overall down time was up due to overnightmaintenance that took multiple sites down.

Jobs The Report Server restore job failed multiple times at the end of the month due to human error. Programming has been put in place to prevent these types of failures inthe future. All other job success rates were consistent with the previous month.

Citrix

The results from the sample of the five servers is complete. We did see periods of 100% usage on one of the servers caused by the DR. Watson program that generatesthe dump files when an application crashes. These periods were up to 3-5 minutes and would have impacted the users on that server. The servers with the higher

number in the queue seemed to be utilized less than the servers with the lower numbers but it is unclear if this is random or actual behavior. We plan to request a follow-up sample of five servers. Overall usage of the Citrix servers was up slightly in CPU as well as Memory usage.


37/37

Any Questions?

Network Performance Monitoring

Documents

Transcript of Network Performance Monitoring