IT Monitor at SDG&E

Ann Moore

San Diego Gas & Electric

September 16, 2004

PI T&D Users Group Meeting-St. Louis

2

Agenda

• SDGE – Who we are and What we do

• Why monitoring

• Monitoring what

• Why “IT Monitor”

• What is “IT Monitor”

• Implementation and Accomplishment

• Future Development

3

Sempra Energy• Sempra Energy is a Fortune 500 energy services holding

company with over 12,000 employees

➢ Sempra Energy Utilities

✓ San Diego Gas & Electric (SDG&E)✓ Southern California Gas Company (SoCalGas)

➢ Sempra Energy Global Enterprises

✓ Sempra Energy International

✓ Sempra Energy LNG Corp.

✓ Sempra Energy Solutions

✓ Sempra Energy Resources

✓ Sempra Energy Trading

✓ Sempra Fiber Links

4

SDGE & Electric T&D

• 1.3 million customers

• 3 million population

• Service territory includes San Diego County and

Southern Orange County

• 3,960 MW system peak load (8/31/98)

• 130 Transmission RTUs (69kV, 138kV, 230kV,

and 500kV) – GE XA21 EMS

• 900 Distribution RTUs (12kV)

– ACS Prism DMS

5

NERC Cyber Security Standard

• NERC Urgent Action Standard 1200 presents

standards to “monitor” and protect critical cyber

assets

• At Sempra, we take compliance seriously

– SDGE Self-Certification – 3/1/04

– “Substantial Compliance” – 3/1/04

– “Full Compliance” – 3/1/05

6

EMS Infrastructure

EMS

DMZOffice

Network

Primary

Control Center

Backup

Control Center

WAN

C

A

I

S

O

WAN

C

O

R

P

CAISO

DMZ

EMS

SYSTEM

WAN

FIREWALL

FIREWALL

FIREWALL

EMS

SYSTEM

CAISO

DMZ

FIREWALL

EMS

DMZ

Office

Network

7

Critical Cyber Assets• EMS nodes: 40+ UNIX boxes

–Application Hosts: IBM AIX

–Oracles: IBM AIX

–Front End Processors: IBM AIX

–Dispatcher Workstations: SUN Solaris

• Windows Servers: 10+ servers

–PI Servers

–PI OPC/Interfaces

–SQL-Servers

–Web Servers

• Network Devices: switches, routers, and firewalls

8

Monitoring All• A proactive and preventive way to monitor EMS

infrastructure resource health to ensure the

system reliability and performance

• Monitoring all EMS infrastructure for disk, file,

paging, cpu, swap and memory usage, …etc.

• Monitoring EMS processes and applications

• Establish performance baseline standards

• Avoid finger pointing

• Root cause analysis and problem solving

• Automatic notification via email and cell phone

9

Why IT Monitor?

PI Server

EMS

DMS

Non-Scada

DNP

Metering

Cyber

Assets

Power SystemControl System Infrastructure

✓150K tags

✓ProcessBook

✓DataLink

✓Hardware/Architecture

✓Multiple interfaces/multiple devices

10

Multiple Devices

11

PING

• Monitoring the network connectivity

12

TCP Response/NetFlow/PerfMon

• TCP Response

➢ Monitoring network service response times,

like telnet, ftp, …etc.

• NetFlow

➢ Monitoring network traffic, to and from the

device

• PerfMon

➢ Monitoring Windows servers performance

counter and statistics

13

PI Server Status

14

SQL & Web Server Status

15

SNMP• Simple Network Management Protocol

• Get statistics from any SNMP enabled devices

• SNMP query agent configuration in IBM AIX

and SUN Solaris - very challenging

• Different MIB (Management Information Base)

variables (containing OID’s-Object Identifier)

describing different aspects of computer/device

operations

• Matching snmpd.conf, Getif info and OID’s in PI

tag “exdesc”

16

EMS Node Status Overview

17

Dispatcher Workstation CPU

18

IT Monitor Trending

19

Application Host Index

20

Application Host 1

21

• Monitoring Oracle database – Using SNMP

• Monitoring ipcs

• Implementing SNMP Traps

• SNMP agent in facility equipments

• Secured SNMP traffic (encrypting SNMP?)

• Utilizing Syslog interface

• Integrating with Cisco Works, IDS and HP

Openview, …etc.

THANKS!!

Future Development