September 2014 Lesson Learned Summary

2

September 2014 LLs

Three NERC lessons learned (LL) were published in September 2014• LL20140901 Redundant Network Interface Cards on

Energy Management System (EMS)• LL20140902 Loss of EMS/Dispatch Workstation

Functionality due to NTP Time Synchronization Device Misconfiguration

• LL20140903 Relaying and Protection Systems

3

Redundant Network Interface Cards on EMS Systems

• Temporary network interruption caused the communication link between primary and backup EMS systems to fail

• Traced to a faulty network interface card• Both the primary and backup systems attempted to become the primary,

having assumed that the other system went offline since it was unreachable• Redundant network interface cards on the primary and backup EMS

systems may have prevented this incident• Equipment failure would likely affect only one network interface card at a

given time• NERC Energy Management System Working Group (EMSWG)

recommends periodic validation of network, server and control room console communications redundancy and teaming operations

4

Loss of EMS Functionality due to NTP Time Synch Device Misconfiguration

• EMS staff separated the redundant time source and three redundant servers from the Production EMS for testing purposes by disconnecting local area network (LAN) cables

• Worked in this configuration with no issue until EMS staff reconnected the redundant server without connecting time source.

• Immediate unexpected failover to newly connected servers and loss of ability to login to EMS workstations after couple of hours

5

• Reconfigured the correct NTP source and resynchronized the EMS/Dispatch workstations

• Document the procedure for EMS server disconnect and reconnect to EMS network

• Connecting and disconnecting EMS from their networks and time sources can lead to erratic behavior and possible disruptions of availability

• Redundancy, security, and consistency of time sources used in EMS, workstations, and associated authentication infrastructure should be reviewed to ensure resiliency during unusual events or network reconfigurations

Loss of EMS Functionality due to NTP Time Synch Device Misconfiguration

6

System Protection Review Prior to Disabling Protective Relays

• Construction to install new gas circuit breaker (GCB) on existing cap bank

• Internal fault of GCB during testing• Cleared by remote-end due to disabled relay protection• Manufacturer’s final report indicates failure due to transient

recovery voltage when the GCB disconnect switch was closed• Process was expanded to include systematic review of

protection systems during construction• Formed team to review commissioning practices

7

System Protection Review Prior to Disabling Protective Relays

• Review commissioning practices and procedures to ensure that adequate system protection prior to energization• If adequate system protection does not exist, the protected equipment

should be removed from service prior to disabling system protection. • Roles and responsibilities to verify adequate system

protection need to be defined among all the work groups involved in construction activities

8

Links to Lessons Learned

• Link to Lessons Learned

Directions to Lessons Learned:Go to www.NERC.com > “Program Areas & Departments” tab > “Reliability Risk Management” (left side menu) > “Event Analysis” (left side menu) > “Lessons Learned” (left side menu)

• NERC’s goal with publishing lessons learned is to provide industry with technical and understandable information that assists them with maintaining the reliability of the bulk power system. NERC requests that industry provide input on lessons learned by taking the short survey. A link is provided in the PDF version of each Lesson Learned.

9

Questions?