Outage Management Systems: Hybrid OMS in hurricane Ike FEMA Considerations

Hybrid OMS in Hurricane Ike

Jeff Lane

Presentation Goals

• About TVEC• Dispatch center setup• Our typical storm• Hurricane IKE response structure• Questions

About Trinity Valley

• Located 30 miles south east of Dallas Texas• 61,000 meters• 6,900 miles of line• 151 employees• SEDC Billing system• DisSPatch OMS (using enhanced Crew Management)• ESRI/WindMilMap• Porche and Twenty First Century Communications IVR• Wireless Matrix AVL integrated with DisSPatch

Service Territory

Dispatch Center Layout

Dispatch Center

Typical “Major” Storm• 50 to 75 concurrent outages affecting 5000 to

7000 meters• 40 troubleshooting/repair personnel• 1 to 2 contract construction and/or ROW crews• 2 Dispatchers• 1 to 2 Supervisors• 1 to 2 Dispatch helpers• Restoration time 10 to 15 hours

Typical Dispatch Structure

Incident Commander

Athens District

Kaufman District

Dispatcher

Crew Coordinator

Dispatcher

Crew Coordinator

DispatchAssistant

20 – 30 Troubleshooting

Personnel

20 – 30 Troubleshooting

Personnel

District line

Hurricane IKE• Approximately 200 concurrent outages (total of 269)• 14,000 out at height of storm (total affected 20,141)• Damage:

– 19 broken poles– 14 damaged transformers– 93 lines down

• 66 troubleshooting/repair personnel (16 from other coops and contractors)

• 6 contract construction crews• 10 contract ROW crews• Had 20 Dispatchers on Saturday, dropped down to 17 on

Sunday• Restoration time 42 hours

Hurricane IKE Event structure

Incident Commander Jeff Lane

Operations Section Chief

Tony Watson

Planning Section Chief

Tim Craig

Logistics Section Chief

Greg Starek

Accounting Section Chief

Frank Skube

Kaufman/Tawakoni Branch

Sam Grimes

Safety Officer Danny Belcher

Public InformationOfficer

Jerry Boze

Support Section Chief

Jerry Woolston

Liason Officer

Customer Service Group

Shirley Murphree

IT Group Jerry Woolston

Mapping

Engineering

Staking

HR

Facilities

Dispatch BranchChase SnufferJerry Williams

Incident Command SystemEvent Name: Hurricane Ike

Event Date: 9/13/2008

East BranchJohn Jackson

West/North BranchChris Durden

South BranchJimmy Splawn

Cedar Creek Branch

Rodney Wesley

Canton Branch Gene

Operations SectionOperations

ChiefTony

Canton/CCRKBranch

Sam

CombineBranchRodney

West/NorthBranch

Chris(14)

EastBranch

Jimmy(13)

SouthBranch

John(14)

Walton DivisionLineman

Cedar DivisionLineman

Bartons ChapelDivisionLineman

WollowSpringsDivisionLineman

AntiochDivisionLineman

New YorkDivisionLineman

CantonDivisionLineman

CoffeeDivisionLineman

Massey LakeDivisionLineman

Carroll SpringsDivisionLineman

MontablaDivisionLineman

TuckerDivisionLineman

MabankDivision

Paul

JacksonDivisionJamie

Glenn PinesDivision

Bo

ScurryDivisionLathem

KempDivision

Virgil

Tool/Seven PointsDivisionEddie

CombineDivisionKenneth

TaltyDivision

Raymond

MarshallDivisionLance

PattonDivision

Jay

RandDivisionFloyd

EnterpriseDivisionShayde

Cobb SwitchDivision

Rudy

Duck CoveDivisionLarry B.

ROW Division(5 Crews)Pete

ROW Division(5 Crews)

Paul

Contractor Division J&L

Jimmy Kirk


Jason

KaufmanDispatch

Jerry

Tawkoni BranchGene

ROW DivisionKelly Massey


Jimmy Ray

AthensDispatchChase

Contractor DivisionJ&L

George

ROW DivisionRamirez Tree

Service


Chad


Service

Contractor DivisionS&H

ROW Division Ramirez Tree

Service

OfficeField

District Dispatchers

• Dispatchers managed the OMS.– Evaluated and corrected predictions when possible– Assigned and printed outage tickets to appropriate branches

• Outage tickets included Name, Map #, Meter #, Address, Phone #, Outage #, and included all calls received for the outage.

– When outages were restored, the tickets were brought back to the Dispatcher to be restored in the OMS.

Operations ChiefTony

KaufmanDispatch

Jerry

AthensDispatchChase

Branches

• Branch Directors– Received outage tickets from Dispatcher and dispatched to appropriate Division leaders– Worked with Dispatcher to modify any incorrect outage predictions– Coordinated Construction/ROW contractor movement between Division Leaders within

their branch– Issued line clearances to Division Leaders– Record restoration date/time, cause, and equipment codes on outage tickets once

restored– Return outage tickets to Dispatcher for closing in the OMS


Canton/CCRKBranch

Sam

CombineBranchRodney

West/NorthBranch

Chris(14)

EastBranch

Jimmy(13)

SouthBranch

John(14)

KaufmanDispatch

Jerry

Tawkoni BranchGene

AthensDispatchChase

Divisions

• Division Leaders– Received outage information from Branch Directors via cell phone and radio– Coordinated personnel with their divisions to restore power– Communicated resource needs back to Branch Director (i.e., need more or less

crews)– Communicated restoration progress back to Branch Director


Canton/CCRKBranchSam

CombineBranchRodney

West/NorthBranch

Chris(14)

EastBranch

Jimmy(13)

SouthBranch

John(14)

Walton DivisionLineman

Cedar DivisionLineman

Bartons ChapelDivisionLineman

WollowSpringsDivisionLineman

AntiochDivisionLineman

New YorkDivisionLineman

CantonDivisionLineman

CoffeeDivisionLineman

Massey LakeDivisionLineman

Carroll SpringsDivisionLineman

MontablaDivisionLineman

TuckerDivisionLineman

MabankDivision

Paul

JacksonDivisionJamie

Glenn PinesDivision

Bo

ScurryDivisionLathem

KempDivision

Virgil

Tool/Seven PointsDivisionEddie

CombineDivisionKenneth

TaltyDivision

Raymond

MarshallDivisionLance

PattonDivision

Jay

RandDivisionFloyd

EnterpriseDivisionShayde

Cobb SwitchDivision

Rudy

Duck CoveDivisionLarry B.

ROW Division(5 Crews)Pete

ROW Division(5 Crews)

Paul


Jimmy Kirk


Jason

KaufmanDispatch

Jerry

Tawkoni BranchGene

ROW DivisionKelly Massey


Jimmy Ray

AthensDispatchChase


George


Service


Chad


Service

Contractor DivisionS&H

ROW Division Ramirez Tree

Service

How did it actually work???• After the event was over, we conducted post-event reviews with everyone involved,

from the CEO to field personnel. All of the feedback we received was positive.• Typical bottlenecks at the Dispatch level were avoided by dividing the workload into

manageable levels.• Even the linemen who were acting as “Division Leaders” and performing dispatch

functions for the personnel under them felt like we should continue to use this structure in future events.

• The normal Operations department supervisors were asking to begin using this structure more often (i.e., events that may not be as large as a hurricane but that are larger than a typical “major” thunderstorm).

• The normal Dispatchers loved it because they had a manageable workload.• Everyone felt like this provided a safer working environment because they weren’t

being overworked and that reduced the possibility for mistakes.• The general consensus was that restoration was achieved sooner than it would have

been if we had used the normal Dispatch structure.

Did we have issues???• Sure we did.• Some of the division leaders were holding onto restoration information until

they had several outages restored. Their reasoning was they wanted to report them all in one call back to the Branch Director instead of multiple calls. This caused issues in the office with accurate reporting to the PUC, Critical Account restoration status, and planning of the next day’s work.

• Outage re-predictions were a problem. The Dispatcher would give the Branch Director several outages, all on separate outage tickets, then a re-prediction would push those outages upline. We had to manage that paper with effective communications between the Dispatchers and the Branch Directors.

• Reallocating workload dynamically based on restoration status was challenging but worked with effective communication between Branch Directors and the Operations Section Chief.

Will this structure work to manage “The Big One”?

• With minor modifications, I believe it will. That’s just one opinion, though, and I haven’t been through “The Big One.” What is your opinion?

Outage Management Systems: Hybrid OMS in hurricane Ike FEMA Considerations

Technology

Transcript of Outage Management Systems: Hybrid OMS in hurricane Ike FEMA Considerations