8th Security SummitPortland, Oregon

Substation Remote AccessEntergy Style

Chris Sistrunk, PE – RTU/SCADA SME

Sr. Engineer – T&D Technical Services

Entergy – Jackson, MS

9/26/2012

8th Security SummitPortland, Oregon

Entergy SCADA

• Entergy has about 1600 substation RTUs

• 1500+ are “smart” microprocessor based

• Approximately 60 are “dumb” card file RTUs

• Approximately 500 Relay Communication Processors connected to the “smart” RTUs

• Many IED types with several protocols

• About 98% of substations are serial only

8th Security SummitPortland, Oregon

• Most of Entergy’s RTU circuits are good ole’ Analog Leased Lines running at 1200 Baud

• ‘Ma-Bell’ won’t support forever

• OPGW, Digital µWave, Wireless, Leased T1

• Can support 4-wire to SCADAnet with same telecom equipment

• SCADAnet uses hardened routers & switches

1200 Baud to SCADAnet

8th Security SummitPortland, Oregon

“Engineering isn't about perfect solutions; it's about doing the best you can with limited resources.”-Randy Pausch, The Last Lecture

Engineering Truth

8th Security SummitPortland, Oregon

via Dezeen

8th Security SummitPortland, Oregon

A New RTU Standard

• Comparison of the major CommProcessors/RTU/Gateways in 2008

• Management Directive: 1 BOX!!!

• Must be able to work with existing and future substation designs

• I led Entergy-wide team that selected new RTU standard in 2010

• KEY piece to moving toward IP connectivity

8th Security SummitPortland, Oregon

A Hybrid Approach to SA

8th Security SummitPortland, Oregon

A Hybrid Approach to SA

• New RTU is a flexible and upgradeable solution that best met all of our requirements

• Migration path for existing RTU fleet

• HYBRID – more MPG for the Substation

– Old Stuff: 80% legacy relays, copper protocol

– New Stuff: SEL, IEDs, DNP, less copper

– New RTU can work with both

– Major building block for utilizing IP networks

8th Security SummitPortland, Oregon

A Hybrid Approach to SA

RTU

DNP

RTUNew RTU

SEL 351

SEL 351

SEL 351

Terminal Server

New RTU

Router

Switch

Serial to

SCADA

SCADAnet

PMUBKR/XFMR

Monitor100% Serial

DA

8th Security SummitPortland, Oregon

Challenges of a SCADA Engineer

8th Security SummitPortland, Oregon

• Started in fall of 2011

• Secure remote access to IEDs in the substation

• Old solution didn’t work – forced to roll trucks

• Must meet NERC/CIP standards

• Remember >>>

• Use new RTU with enterprise IED access solution in a new remote access solution

SUBCIP Project

Compliance != security

8th Security SummitPortland, Oregon

• Implement NERC/CIP v3 at new sites by June 30, 2012 for Phase 1 & Phase 2 by June 2013

• We know SCADAnet is the future, but routable protocols means locking cabinets or the entire control house, which is a challenge

• Using only serial communications for SCADA, engineering access, and file transfer will eliminate CIP002-R3 CCAs

SUBCIP Project

8th Security SummitPortland, Oregon

8th Security SummitPortland, Oregon

• REAAP – Resilient External Access & Authentication Project

• Provides a solution to address the need to provide additional security controls for external and remote access to Entergy’s Energy Delivery process control environment (e.g., EMS/SCADA) using additional security controls for authorized employees and contractors.

SUBCIP Project: REAAP

8th Security SummitPortland, Oregon

• REAAP uses Two-Factor Authentication

– Hardened passwords

– Smart cards

• In addition to TFA, remote access is via a virtual desktop environment

– Must use VPN if not on Corp network

– Virtual machines have security & virus scanning

– Short-term file storage for file transfers

SUBCIP Project: REAAP

8th Security SummitPortland, Oregon

SUBCIP Project: REAAP

ESP - Secure Environment

VPN

8th Security SummitPortland, Oregon

SUBCIP Project

RTU

SEL 351

SEL 351

SEL 351

Terminal Server

SEL 351

REEAP

Switch

SCADA

RS-232

4-Wire

RS-232

Zmodem

IED Access

Passwords

RecordsSub LAN

Corp/VPN

SUBSTATION

Why oh why

didn’t I

take the

blue pill?

8th Security SummitPortland, Oregon

8th Security SummitPortland, Oregon

• Remote serial connection from REAAP Enterprise system to RTU via channel banks

• 9600 Baud SCADA – 8X the bandwidth!

• Hardened Switch for SUB LAN & Future

• New RTU replaces old RTU and comm processors

• Relay techs only use serial in the Substation

– Zmodem (old school!) for file xfers to RTU

• Open USB & Eth ports are physically locked

SUBCIP Project: Substation (No CCAs)

8th Security SummitPortland, Oregon

…and it works…

8th Security SummitPortland, Oregon

• CIP v5 is on the horizon

• Some serial IEDs won’t be exempt anymore from becoming CCA/BES Cyber Assets

• Roll out SCADAnet to IEDs where serial isn’t sufficient or other requirements where IP is more beneficial

• Implement automatic IED password management & fault collection

SUBCIP Project: Phase 3

8th Security SummitPortland, Oregon

Final Thoughts

• SCADA Security isn’t easy

– Doing the best we can with what we have

• SCADA, Relay, & Security Labs

– Having a lab is so valuable for testing, troubleshooting, breaking & fixing stuff

– Yes I have a fuzzer and I’m not afraid to use it

• DNP3/IP Secure Authentication v5

– Please tell your vendors you want it

8th Security SummitPortland, Oregon

Follow @chrissistrunk

Chris Sistrunk, PEcsistru@entergy.com