Extending Digital Networking to the Field ©2004 InfoNetrix, LLC and Cyber Security Consulting All...

Extending Digital Networking to the FieldExtending Digital Networking to the Field

©2004 InfoNetrix, LLC and ©2004 InfoNetrix, LLC and Cyber Security ConsultingCyber Security Consulting

All rights reservedAll rights reservedUnauthorized reproduction or distribution ofUnauthorized reproduction or distribution of

this document is expressly prohibitedthis document is expressly prohibited

From ‘Real-time’ to the ‘Real-world’ Over the NET From ‘Real-time’ to the ‘Real-world’ Over the NET

P R E S E N T E D B YP R E S E N T E D B Y

William T. Shaw - PhD, CISSPWilliam T. Shaw - PhD, [email protected]@direcway.com

www.cybersecconsulting.comwww.cybersecconsulting.com

The challenge of incorporating remote automation facilities The challenge of incorporating remote automation facilities and systems into the Corporate Enterprise digital networkand systems into the Corporate Enterprise digital network

Net Communications in Utility

Automation/IT (Wednesday, October 27,

2004)





In many industries corporations have geographically distributed In many industries corporations have geographically distributed facilities separate from those of the main corporate offices. facilities separate from those of the main corporate offices. Corporate IT and Telecommunications groups have generally Corporate IT and Telecommunications groups have generally done a good job of creating digital/IP networks that seamlessly done a good job of creating digital/IP networks that seamlessly link all of these facilities and personnel.link all of these facilities and personnel.

Presentation Focus

But, in certain industries there have always been widely But, in certain industries there have always been widely scattered, moderate to small-sized facilities that may be scattered, moderate to small-sized facilities that may be unattended, minimally staffed or only occasionally staffed. These unattended, minimally staffed or only occasionally staffed. These facilities have rarely been tied into the corporate network.facilities have rarely been tied into the corporate network.

These are field sites where process-related operations are These are field sites where process-related operations are taking place and where local monitoring & control systems taking place and where local monitoring & control systems manage important activities. These sites, and their respective manage important activities. These sites, and their respective automation systems, are now being considered as candidates for automation systems, are now being considered as candidates for integration into the corporate digital/IP network.integration into the corporate digital/IP network. This presentation addresses some of the issues, technologies, This presentation addresses some of the issues, technologies, standards and security requirements relevant to extending IP standards and security requirements relevant to extending IP networking to the field.networking to the field.





Typical Corporate Networking ArchitecturesTypical Corporate Networking Architectures

Typical (Remote) Automation System technologyTypical (Remote) Automation System technology

Local Area and Wide Area Networking technology Local Area and Wide Area Networking technology

Networking Protocol StandardsNetworking Protocol Standards

IP Security and DHS Security RequirementsIP Security and DHS Security Requirements

Functional advantages of IP-based networkingFunctional advantages of IP-based networking

Performance issues with WAN/Internet technologyPerformance issues with WAN/Internet technology

Real world examples of successful implementationsReal world examples of successful implementations

Topics to be Discussed





Typical Corporate Networking Architectures

Through the late 1980s and 1990s most large corporations Through the late 1980s and 1990s most large corporations adopted “office automation” technologies and built corporate adopted “office automation” technologies and built corporate Information technology (IT) staffs to manage and support their Information technology (IT) staffs to manage and support their computing assets. computing assets.

Most large corporations, with multiple operating locations and Most large corporations, with multiple operating locations and facilities, created wide-area networks to link their facilities, facilities, created wide-area networks to link their facilities, systems and personnel together for e-mail, automated systems and personnel together for e-mail, automated information transfers, centralized “data warehousing” and other information transfers, centralized “data warehousing” and other such applications of computer technology.such applications of computer technology.

With the “privatization” of the Internet, and the proliferation of With the “privatization” of the Internet, and the proliferation of hardware and software products based on the “TCP/IP” network hardware and software products based on the “TCP/IP” network architecture, most corporations migrated to the Internet as their architecture, most corporations migrated to the Internet as their chosen wide-area network technology, and away from “private” chosen wide-area network technology, and away from “private” WAN approaches.WAN approaches.





Corporate IT SystemsCorporate IT Systems

•ManagementManagement•AccountingAccounting•FinanceFinance•HR/LegalHR/Legal

Plant/Site IT SystemsPlant/Site IT Systems

PlantPlantAutomation/ControlAutomation/Control

Systems Systems

•EngineeringEngineering•OperationsOperations•MaintenanceMaintenance•MRP/ERPMRP/ERP•Shipping Shipping

•ProductionProduction•ManufacturingManufacturing•HVACHVAC•QA/QCQA/QC•Inventory Inventory

Remote SiteRemote SiteAutomationAutomation Systems Systems

•Pump stationPump station•SubstationSubstation•Storage tankStorage tank•Lift stationLift station•Metering Metering •CustodyCustody


This final “remote-site layer” is NOT common to all industriesThis final “remote-site layer” is NOT common to all industries





DepartmentalDepartmentalserversservers

End-UserEnd-UserPCsPCs

EngineeringEngineering

CorporateCorporateServersServers

CorporateCorporateIntranetIntranet

Planning/Design/ConstructionPlanning/Design/Construction

Operations/MaintenanceOperations/Maintenance

Corporate UsersCorporate Users

IT DepartmentIT Department

INTERNETINTERNET

Remote UsersRemote UsersCustomersCustomersPartnersPartnersSuppliersSuppliers

CellularCellularTelCoTelCo

Mobile UsersMobile Users


Additional corporate facilitiesAdditional corporate facilities

Web siteWeb site E-mailE-mail B2B applicationsB2B applications

Local Area NetworkLocal Area Network

““Seamless” IP Networking across the EnterpriseSeamless” IP Networking across the Enterprise(in the typical view of the IT group)(in the typical view of the IT group)

CIT

PIT

PAS

RAS





““Seamless”Seamless”IP Networking IP Networking

Originally these real-time systems were rarely-if-ever connected to the corporate WAN, both for Originally these real-time systems were rarely-if-ever connected to the corporate WAN, both for securitysecurity purposes (keeping IT away) and because typical corporate-level purposes (keeping IT away) and because typical corporate-level applicationsapplications did not did not generally require continuous access to the data contained within these dedicated control systems.generally require continuous access to the data contained within these dedicated control systems.

Special consoles Special consoles with special SW with special SW dedicated to dedicated to information displayinformation display


SupervisorySupervisoryControl SystemsControl Systems

DistributedDistributedControl SystemsControl Systems

Integrating the Plant Automation Systems

INTERNETINTERNETCorporateCorporateIntranet (IP)Intranet (IP)

““Wired”Wired”TelCoTelCo

Private/LeasedPrivate/LeasedTelecomTelecom

RTU units typically connected via non-IP TelCo or private/leased Telecom services


Systems Systems





““Seamless”Seamless”IP Networking IP Networking

Over the last decade many corporations have extended their IP networks to incorporate plant Over the last decade many corporations have extended their IP networks to incorporate plant control and automation systems. This connectivity has been control and automation systems. This connectivity has been enabledenabled by system vendors adopting by system vendors adopting IP-based LAN/WAN standards and IP-based LAN/WAN standards and drivendriven by the deployment of centralized applications such as by the deployment of centralized applications such as production optimization, asset utilization, resource management and reliability-centered production optimization, asset utilization, resource management and reliability-centered maintenance.maintenance.


SupervisorySupervisoryControl SystemsControl Systems

DistributedDistributedControl SystemsControl Systems

INTERNETINTERNETCorporateCorporateIntranet (IP)Intranet (IP)

““Wired”Wired”TelCoTelCo

Private/LeasedPrivate/LeasedTelecomTelecom

RTU units typically connected via non-IPTelCo or private/leased Telecom services

Although many plant control systems have been integrated, those at remote sites generally have not

RouterRouterFirewallFirewall

Of course, in light of the events of 9/11, andOf course, in light of the events of 9/11, andbased on real-world cyber assaults, this based on real-world cyber assaults, this connectivity is being reviewed and revised !connectivity is being reviewed and revised !

CIT

PIT

PAS

RAS


Systems Systems

Integrating the Plant Automation Systems





Industrial Application Industrial Application .. Typical Remote Site & Typical Remote Site & Facilities Facilities ..FacilityFacility AutomationAutomation CommunicationsCommunications

Electric PowerElectric PowerTransmission &Transmission &DistributionDistribution

Substations with Substations with transformers, transformers, LTCs, circuit LTCs, circuit breakers and breakers and switch-gear plus switch-gear plus various types of various types of IEDs IEDs

Remote Terminal Remote Terminal Unit (RTU) or Unit (RTU) or Substation data Substation data concentrator concentrator connected to IEDs connected to IEDs and assorted I/O and assorted I/O pointspoints

Dedicated, leased Dedicated, leased analog telephone analog telephone line or frame relay line or frame relay digital connection. digital connection. Possibly private Possibly private digital WAN using digital WAN using microwave or fiber microwave or fiber optic linksoptic links

Water/SewageWater/SewageTreatment &Treatment &TransportationTransportation

Treatment plants Treatment plants and booster/lift and booster/lift stations with stations with pumps, drives, pumps, drives, valves and valves and instrumentation instrumentation

PLC-based control PLC-based control system (with PC-system (with PC-based HMI) or PLC-based HMI) or PLC-based Remote based Remote Terminal Unit (RTU)Terminal Unit (RTU)

Dedicated, leased Dedicated, leased analog telephone analog telephone line or frame relay line or frame relay digital connection. digital connection. Possibly private Possibly private digital WANdigital WAN

Gas/Oil PipelineGas/Oil PipelineBooster StationBooster Stationor custody pointor custody point

Pump/compressor Pump/compressor station with large station with large “prime movers”, “prime movers”, pumps, valves, pumps, valves, ancillary processes ancillary processes and and instrumentation instrumentation

Private Private communications communications infrastructure, infrastructure, possibly microwave possibly microwave or fiber optic based. or fiber optic based. Possibly satellite Possibly satellite connected.connected.

DCS/PLC control DCS/PLC control system with local system with local HMI or large HMI or large Remote Terminal Remote Terminal Unit with some Unit with some control and control and regulatory functionsregulatory functions

“Remote” Automation System technology





DCS Architectural Evolution

DumbDumbinstrumeninstrumen

tsts

ProprietarProprietaryy

LANLAN

ProprietaryProprietaryWorkstationsWorkstations

EthernetEthernet““gatewaygateway

””and and

serversservers

ProprietarProprietaryy

I/O busI/O bus

Legacy DistributedLegacy DistributedControl SystemControl System

(DCS)(DCS)

Proprietary or Proprietary or legacy operating legacy operating

systemsystem

Pre-defined data only (custom application)Pre-defined data only (custom application) “ “Local” use of configuration toolsLocal” use of configuration tools “ “Bolt on” web server, if anyBolt on” web server, if any “ “Bolt on” OPC server, if anyBolt on” OPC server, if any No standard TCP/IP applicationsNo standard TCP/IP applications

Remote SQL-based data accessRemote SQL-based data access Remote use of configuration toolsRemote use of configuration tools Integral web server (HMI driven by it)Integral web server (HMI driven by it) OPC client/server links componentsOPC client/server links components All standard TCP/IP applicationsAll standard TCP/IP applications Able to support IPAble to support IPSECSEC technologies technologies

Process controllersProcess controllers(redundant)(redundant)

LocalLocalloggerlogger

Modern DistributedModern DistributedControl SystemControl System

(DCS)(DCS)

RedundantRedundantEthernet switchEthernet switch

StandardStandardInstrumenInstrumen

ttbusbus

PC-based HMIPC-based HMIWindows OS Windows OS

COTS operating systemCOTS operating systemCOTS I/O hardwareCOTS I/O hardwareCOTS computer HWCOTS computer HW

10/100baseT10/100baseT

FieldBus, FieldBus, ProfiBus, ProfiBus, DeviceNEDeviceNE

TT

ProprietaryProprietaryController Controller HW/SWHW/SW






PLC Architectural Evolution

PLCs with PLCs with distributed I/Odistributed I/O

Modern PLC-basedModern PLC-basedControl SystemControl System

Ethernet switchEthernet switch

ENET readyENET readyanalyzeranalyzer

““Industrial”Industrial”EthernetEthernet

PC-based HMIPC-based HMI

SQL RdBs, web serversSQL RdBs, web servers and OPC serversand OPC servers

StandardStandardInstrumenInstrumen

ttbusbus

LAN to SerialLAN to Serialadapteradapter

PC-based “gateway”PC-based “gateway”and data serverand data server

EthernetEthernetLANLAN

Legacy PLC-basedLegacy PLC-basedControl SystemControl System

ProprietaryProprietaryData highwayData highway

PLCs with PLCs with centralized I/Ocentralized I/O

ProprietaryProprietaryHMIHMI

DumbDumbinstrumentinstrument

ss

Pre-defined data only (custom application)Pre-defined data only (custom application) “ “Local” use of configuration toolsLocal” use of configuration tools “ “Bolt on” web server, if anyBolt on” web server, if any “ “Bolt on” OPC server, if anyBolt on” OPC server, if any No standard TCP/IP applicationsNo standard TCP/IP applications

File and database-query data accessFile and database-query data access Remote use of configuration toolsRemote use of configuration tools Support for web serverSupport for web server OPC client/server links componentsOPC client/server links components All standard TCP/IP applicationsAll standard TCP/IP applications Able to support IPAble to support IPSECSEC technologies technologies






IP WANIP WAN

low bandwidthlow bandwidthconnectionconnection

PLCs with PLCs with distributed I/Odistributed I/O

Local PLC-based Control SystemLocal PLC-based Control System


Ethernet switchEthernet switch(redundant/F.O.)(redundant/F.O.)

Local HMILocal HMI(optional)(optional)

ENET readyENET readyanalyzeranalyzer

IP WANIP WAN

ENET readyENET readyinstrumentsinstruments

Local HMILocal HMI


Process controllersProcess controllersand I/Oand I/O

(redundant)(redundant)

Local Local loggerlogger

10/100baseT10/100baseTEthernetEthernet

Local Distributed ControlLocal Distributed ControlSystem (DCS)System (DCS)



NOTICE:NOTICE:

You STILL have to have applications at You STILL have to have applications at the Central site that WANT data from the Central site that WANT data from these subsystems, and a common IP these subsystems, and a common IP

protocol that will be used between these protocol that will be used between these subsystems and those applications, in subsystems and those applications, in order for the network connectivity to order for the network connectivity to

accomplish anything !accomplish anything !

CIT

PIT

PAS

RAS

Integrating “Remote” Systems





New and Legacy RTU equipmentNew and Legacy RTU equipment

High bandwidthHigh bandwidthconnectionconnection

IP WANIP WAN

““IP-enabled”IP-enabled”SCADA systemSCADA system

RouterRouterCSU/DSUCSU/DSU

Low bandwidthLow bandwidthconnectionconnection

EthernetEthernetswitchswitch

10baseT10baseT

SerialSerial

ProtocolProtocolconverterconverter

““IP-enabled”IP-enabled”RTURTU

LegacyLegacyRTURTU


CorporateCorporateWANWAN

Legacy IED/RTU equipmentLegacy IED/RTU equipment

High bandwidthHigh bandwidthconnectionconnection

IP WANIP WAN

““IP-enabled”IP-enabled”SCADA systemSCADA system

RouterRouterCSU/DSUCSU/DSU

10baseT10baseT

SerialSerial

DataDataConcentratorConcentrator

LegacyLegacyRTURTU

IP-readyIP-readyIEDIED


CIT

PIT

PAS

RAS

NOTICE:NOTICE:

For Supervisory Control (SCADA) For Supervisory Control (SCADA) applications a major consideration is applications a major consideration is

providing communications security all the providing communications security all the way “to the field”. The DHS, NIST, FERC way “to the field”. The DHS, NIST, FERC

and NERC are all “encouraging” the and NERC are all “encouraging” the implementation of security features. By implementation of security features. By going to IP there are a range of security going to IP there are a range of security

features (such as VPN “tunnels”) that can features (such as VPN “tunnels”) that can be implemented in the routers, even if be implemented in the routers, even if

the field-based equipment (RTUs) doesn’t the field-based equipment (RTUs) doesn’t support this capability !support this capability !

Integrating “Remote” Systems





Local Area and Wide Area Networking

StandardsLocal Area Networking (LAN) Standards

Token Ring (IBM)Token Ring (IBM)

Ethernet (“wired”)Ethernet (“wired”)

Wireless Ethernet (WiFi)Wireless Ethernet (WiFi)

Modbus-plusModbus-plus

FieldBusFieldBus

ProfibusProfibus

DeviceNetDeviceNet

ARCNetARCNet

FirewireFirewire

U.S.B.U.S.B.

F.D.D.I.F.D.D.I.

Really only used by IBM, for office networks of PCs to mainframes. Outdated.Really only used by IBM, for office networks of PCs to mainframes. Outdated.

Available in many formats/bandwidths. “Industrial” versions. De Facto winner. Available in many formats/bandwidths. “Industrial” versions. De Facto winner.

Just one more version of Ethernet. Short range, line-of-site. Just one more version of Ethernet. Short range, line-of-site.

For Modicon PLCs. Outdated and replaced by Ethernet in new applications. For Modicon PLCs. Outdated and replaced by Ethernet in new applications.

}Special-purpose LAN technologies. Not very scalable. Limited utility. Special-purpose LAN technologies. Not very scalable. Limited utility.

For process control instruments and PLCs. Now available in Ethernet/IP version.For process control instruments and PLCs. Now available in Ethernet/IP version.All come from proprietary beginnings with a specific vendor’s products. All come from proprietary beginnings with a specific vendor’s products. }

100Mbps, self-healing, long distance (ring), bridges to Ethernet. Somewhat outdated.100Mbps, self-healing, long distance (ring), bridges to Ethernet. Somewhat outdated.

And the “winner” in mostAnd the “winner” in mostinstances has been….instances has been….

ETHERNETETHERNET





ETHERNETNetworking

IEEE 802.11-a/b/gIEEE 802.11-a/b/gWireless ENETWireless ENETHub “hot spot”Hub “hot spot”

10base2 “Thinwire” Ethernet10base2 “Thinwire” Ethernet Bridge/RepeaterBridge/Repeater

Hub (10-2 to 10-T)Hub (10-2 to 10-T)

Ethernet Physical Architectures

Fiber OpticFiber Optichubs andhubs andswitchesswitches

F.O. Patch cordsF.O. Patch cords

““Stackable”Stackable”hubs andhubs andswitchesswitches

Cat-4/5/6 cableCat-4/5/6 cable

Multiple Multiple ““stars”stars”





The many “flavors”of Ethernet

ETHERNETVariations

Thinwire Ethernet was popular Thinwire Ethernet was popular because it was a point-to-point because it was a point-to-point multi-dropped design using “T” multi-dropped design using “T” connectors to tap where needed.connectors to tap where needed.

This supplanted thinwire ENET by This supplanted thinwire ENET by providing central hubs and “telco” providing central hubs and “telco” style plug-in connectors.style plug-in connectors.

This is the fiber-optic version of This is the fiber-optic version of thinwire ENET. It also uses thinwire ENET. It also uses modular connectors and “patch modular connectors and “patch cord” connections to central hubs. cord” connections to central hubs.





Local AreaNetworking Standards

Ethernet Physical Interface IEEE Designations

10BaseT - Twisted pair - CAT 5/6 cable (10BaseT - Twisted pair - CAT 5/6 cable (IEEE 802.3IEEE 802.3))

10BaseFl - Multi-mode fiber (10BaseFl - Multi-mode fiber (IEEE 802.3IEEE 802.3))

10Base2 - Thin wire coax (10Base2 - Thin wire coax (IEEE 802.3IEEE 802.3))

10Base5 - Thick wire coax (10Base5 - Thick wire coax (IEEE 802.3IEEE 802.3))

100BaseTx - Twisted pair CAT 5/6 cable (100BaseTx - Twisted pair CAT 5/6 cable (IEEE 802.3uIEEE 802.3u))

100BaseT4 - Twisted pair CAT 3 cable (100BaseT4 - Twisted pair CAT 3 cable (IEEE 802.3uIEEE 802.3u))

100BaseFx - Multi-mode fiber @ 1330nm(100BaseFx - Multi-mode fiber @ 1330nm(IEEE 802.3uIEEE 802.3u))

1000BaseF - Multi-mode fiber (1000BaseF - Multi-mode fiber (IEEE 802.3ae and abIEEE 802.3ae and ab))

10000BaseF – Single/Multi-mode fiber (10000BaseF – Single/Multi-mode fiber (IEEE 802.3zIEEE 802.3z))

WiFi – Wireless Ethernet (WiFi – Wireless Ethernet (IEEE 802.11a,b,gIEEE 802.11a,b,g))





ETHERNETNetworking Alternatives

Industrial Ethernet

The repeaters would keep messages circulating forever if a “ring” existed. Backup The repeaters would keep messages circulating forever if a “ring” existed. Backup link is not in operation as long as the bus isn’t damaged (only link “test” messages). link is not in operation as long as the bus isn’t damaged (only link “test” messages).

•Typically dual/redundant networksTypically dual/redundant networks• Higher temperature specsHigher temperature specs• May use a Hirschmann RingMay use a Hirschmann Ring• Often AC/DC poweredOften AC/DC powered• May support drop-out on faultMay support drop-out on fault• Protocol may do token passing Protocol may do token passing

The repeaters place The repeaters place messages on the local messages on the local ENET segment and pass ENET segment and pass them down/up the bus to them down/up the bus to the next repeater(s). the next repeater(s).

The final repeaters place The final repeaters place messages on the local messages on the local ENET segment drop the ENET segment drop the message off the bus. message off the bus.





Industrial Ethernet

FAULTFAULT

The various repeaters use The various repeaters use “out of band” messages to “out of band” messages to test topology and to signal test topology and to signal when link failures are when link failures are detected. detected.

ETHERNETNetworking Alternatives

But ETHERNET But ETHERNET does notdoes not provide a provide a complete communications facility. complete communications facility.

There must be a There must be a protocolprotocol employed employed by programs in each computer, that by programs in each computer, that

will be used to transmit messages and will be used to transmit messages and data across the LAN/WANdata across the LAN/WAN

But ETHERNET But ETHERNET does notdoes not provide a provide a complete communications facility. complete communications facility.

There must be a There must be a protocolprotocol employed employed by programs in each computer, that by programs in each computer, that

will be used to transmit messages and will be used to transmit messages and data across the LAN/WANdata across the LAN/WAN





All

pri

mar

ily

inte

nded

All

pri

mar

ily

inte

nded

for

ET

HE

RN

ET

LA

Nfo

r E

TH

ER

NE

T L

AN

• DNP3.0 – IP versionDNP3.0 – IP version• Modbus – IP versionModbus – IP version• Fieldbus – IP versionFieldbus – IP version• DeviceNet – IP versionDeviceNet – IP version• Profibus – IP versionProfibus – IP version• OPC data exchange standardOPC data exchange standard• UCA2.0UCA2.0• UCA1.0 – ICCP/TASE.2 (IEC-60870-6)UCA1.0 – ICCP/TASE.2 (IEC-60870-6)• IEC-60870-5-#IEC-60870-5-#• Internet ProtocolsInternet Protocols

ftp ftp rtp rtp udp udp

• Other/ProprietaryOther/Proprietary EthernetEthernet TCP/IP TCP/IP

(IP) Protocols for Real-Time Applications

LAN/WAN ProtocolStandards

All competing to be THEAll competing to be THE““Industrial Ethernet” standardIndustrial Ethernet” standard}

Connection of EMS-EMS, EMS-RTUConnection of EMS-EMS, EMS-RTU

European version of UCA2.0, gaining acceptance hereEuropean version of UCA2.0, gaining acceptance here

}All require higher-level applications that All require higher-level applications that utilize these protocols for transportutilize these protocols for transport

Apple, DEC, Novell and others developed their own Net architecturesApple, DEC, Novell and others developed their own Net architecturesVendors develop their own application layer protocolsVendors develop their own application layer protocols

Wor

k w

ell i

n a

WA

N o

r W

ork

wel

l in

a W

AN

or

LA

N e

nvir

onm

ent

LA

N e

nvir

onm

ent





A variety of physical transport mechanisms are available for establishing

Wide-Area “IP” network connectivity to multiple remote locations:• Frame-RelayFrame-Relay

• X.25 Packet SwitchingX.25 Packet Switching

• The INTERNETThe INTERNET

• Fiber Distributed Data Interface (FDDI)Fiber Distributed Data Interface (FDDI)

• Asynchronous Transfer Mode (ATM) & SONETAsynchronous Transfer Mode (ATM) & SONET

• Cellular Telephony Technologies Cellular Telephony Technologies

Available in a range of bandwidths, leased from 3Available in a range of bandwidths, leased from 3 rdrd-parties, available in (sub)urban areas-parties, available in (sub)urban areas

Lower bandwidths, leased from 3Lower bandwidths, leased from 3rdrd-parties, available anywhere via satellite, older tech-parties, available anywhere via satellite, older tech

Range of bandwidths, leased from 3Range of bandwidths, leased from 3 rdrd-parties, available worldwide even via satellite-parties, available worldwide even via satellite

““Do it yourself” WAN, 100 Mbps, for campus/metro-area WANs, fiber optic loopDo it yourself” WAN, 100 Mbps, for campus/metro-area WANs, fiber optic loop

““Do it yourself” WAN, Gbps bandwidth, no limit on size, fiber optic mesh/tree/loopDo it yourself” WAN, Gbps bandwidth, no limit on size, fiber optic mesh/tree/loop

Lower bandwidths, leased from 3Lower bandwidths, leased from 3rdrd-parties, available in (sub)urban areas, pay by packets-parties, available in (sub)urban areas, pay by packets

Wide Area (IP)Networking Standards





The Numerous Ways to the Internet

INTERNETINTERNETISPISP ISPISPTelCo

CorporateCorporateWANWAN

CorporateCorporateFirewallFirewall

Local Local CableCable

TV TV CompanyCompany

Dial-Up or Dial-Up or Leased analogLeased analog

phone phone

NeighborhooNeighborhoodd

hubhub

ISPISP

SatelliteSatelliteISPISP

AnalogAnalogcell phonecell phone

dial-updial-up

CellCo

CDPDCDPDor 3G or 3G

servicesservices

xDSL, xDSL, ISDNISDN

Frame Frame RelayRelay

f-T1f-T1T1T1T3T3

Ethernet, FDDI, ATM, Ethernet, FDDI, ATM, Frame RelayFrame Relay

Asymetric bandwidthAsymetric bandwidth

Shared bandwidthShared bandwidth

BurstyBurstytransmissiontransmission

RestrictedRestrictedservicesservices

TemporaryTemporaryconnectivityconnectivity

Variable Variable loadingloading

PermanentPermanentconnectivityconnectivity

Low Low BandwidthBandwidth

NOTICE:NOTICE:

Once you connect to the Internet (or another Once you connect to the Internet (or another system that is connected to the Internet) you system that is connected to the Internet) you MUST take all possible precautions to protect MUST take all possible precautions to protect

your systems from cyber attack because your systems from cyber attack because statistically you have a nearly 100% statistically you have a nearly 100%

probability that you WILL get “probed” and probability that you WILL get “probed” and then attacked if your systems have known then attacked if your systems have known

vulnerabilitiesvulnerabilities

Wide Area (IP)Networking Standards





Communications & System Security

IP Security and DHS Security Requirements

• The Department of Homeland Security (DHS), The Department of The Department of Homeland Security (DHS), The Department of Energy (DOE), FERC, NERC and various industry/standards Energy (DOE), FERC, NERC and various industry/standards organizations are calling for cyber security standards for Industrial organizations are calling for cyber security standards for Industrial Control Systems.Control Systems.

• The Process Control Security Requirements Forum (PCSRF) is The Process Control Security Requirements Forum (PCSRF) is attempting to define standards, in cooperation with NSA, DOE and attempting to define standards, in cooperation with NSA, DOE and NIST. Using ISO/IEC-15408 (“Common Criteria”) as their baseline.NIST. Using ISO/IEC-15408 (“Common Criteria”) as their baseline.

• NERC has issued Cyber Security Standard-1200NERC has issued Cyber Security Standard-1200 Addresses generation control systems (EMS/ISO)Addresses generation control systems (EMS/ISO) Addresses T&D supervisory control systemsAddresses T&D supervisory control systems Has sixteen sections: 1201-1216 each with a separate focusHas sixteen sections: 1201-1216 each with a separate focus Standard 1300 by 2005 will include substation equipment, Standard 1300 by 2005 will include substation equipment,

generator plant DCS systems and establish audits/penaltiesgenerator plant DCS systems and establish audits/penalties






Communications & System Security

• The Instrumentation, Systems & Automation Society (ISA SP99)The Instrumentation, Systems & Automation Society (ISA SP99)• The Institute of Electrical and Electronic Engineers (IEEE)The Institute of Electrical and Electronic Engineers (IEEE)• The American Gas Association (AGA)The American Gas Association (AGA)• The American Water Works Association (AWWA)The American Water Works Association (AWWA)• Chemical Industry Data Exchange (CIDX)Chemical Industry Data Exchange (CIDX)• ISO/IEC [International Standard 17799]ISO/IEC [International Standard 17799]

Several Industry/Standards groups have taken the initiative and are Several Industry/Standards groups have taken the initiative and are working on defining standards and recommendations for their industries:working on defining standards and recommendations for their industries:





• VPN “tunnels” for non-secure applications/devicesVPN “tunnels” for non-secure applications/devices• IPIPSECSEC or TLS within devices that can support it or TLS within devices that can support it• Use of SSL “in front” of non-secure applicationsUse of SSL “in front” of non-secure applications• Use of public key encryption, PKI and certificatesUse of public key encryption, PKI and certificates• Use https and s/mime for web and e-mail functionsUse https and s/mime for web and e-mail functions

VPN tunnelVPN tunnel


““IP”IP”LAN/WANLAN/WAN

““IP”IP”LAN/WANLAN/WAN

TLS or IPTLS or IPSECSEC

Non-secure Non-secure traffictraffic

Non-secure Non-secure traffictraffic

Secure trafficSecure traffic




SECURE LINKSECURE LINK

SECURE IP STACKSECURE IP STACK





•The TCP/IP network architecture has been adopted by just about The TCP/IP network architecture has been adopted by just about everyone as the “everyone as the “de facto”de facto” standard for networking. standard for networking.

Functional advantages of IP-based networking

Why Implement TCP/IP Networking

•TCP/IP networks are capable of being made very secure, using a range of TCP/IP networks are capable of being made very secure, using a range of security enhancements such as session encryption, destination security enhancements such as session encryption, destination authentication, etc…authentication, etc…

•The INTERNET provides a low-cost “backbone” for making network inter-The INTERNET provides a low-cost “backbone” for making network inter-connections almost anywhere in the world.connections almost anywhere in the world.

•The hardware/software building blocks for assembling a LAN/WAN are The hardware/software building blocks for assembling a LAN/WAN are readily available and guaranteed to “plug and play” (switches, gateways, readily available and guaranteed to “plug and play” (switches, gateways, bridges, routers, hubs, etc.)bridges, routers, hubs, etc.)

•TCP/IP communications can be built into even the most simple of devices, TCP/IP communications can be built into even the most simple of devices, including basic IP applications like “ftp”, and up to the level of a full-blown including basic IP applications like “ftp”, and up to the level of a full-blown web server.web server.•Performance enhancements introduced with IPv6 provide for “virtual Performance enhancements introduced with IPv6 provide for “virtual circuit” capabilities and for applications that require “streaming” circuit” capabilities and for applications that require “streaming” functionality.functionality.

•Most (all) of the “Ethernet” protocols are IP based Most (all) of the “Ethernet” protocols are IP based





With high-speed LANs, such as Ethernet, and broadband WANS,we With high-speed LANs, such as Ethernet, and broadband WANS,we can create the equivalent of a “conventional” telephone switching can create the equivalent of a “conventional” telephone switching

systemsystem

VoIP (Voice Over IP) runs “on top” of IP over high-speed LAN/WANs and delivers telephone-like servicesVoIP (Voice Over IP) runs “on top” of IP over high-speed LAN/WANs and delivers telephone-like services

IP header UDP header RTP header Digitized voice

““RTP” packets sent every few millisecondsRTP” packets sent every few millisecondsto keep the QoS close to TelCoto keep the QoS close to TelCo

BroadbandBroadbandWAN WAN

RouterRouter

EthernetEthernetSwitch Switch

VoIPVoIP““telephone” telephone”

VoIP – Voice over IP technologies

Ethernet LANEthernet LAN


ConfigurationConfigurationworkstation workstation





Standard devices are available to transport “Video” streams over IP Standard devices are available to transport “Video” streams over IP

10/100 Mbps ETHERNET LAN or Broadband WAN10/100 Mbps ETHERNET LAN or Broadband WAN

Still image capture (JPEG) or Streaming Video (MPEG) to web browser asStill image capture (JPEG) or Streaming Video (MPEG) to web browser asa web page (HTTP) per camera. Each camera has a unique IP address.a web page (HTTP) per camera. Each camera has a unique IP address.


WebCam Technologies






Legacysystem

Legacysystem

Terminal Server Terminal Server

TCP/IPTCP/IPWAN WAN

Remote serial ports Remote serial ports

Supporting Legacy SW/HW


MODEM MODEMTelCoTelCo

Connection Connection

LegacyLegacyApplicationApplication

(COM1:) (COM1:) Local serial portLocal serial portCOM1: COM1:

LegacyLegacyApplicationApplication

(COM1:) (COM1:) Remote serialRemote serial port COM1: port COM1:

Local serial portLocal serial portCOM1: COM1:

Point-to-point “serial” communication circuit Point-to-point “serial” communication circuit

Local LAN Local LAN Local LAN Local LAN

Special drivers Special drivers

Port to Port Transfer





Serial Communications Over IP

Serial-ENET converters“wrap” serial messages for IP transmission andcan be “linked” in pairs to create “virtual” serialcircuits

BroadbandBroadbandWANWAN

Local Area and Wide Area Networking Standards





Serial Communications Over IP

Serial-ENET converters“wrap” serial messages for IP transmission andcan be “linked” in pairs

BroadbandBroadbandWANWAN

“Virtual” COM ports arecreated in software sothat “legacy” software

still functions !

COM1:COM1:

COM2:COM2:

COM3:COM3:

COM4:COM4:

Remote applications to any/all serialdevices using “virtual” COM ports

Local Area and Wide Area Networking Standards





RouterRouterwith Firewallwith Firewall

functions functions


EthernetEthernetSwitch Switch

System withEthernetTCP/IPsupport

System withRS-232/C

port

Integrating IP Technologies


OtherOtherSites Sites

Corporate Corporate Servers Servers

IP networks support IP networks support concurrent data concurrent data

streams streams

VoIP phone VoIP phone WebCam WebCam

Terminal Terminal server server





• IPv6 defined a priority designation that can be used to request special treatment IPv6 defined a priority designation that can be used to request special treatment

• Flags in the IP header can indicate message priority and preferential treatmentFlags in the IP header can indicate message priority and preferential treatment

• Real-Time Transfer Protocol (RTP) is intended to provide “stream like” performanceReal-Time Transfer Protocol (RTP) is intended to provide “stream like” performance

• RSVP protocol can verify and “reserve” necessary bandwidth availability on pathRSVP protocol can verify and “reserve” necessary bandwidth availability on path

• IPv6 added a virtual connection-oriented session setup capability to IPIPv6 added a virtual connection-oriented session setup capability to IP

• IPv6 is deployed on most/all of the Internet backbone computersIPv6 is deployed on most/all of the Internet backbone computers

• Most vendors have upgraded to IPv6 in their softwareMost vendors have upgraded to IPv6 in their software

Many of the applications being deployed over the Internet Many of the applications being deployed over the Internet suffered from a lack of guaranteed throughput. Things like suffered from a lack of guaranteed throughput. Things like streaming audio and video did not operate acceptably under an streaming audio and video did not operate acceptably under an IPv4 environment. Thus the IETF added changes to IP, in the IPv4 environment. Thus the IETF added changes to IP, in the version 6 release, which were specifically aimed at providing a version 6 release, which were specifically aimed at providing a mechanism for guaranteed, real-time performance:mechanism for guaranteed, real-time performance:

Performance issues with WAN

& Internet TechnologyBandwidth Guarantees & Traffic Priority





Real-world Examples of Successful

Implementations

High-Bandwidth digitalHigh-Bandwidth digitalnetwork with dual T1snetwork with dual T1sand backup ISDN to alland backup ISDN to all

key generator sites.key generator sites.

Individual Individual generator generator

unitsunits

PRIMARY EMS

BACKUP EMS (redundant)

Real-time status and measurements Real-time status and measurements plus meter data and breaker statusplus meter data and breaker status

Generator raise/lower and exciter Generator raise/lower and exciter control plus breaker controlscontrol plus breaker controls

VoIPVoIPphonephone

LegacyLegacyRTURTU

Encryption andEncryption anddigital certificate baseddigital certificate basedauthentication (VPN)authentication (VPN)

ISO X.509v3ISO X.509v3

2 – 3 Second updates

1 – 2 Second response

Remote Intelligent Gateway

(RIG) Project

Frame RelayFrame RelayNetworkNetwork

TelCTelCoo

DNP3.0 Serial





Real-world Examples of Successful

Implementations


Data Provider Gateway (DPG)

Project

TheTheINTERNETINTERNET

LocalLocalISPISP

LocalLocalISPISP

LocalLocalISPISP

LocalLocalISPISP

LocalLocalISPISP

InternetInternetProxy ServerProxy Server

PRIMARY EMS

BACKUP EMS

Local LANLocal LAN

Local LANLocal LAN

BroadbandBroadbandMODEMMODEM

DPGDPG

Metering Metering and I/Oand I/O

IP-DNP3.0IP-DNP3.0protocol protocol with SSLwith SSL

Digital “cert” based authentication Digital “cert” based authentication and encryption out to each DPGand encryption out to each DPG

with ISO X.509v3 certificateswith ISO X.509v3 certificates

Non-VitalNon-Vital(black start, AGC, etc..)(black start, AGC, etc..)

Generating UnitsGenerating Units

Real-time status and measurements Real-time status and measurements plus meter data and breaker statusplus meter data and breaker status



Shed-able commercial Shed-able commercial & light industrial & light industrial

loadsloads

Hundreds of sites Hundreds of sites scattered around the scattered around the

state of Californiastate of California





From ‘Real-time’ to the ‘Real-world’ Over the NET From ‘Real-time’ to the ‘Real-world’ Over the NET

P R E S E N T E D B YP R E S E N T E D B Y

William T. Shaw - PhD, CISSPWilliam T. Shaw - PhD, [email protected]@direcway.com

www.cybersecconsulting.comwww.cybersecconsulting.com

The challenge of incorporating remote automation facilities The challenge of incorporating remote automation facilities and systems into the Corporate Enterprise digital networkand systems into the Corporate Enterprise digital network

Net Communications in Utility

Automation/IT (Wednesday, October 27,

2004)

Extending Digital Networking to the Field ©2004 InfoNetrix, LLC and Cyber Security Consulting All...

Documents

Transcript of Extending Digital Networking to the Field ©2004 InfoNetrix, LLC and Cyber Security Consulting All...