The POWERful Choice – Carrier Ethernet or MPLS For Power Utilities Yaakov (J) Stein CTO.

The POWERful Choice –Carrier Ethernet or MPLSFor Power Utilities

Yaakov (J) SteinCTO

The POWERful Choice - Carrier Ethernet or MPLS 2

SONET/SDH is being phased out

SONET technology is widely deployed, but• SONET technology is aging • SONET equipment is becoming obsolete and hard to find• SONET is hard to maintain (parts hard to obtain and expensive)• finding staff with SONET expertise is becoming ever more difficult• no new rates/functionality/standards/applications are being developed for SONET

Modern packet-based networks (based on Ethernet, MPLS, and IP)• are the present and future• are broadband and becoming even more so• are less expensive (both CAPEX and OPEX) and more flexible• are being actively extended (e.g., migration to 61850)

But there are open questions• can all the relevant services be migrated to packet (e.g., teleprotection, synchrophasors)?• which packet-based network to choose ?


The options

• Carrier Ethernet– Based on most popular technology in the world– Look and feel similar to SONET/SDH networks– Mature carrier-grade technology– Support for synchronization– Network security mechanisms available

• MPLS– Core network technology– Inherits rich IP control plane– Deterministic paths available (MPLS-TE)– Has no inherent network security

• MPLS-TP– Based on MPLS, but adds mechanisms patterned after Carrier Ethernet

• OAM and protection switching (including rings)– Look and feel similar to SONET/SDH networks– Does not require IP forwarding or control plane– Has no inherent network security


What is Carrier Ethernet ? (1)

Ethernet started out as a LAN technologyLAN networks are small

and operated by consumerand hence are easily managed

When Ethernet left the LAN environmentnew mechanisms were needed, e.g.– scalability (to reach 100s of thousands of end-points)– OAM (Fault Management, Performance Monitoring)– deterministic (Connection-Oriented) connections– support for various topologies (e.g., point-point, rings, trees)– resilience mechanisms (e.g., Automatic Protection Switching) – support for synchronization

Carrier Ethernet (CE) adds carrier-grade features to Ethernetso that it can replace SONET/SDH as a transport network

Metcalf’s original sketch of Ethernet

Blue means Ethernet


What is Carrier Ethernet ? (2)

• Mature Technology– widely deployed by service providers– promoted and maintained by Metro Ethernet Forum (MEF)

• Deterministic and Connection Oriented (unlike connectionless IP) – provisioning through management system (not routing)– support for point-point, multipoint-multipoint, ring, tree, … topologies

• Support for Quality of Service (up to 8 Classes of Service)– enforcement of bandwidth profiles (dual token bucket shaping/policing)– color (conformance) marking

• Carrier-grade operations mechanisms:– service activation testing (Y.1564)– Fault Management (802.1ag, Y.1731)– Performance Monitoring (Y.1731)– Automatic Protection Switching (G.8031, G.8032)– Synchronization <timing distribution> (SyncE, 1588)

• Network security mechanisms:– access authorization (802.1X) – source authentication, integrity and optional encryption (MACSec)


What is MPLS ? (1)

MPLS started out as a technology to accelerate IP forwardingby setting up tunnels to transport IPother traffic can be transported via pseudowires

MPLS defined by the IETF, and inherits the rich IP protocol suite like all IETF protocols, MPLS does not define layer 2 or below

MPLS is a mature technology for core IP networksfull Traffic Engineering is available, but not traffic conditioning (policing/shaping)supports mesh topologies uses local Fast ReRoute (not protection switching) for resilience no network security mechanisms (since core elements are trusted)

A new MPLS version (MPLS-TP)takes MPLS out of the core network into the transport domainWARNING: there are two non-interoperable versions (from IETF and ITU-T)

Red means MPLS


What is MPLS ? (2)

We can now distinguish four distinct flavors of MPLS:

1. best effort MPLS (usually with LDP, perhaps with RSVP-TE for FRR)not true CO – pinned to route not to Network Elementsused in Internet core

2. MPLS for L3VPN services (RFC 4364 <ex-2547> using BGP)used to deliver VPN services to business users

3. traffic engineered MPLS-TE (currently with RSVP-TE)true CO with resource reservationused when strict SLA guarantees must be given (banks, government, …)

4. transport profile - MPLS-TP (with management or RSVP-TE)– does not assume the existence of IP forwarding plane– does not require the IP control plane (can work with management systems)– implements OAM and APS functionality (based on Carrier Ethernet)– supports ring topologies– still in initial phases of deployment (little interop testing has been performed)– does not add network security features (still susceptible to attack)


The battlefront

• Ethernet started in the local network (LAN) and for many years has moved into transport networks

• MPLS started in the core network (WAN) and is now trying to conquer transport networks with MPLS-TP

local networkTRANSPORT NETWORK

corenetwork

ETHERNET

MPLS

first mile

last mile

Technical Comparison


Features in common

Both Ethernet and MPLS (all flavors) :• can natively transport IP traffic

– Ethernet can natively transport other traffic types (EtherType)– MPLS can transport other traffic types via pseudowire technology

• can be transported over SONET/SDH and OTN• are being actively developed (by multiple standards organizations)

– Ethernet by the IEEE, MEF, ITU, …– MPLS by the IETF, ITU-T, …

• may exhibit very high or very low transit delays (and everything in-between)(unlike SONET/SDH which has constant switching latency)– very high delay when packets need to wait in a queue– very low delay (much lower than SONET/SDH) for prioritorized traffic

Both CE and MPLS-TP :• typically use network management systems for configuration• define FM/PM OAM and diagnostic tests• support rings and define APS


1st reason for differences – format

Ethernet packet headers are self-describing

• a globally unique source address• a globally unique destination address• an optional connection identifier (VLAN)• optional Class of Service and Drop Eligibility Indicator• a payload protocol type identifier (EtherType)

MPLS packet headers are only locally meaningful

• no unique addresses• a locally meaningful label (stack)• a TTL field (to avoid packet looping)• optionally a Traffic Class (TC) field

DA(6B) SA(6B) T/L(2B)VT(2B) VLAN(2B)

Label (20b) TC(3b) S(1b) TTL (8b)


2nd reason for differences – control

• Ethernet was zero-touch in broadcast domain LANs

• CE uses network management to support large networks

• Ethernet does define L2 control protocols (STP, LACP, LLDP, …) but does not define a routing protocol (neglecting TRILL, E-VPN, etc.)

• Best effort MPLS tunnels according to topology found by IP routing protocols• So best effort MPLS:

– does not require sophisticated management system– does requires the full logistics of an IP network

• MPLS-TE requires both IP routing and a sophisticated management system

• MPLS-TP is the only flavor of MPLS that does not require IP routing but when routing is not used, configuration management is required

(basically equivalent to Carrier Ethernet)


Additional differences

• Ethernet defines physical (L1) layers (but may run over MPLS as a PW)MPLS requires a server layer to transport it (which is usually Ethernet)

• Ethernet can not tolerate forwarding loopsCarrier Ethernet supports rings with G.8032 and Industrial Ethernet supports them with High-availability Seamless RedundancyMPLS can (since it contains a TTL field)

• Carrier Ethernet supports bandwidth profiles (bucketing)

• Ethernet supports IEEE 1588 timing distribution over packet and defines a physical layer to support Synchronous Ethernet

MPLS may obtain support for 1588 (work ongoing in IETF) but since MPLS does not a physical layer it can not provide physical layer synchronization support

• Ethernet has network security mechanisms (MACsec, 802.1X, SNMPv3)MPLS does not define any standardized network security mechanisms and since MPLS has no source address it can not provide source authentication


The new trend – SDN

Distributed routing protocols are limited to • finding simple connectivity• minimizing number of hopsbut can not perform more sophisticated operations• optimizing paths under constraints (e.g., delay, security)• setting up backup paths• integrating networking functionalities (e.g., NAT, firewall) into paths

Lately, a new paradigm has arisen – Software Derived Networking, which: • removes control protocols from network elements• replaces distributed routing with centralized path computation• configures the forwarding actions of the switches from a central site

SDN sees the IP/MPLS control plane as a disadvantage and adopts the Carrier Ethernet / MPLS-TP approach

New SDN tools can optimally manage operational networks • SDN services can be added and modified at the speed of software• SDN should lead to significant OPEX reductions


Why not use both ? (1)

We have seen that MPLS is missing several critical features in particular, synchronization and network security

So, why not use both Ethernet and MPLS taking the best features of each ?

In fact, MPLS does not define its own physical layer and the most common physical layer supporting MPLS is Ethernet although MPLS can be transported over other physical layers, e.g., SDH or OTN

So the real question is whether to maintain an Ethernet network or an MPLS network in addition to an Ethernet network !

ETHERNET

MPLS



How many networks are there ?

Ethernet defines its own physical layer although Ethernet can be transported over other physical layers

When transporting IP over Ethernet there are actually 2 or 3 networks3 IP2 Ethernet1 Ethernet or optionally SONET/SDH or OTN

MPLS does not define its own physical layerWhen transporting IP over MPLS there are actually 3 or 4 networks

3 IP2.5MPLS2 Ethernet1 Ethernet or optionally SONET/SDH or OTN

Do we care how many networks there are ?



Yes, because maintaining networks is never trivial or expense-free!

• Attempts to design a network to use Ethernet as a dumb pipe under MPLS usually end up using a large number of Ethernet mechanisms• For example, when running MPLS over Ethernet, one usually needs :

– staff trained in Ethernet technologies and staff trained in IP/MPLS technologies– to be able to run Ethernet OAM and MPLS diagnostic tools– to maintain an Ethernet NMS and MPLS management screens

• Network management is the core business of a network service provider and for them it may be reasonable to maintain duplicate staff, tools, operations centers, etc.

Network maintenance is not the core business of a power utility and the duplication and added complexity is usually not justifiable

Operational Comparison


Utilities network requirements

• Traffic types (not an exhaustive list)– SCADA operational traffic – teleprotection traffic– synchrophasor traffic– surveillance video– general TCP/IP

and there is a growing demand for bandwidth• Determinism (CO behavior)

– best effort / nondeterministic (Internet-like) behavior is not acceptable• Resilience (critical infrastructures must be highly reliable)

• Low (and constant) end-end delay (for SCADA and teleprotection applications)

• Management– networks presently employ centralized management– end-to-end provisioning and maintenance are musts

• Synchronization• Network security (merits discussion in a separate section)

– cyber security is a growing concern– regulatory requirements are appearing


Traffic types

• SONET/SDH was designed to transport certain traffic types and rates– mapping new traffic types is difficult and complex– transport of most traffic rates is inefficient– no higher rates are being defined for SONET/SDH

• Ethernet was designed to transport arbitrary traffic types and rates– EtherType mechanism to indicate payload types– pseudowire technology may also be used– no rate constraints– higher rates being defined (presently 100Gbps)

• MPLS was designed to transport IP traffic– pseudowire technology enables transport of arbitrary traffic types– MPLS imposes no rate constraints or limitations

So, regarding traffic, SONET/SDH is reaching End-of-Lifewhile Ethernet and MPLS are future proof!


Determinism

Networks are deterministic when traffic consistently flows through the network in the same way

With nondeterministic networks (e.g., IP and best effort MPLS)each packet may take a different route through the network, thus

• enabling intermittent faults (only when the packets happen to go there)• complicating troubleshooting (where did the packets go?)• excluding the reservation of resources or specific processing

at particular network elements (you can’t be sure the packets will go where you want …)

SONET/SDH networks are Circuit Switched, and thus completely deterministic

CE and some types of MPLS (TE, TP) are Connection Orientedand thus relatively deterministictraffic consistently takes the same path through the networkbut does not always take precisely the same time to traverse

So, due to lack of determinism, best effort MPLS is not a reasonable candidate for a power utility operational network


Resilience

• SONET/SDH is well-known for its Automatic Protection Switching– gold standard 1:1 APS supports < 50 millisecond protection switching time– 1+1 APS can provide hitless switching (at the cost of increased bandwidth)

• Best effort MPLS relies on slow rerouting for recovery

• MPLS with Fast ReRoute performs local detours around failures– at the expense of loss of determinism

• CE and MPLS-TP support several types of APS– CE’s G.8031 and G.8032 and MPLS-TP’s RFC 6378, 6974, ITU-T G.8131 – 1+1 pseudowire redundancy achieves hitless switching

at the cost of increased bandwidth consumption

So, from the point of view of resilience CE and MPLS-TP are as good as SONET/SDH !


End-end delay and delay consistency

Some operational traffic require low and consistent delayFor example, teleprotection’s end-end delay budget may be 6 milliseconds

• SONET/SDH latency is typically sufficiently low (e.g., under 2 msec.)– is constant– is independent of SONET/SDH rate (whether OC3 or OC192)

• Carrier Ethernet and MPLS may have much lower transit latencies prioritorized packets only wait for the packet already exiting the switch for the worst case (1500B packet that just started) this latency is:– 1 2 3 msec at 100 Mbps (about the same as a SONET/SDH frame)– 12.3 msec at 1 Gbps– 1.23 msec at 10 Gbps

• TDM pseudowire traffic requires a jitter buffer – eliminates delay variation– adds additional latency (under 1 msec for prioritorized, low PDV, traffic)

So, delay considerations actually favor CE and MPLS over SONET/SDH !


What about delay asymmetry?

For some bi-directional applicationsthe delay must be symmetric(the same in both directions)

• SONET/SDH– ADM rings have constant delay asymmetry

(without “spatial reuse” management)– teleprotection mechanisms compensate for this

• CE and MPLS– CE is always co-routed and thus symmetric– best effort MPLS may not be co-routed– but MPLS-TE and MPLS-TP can be

• TDM pseudowire– may introduce buffer asymmetry– correct implementation keeps this very low

So, delay asymmetry considerations actually favor CE and MPLS-TP over SONET/SDH !

SONET/SDH–Delay asymmetry

CE or MPLS Symmetric delay


Management

• SONET/SDH networks typically are typically supportedby sophisticated management platforms(Operation Support Systems, Network Management Systems)developed by vendors or users over decades

• Carrier Ethernet was developed to replace SONET/SDH in service provider networks

and thus borrowed heavily from existing SONET/SDH management architecture, terminology, and look-and-feel

• MPLS-TP was developed to be functionally equivalent to previously developed CE

and thus borrowed heavily from existing SONET/SDH management architecture, terminology, and look-and-feel

So, from the point of view of management SONET/SDH, CE and MPLS-TP are exceptionally similar while best-effort MPLS is completely different


Synchronization

• Synchronization (AKA timing) the ability to transfer highly accurate frequency or time

over a network (obviating reliance on GPS)While timing may not be a requirement in present-day utilities networks

it is crucial to support some imminent applications such as new teleprotection mechanisms and synchrophasors

• SONET/SDH has native support for frequency transfer as it requires highly accurate frequency for its own operation but does not support time transfer

• Ethernet fully supports both time and frequency transfer by use of Synchronous Ethernet (ITU-T G.8261/2/4) for physical layer support and support for IEEE 1588 Precision Time Protocol for packet layer distribution

• MPLS does not currently support timing at all work in IETF-TICTOC is progressing to provide some support for IEEE 1588 having no physical layer, MPLS will never support physical layer frequency distribution

So, regarding synchronization CE is the best alternative followed by SONET/SDH (and MPLS has no support)


Summary (so far)

So far we have compared CE, MPLS, and MPLS-TP to SONET/SDH, and foundTraffic types and growing demand for bandwidth• Determinism

– SONET/SDH, CE and MPLS-TP are all acceptable – best effort MPLS is unacceptable for critical operational networks

• Resilience – CE and MPLS-TP (but not non-TP MPLS) are as good as SONET/SDH

• Delay (including consistency and asymmetry) – favors CE and MPLS (for asymmetry only MPLS-TP) over SONET/SDH

• Management– CE and MPLS-TP (but not non-TP MPLS) are equivalent to SONET/SDH

• Synchronization– CE has full support, SONET/SDH supports frequency, MPLS is deficient

In the final section we will discuss Network Security and discover further differences between Carrier Ethernet and MPLS

Network Security for Power Utilities


Security highlights

• MPLS was invented for core networkswhere network elements are in secure locations, and therefore trustedand was thus designed without any security mechanisms

• In particular, the MPLS forwarding plane– can not be source authenticated (no source address!)– has no standardized integrity mechanism and the MPLS control plane uses soft-state protocols

• Ethernet was designed for untrusted network elements• CE does not suffer from most of these ailments since Ethernet ports can be:

– Authorized (by 802.1X) and Ethernet packets can be – Source authenticated (by MACsec)– Integrity (and replay) tested (by MACsec)

and CE uses a security-enabled management plane (instead of a control plane)

Let’s see why this is important !


MPLS data plane DoS (injection) attack

• Once a packet is inside an MPLS network it can not be blocked (no authentication) • If an attacker gains physical access to an MPLS network node (e.g., by using a free port)

he/she can inject fake MPLS packets (guessing until a valid label is found) • At high rates this injection can overwhelm forwarding resources

MPLSCore

SubstationRTU

LAN

TPR

Central Site

DMS/EMS

Data Center

Connect to any freeMPLS port

PE

CE can block this attack using 802.1X authorization

LSP

LSP

PE

PE

PE

Data Plane

http://www.google.co.il/url?sa=i&rct=j&q=&esrc=s&frm=1&source=images&cd=&cad=rja&uact=8&docid=kj5J4ALxjj5WEM&tbnid=dGFhLVlqFQVYIM:&ved=0CAUQjRw&url=http://www.thisiswhyimbroke.com/guy-fawkes-mask&ei=UMymU9esJsbI0QW_rIC4Bw&bvm=bv.69411363,d.ZGU&psig=AFQjCNFU018jNU9G_bZtMQf8gRo007D3Bw&ust=1403526604261322


Central Site

DMS/EMS

Data Center

MPLS man in the middle attack

• Tampering means falsifying SCADA RTU/IED <-> control station data• Can be implemented by owning the switch or by inserting an evil SFP into a port• MPLS has no integrity mechanisms to detect tampering• Result can be power disruption and/or physical damage to equipment

MPLSCore

SubstationRTU

LAN

TPR

LSP

LSP PE

Data Plane

CE can block this attack using MACSec’s integrity check



Central Site

DMS/EMS

Data Center

MPLS LSP swap attack

• The attacker exchanges the internal labels belonging to 2 substations• Implemented by owning the switch or via an Evil SFP• MPLS has no source authentication mechanisms• The Central Site control systems now believe that indications from substation A

belong to substation B (and vice versa)

MPLSCore

Substation ARTU

LAN

TPRLSP PE

Data Plane

CE can block this attack using MACSec’s source authentication

Data A

Substation BRTU

LAN

TPRPE

Data B

Data A

Data B

Data A

Data B



MPLS control plane attackNot relevant for MPLS-TP w/o control plane

• MPLS control protocols (e.g., LDP and RSVP-TE) are soft-state (when contact with a peer is lost, LSPs are withdrawn)

• Intermittently deleting consecutive few heartbeat packets causes massive denial of service

• A more complex attack can poison the Label Information Base

SubstationRTU

LAN

TPR

Central Site

DMS/EMS

Data Center

MPLSCore

LDP or RSVP-TE

LSP PE

Control Plane

Attack is not applicable to CE which doesn’t use a Control Plane



Summary (final this time)

In our previous summary we saw that • Carrier Ethernet and MPLS-TP (but not MPLS) were as good as, or even better than, SONET/SDH on most accounts

and had the further advantage of being future proof• Best effort MPLS is nondeterministic

and should not be considered for operational networks• Concerning synchronization (crucial for up-and-coming applications) Carrier Ethernet has full support while MPLS has none (thus diminishing its status as being future proof)Now we have seen that • Regarding Network Security

MPLS is highly vulnerable while Carrier Ethernet possesses mechanisms to fight off attacks

These facts should be taken into account when planning future transport networks

Yaakov (J) Stein

CTO

[email protected]

The POWERful Choice – Carrier Ethernet or MPLS For Power Utilities Yaakov (J) Stein CTO.

Documents

Transcript of The POWERful Choice – Carrier Ethernet or MPLS For Power Utilities Yaakov (J) Stein CTO.