Interoperability Update: Dynamic Ethernet Services Via Intelligent Optical Networks

James D. Jones, Alcatel; Lyndon Ong, Ciena; Monica Lazer, AT&T

Abstract This article describes the 2005 Worldwide Interoperability Demonstration held by the Optical Internetworking Forum (OIF) and showcased during SUPERCOMM 2005. The event highlighted Ethernet services transported over intelligent optical networks, using equipment from 13 of the industry’s leading vendors located in 7 carrier lab facilities around the world. The demonstration utilized a distributed optical control plane based on OIF Implementation Agreements to control a multi-layer network providing Ethernet over SONET/SDH adaptation and transport. The article describes the global test network, services, architecture and overall test approach. It also describes innovations made to the optical control plane to handle multi-layer signaling and lists further refinements needed to make these services operational.

1 Introduction The Optical Internetworking Forum (OIF) conducted its second World Interoperability Demonstration, in conjunction with SUPERCOMM held in Chicago on June 7 - 9, 2005. Member companies demonstrated dynamic Ethernet services enabled over a global optical network, building on the results of a similar global demonstration in 2004. At SUPERCOMM 2004, OIF demonstrated dynamic end-to-end SONET/SDH connection management between client devices and transport network elements from many vendors in a multi-domain, transport network spanning multiple carrier laboratories. The 2004 event also demonstrated Ethernet adaptation over SONET/SDH using GFP (Generic Framing Procedure), VCAT (Virtual Concatenation) and LCAS (Link Capacity Adjustment Scheme) as a separate objective. At SUPERCOMM 2005, the OIF took a major step by integrating these two features and creating a multi-layer control plane control to trigger end-end Ethernet connections over a SONET/SDH network. The result was a global network enabling clients to directly request Ethernet services over carriers’ SONET/SDH networks. While the demonstration focused on Ethernet Private Line service enabled by the distributed control plane, it also evaluated Ethernet Virtual Services (Virtual Private Line, Virtual Private LAN and Internet Trunking) over the optical transport network. This demonstration was motivated by continued growth in demand for Ethernet services in public networks, and the carriers’ imperative to maximize utilization of their existing SONET/SDH transport infrastructure. To do this, interoperability is required at many levels (i.e., transport, control and management planes) to allow flexibility as the network evolves to support present and future Ethernet services. At the same time, carriers have heterogeneous core optical transport networks comprised of a range of bearer technologies, infrastructure granularity options, and survivability mechanisms. Products and systems tested for interoperability included routers,

multi-service provisioning platforms (MSPPs), SONET/SDH cross-connects, optical switches, optical add-drop multiplexers (OADMs) and reconfigurable OADMs (ROADMs). OIF Implementation Agreements and interoperability trials address the challenge by requiring control plane solutions be developed in the context of such heterogeneous environments, and be able to co-exist with the existing network. The demonstration was executed on a global stage with seven Carriers across three continents inter-networking through an intelligent control plane with equipment from thirteen vendor participants. A network of over 70 nodes was built up in progressive stages, beginning with local lab testing, followed by intra-continental regional testing and culminating in a live, global real-time network test. During the global demonstration, over 20 optical connections were simultaneously active among the test sites. This included a live video feed between two carrier labs, which was then transported to the SUPERCOMM show site. This video connection was dynamically setup and torn down remotely from the show floor via the optical control plane.

2 Creating a World Wide Demonstration The OIF World Interoperability Demonstration at SUPERCOMM 2005 was built as a global network, in that all equipment was located at carriers’ research laboratories across the world: Asia, Europe, and North America. Equipment from multiple vendors was interconnected within the laboratories. Virtual connections were established among carriers, with the exception of several instances, as discussed in more detail below. Participants included: • Seven Carrier Lab Locations:

• Europe: Deutsche Telekom, France Telecom, Telecom Italia • Asia: China Telecom, NTT • North America: AT&T, Verizon

• Thirteen Vendors:

Alcatel Lucent Technologies Avici Systems Mahi Networks CIENA Corporation Marconi Corporation Cisco Systems Nortel Networks Fujitsu Sycamore Networks Huawei Tellabs Lambda Optical Systems

The overall equipment topology for this event is shown in Figure 1. This diagram shows all the equipment involved in the OIF World Interoperability Demonstration, its function in each of the carrier labs and the transport plane. As mentioned above, in addition to the virtual links interconnecting carrier labs facilities, there were several real links that were used to carry video application to showcase the demonstration: one OC3 link connected the AT&T and Verizon facilities, and one DS3 link connected the AT&T facilities to the SUPERCOMM demonstration booth. Video streams were used to illustrate the status of calls during the demonstration.

Asia to Europe

Figure 1 Overall Topology of OIF World Interoperability Demonstration

In addition, a Signaling Control Network (SCN) was set-up in an architecture simulating operational networks. In each carrier lab, an internal SCN connected all the equipment. The individual carrier SCNs were interconnected simulating carrier-to-carrier interconnections. However, since this is still a demonstration network, not an operational network, IPsec tunnels

NTT Labs

NTT

Avici NTT

NTT

NTT

Avici

AT&T Labs

America to Europeia

Intra inental

or America to As -cont

TNE Client

VerizonLabs

FranceTelecom

DeutscheTelekom

TelecomItalia

China Telecom

Fujitsu

Alcatel

Alcatel

Tellabs

Tellabs

Tellabs

Cisco

Ciena

Avici Avici

AviciAvici

Sycamore Marconi

Alcatel/EoS

Marconi

Alcatel

Ciena

Lucent/EoS

Navtel Huawei

CienaHuawei

Marconi

Lambda Optical

Cisco

CiscoAvici

Avici

CienaSycamore

Mahi

OC-3

Cisco/EoS Ciena/EoS

Cisco/EoSLucent/EoS

LEGENDSycamore/EoSAlcatel

Ciena/EoS

Nortel Tellabs

Alcatel

Fujitsu/EoS

Fujitsu

Sycamore

were used over the public Internet (as opposed to dedicated signaling networks). This SCN was used for signaling and routing information exchanges. For display purposes, a custom software application intercepted signaling messages sent over the SCN to build a live view of the active calls. This application analyzed data fields within the signaling messages to build the picture of the active calls, as shown in the example in Figure 2. These global topology views were available both in the lab sites and at the SUPERCOMM show floor.

Figure 2 Dynamic Display of Active Calls

A generic diagram illustrating the service and interface types included in the demonstration is shown in Figure 3. As seen in the diagram, all client devices were connected to the optical network via Ethernet interfaces. This illustrates the evolution of the OIF UNI [1] from supporting SONET/SDH services to support of Ethernet services. The 2005 demonstration was focused on dynamic Ethernet services enabled by intelligent optical networks. Ethernet services were delivered across multiple carrier labs with various bandwidth characteristics by Switched Connections (SCs, initiated by client UNI signaling), Soft Permanent Connections (SPCs, initiated by a management system) and hybrids of both. To support dynamic Ethernet calls, several key technologies were required as described below: GFP [2], VCAT [3], LCAS [4], UNI 2.0 [5], E-NNI 1.0 [6] (with extensions), inter-layer call and connection coordination. For these calls, the Ethernet signals were mapped to SONET/SDH payloads using GFP/VCAT/LCAS standards. The test cases included mapping of both full and partial rate Gigabit Ethernet signals and both co-routed and diversely routed SONET/SDH containers carrying an Ethernet signal.

... VCATLCAS

GFP-FVCATLCAS

GFP-FVirtual Concatenation Group (21 STS-1 or 7 VC-4)

......

GigE GigE

Carrier CDomain

OIF UNI OIF E-NNI OIF UNI

Carrier ADomain

Carrier BDomain

OIF E-NNINE NE NE NE NE NE

EthernetClient

EthernetClient

Ethernet Layer Call/Connection Flow

SONET/SDH Layer Call/Connection Flow

UNI-N UNI-N UNI-CUNI-C

SONET/SDH

Control PlaneView

Transport PlaneView

Ethernet Ethernet

Figure 3 Multi-Layer Control Plane and Transport plane Architecture

In addition, the control plane operated at multiple layers and inter-layer coordination was supported in the optical network elements at the edges of the networks. For a Switched Connection, the client device control plane interfaced to the network control plane via a UNI 2.0 interface supporting Ethernet transport. Using UNI 2.0, there is no need for the customer equipment to have any awareness of how the network implements support for Ethernet services (whether Ethernet mapped to SONET/SDH, or native Ethernet, or Ethernet mapped directly to optical wavelengths). In this demonstration SONET/SDH transport was used for Layer 1 transport. The edge NE performed the appropriate mapping of the Ethernet service to SONET payload (consistent with the service parameters requested in the signaling messages) and originated signaling at both Layer 1 and Layer 2 in support of the request. The reverse process took place at the egress point of the network. To illustrate the utility of the optical control plane a video application was set-up between the AT&T and Verizon labs as follows (see Figure 4):

• A video server was connected to a router (client to the intelligent optical network) at each site. The server was used to transmit video streaming between sites when the Ethernet call was when available.

• Whenever the inter-site Ethernet call was established, each lab could view both the video that originated from their lab and the video from the remote lab.

• Whenever the Ethernet call was deleted each site could only view video originated in its own lab.

• During the SUPERCOMM show, an additional DS3 link connected the AT&T labs facilities and the SUPERCOMM booth. The AT&T video streaming was available continuously at the SUPERCOMM booth. The Verizon video stream was available during the time that the Ethernet call between the routers at the two sites was established. The OIF booth demonstrations started with the Ethernet call established and both videos were visible in the booth. The Ethernet call between the two sites was then deleted and the visitors could see the change in the call map topology on one monitor (similar to

Figure 2), the video control stream from AT&T on a second monitor, and the interrupted Verizon video stream on a third monitor. Next, the Ethernet call was re-established and the call map topology reflected the change. At the same time, the visitors could see the Verizon video streaming again in real time.

OIF Booth (Chicago, IL)

Figure 4 Video Application Configuration

3 Background OIF subscribes to the ITU-T ASON architecture, as discussed in [7], and has based its Implementation Agreements for the optical UNI and the optical E-NNI on the ITU-T ASON Recommendations, especially ITU-T G.7713.2 [8] for RSVP-based signaling. The optical UNI [1] enables clients of optical networks to dynamically request connections without knowing network internal topology, while the E-NNI [6] automates the establishment of these connections between optical networks. Together, UNI and E-NNI permit dynamic A-to-Z provisioning of services across an optical network in real time without manual intervention, resulting in faster and more efficient operation than traditional optical networks. Link state routing protocol based on OSPF-TE based on [9] is used for automated network topology distribution and link status updates inside the network.

3.1 Multi-Layer Networking for Ethernet-over-SDH This year’s testing focused on multi-layer networking, where connections in a client layer are supported by the dynamic establishment of connections in a server layer. In practice, carrier networks consist of multiple technology layers, ranging from Layer 3 IP connectionless packet transport down to Layer 0 physical connectivity, such as fiber cross-connection. Often new services arise at one layer and must be transported efficiently using a core lower layer network.

E-NNI OC3 AT&T-VZ

DS3 Private Line

GigE GigE UNI 2.0

Tellabs 8860

Alcatel 1677

Ciena CD

VZ Video Server

ATT Video

AT&T Labs (Middletown, NJ)

Verizon Labs (Waltham, MA)

Server

Avici TSR

Control plane Static Segments of Video Path

UNI 2.0

Dynamic Segments of Video Path Enabled by Control Plane

One current example of this is the support of Ethernet services, which are growing rapidly as a carrier service offering, and must be transported efficiently over the carriers’ core optical transport networks. A number of technologies have been developed in the transport plane to improve the efficiency and flexibility of SONET/SDH for packet/frame transport, including GFP, VCAT and LCAS. Testing in 2005 focused on the use of the optical control plane to control connections in the optical core network to support Ethernet layer services. As shown in Figure 5, connections between client devices at the Ethernet layer were supported by dynamically established connections at the optical (SONET/SDH) layer. The number and type of connections at the SONET/SDH layer corresponded to the amount of bandwidth requested at the Ethernet layer, and an adaptation function using GFP, VCAT and LCAS was used at the originating and terminating points to encapsulate the Ethernet frames into SONET/SDH paths. To create a connection, the Ethernet UNI Client (UNI-C) sends an Ethernet connection request to the Ethernet UNI Network-side switch (UNI-N). The UNI-N is then responsible for determining the corresponding SONET/SDH requirements, creating the required SONET/SDH connections, and then signaling to the remote UNI-N that the underlying connections are available and are to be used for an Ethernet client connection. The additional stage of signaling between the source and destination UNI-Ns carries the actual Ethernet layer connection requirements, allowing the mapping at the destination UNI-N from SONET/SDH back to the Ethernet service. Both UNI-Ns then apply Ethernet-SONET/SDH adaptation using GFP. This whole process involves multiple stages of signaling to coordinate events at different layers, making the control plane processing significantly more complex than previous years’ demonstrations.

II--NNNNII DDoommaaiinn

STS-3c/VC-4 Connection STS-3c/VC-4 Connection STS-1 Connection

Gigabit Ethernet

OC48/STM16

PPhhyyssiiccaall LLiinnkk SSOONNEETT//SSDDHH LLaayyeerr

350 Mbps Connection 250 Mbps Connection 100 Mbps Connection

EEtthheerrnneett LLaayyeerr

OIF UNI 2.0

OIF UNI 2.0

OIF E-NNI

II--NNNNII DDoommaaiinn

II--NNNNII DDoommaaiinn

OIF UNI 2.0

OIF E-NNI

OIF E-NNI

OIF E-NNI

Figure 5 Multi-layer Intelligent Optical Transport Network

The key extension to the protocols was the ability to dynamically trigger the creation of the supporting server layer connection upon detecting that new optical capacity was needed. When the ingress optical switch received a UNI 2.0 call request (i.e., for an Ethernet connection), it initiates the process for creation of new optical connections, computes the required path across the optical core and creates the connection, which is then used to carry GFP-encapsulated Ethernet frames. While there is a one-to-one relationship between Ethernet connection and SONET/SDH connection (or VCAT group) for Ethernet Private Line services, as were tested, future Ethernet Virtual Private Line and Private LAN services can allow the same SONET/SDH connection or group to be used for multiple Ethernet services. Standards are not complete for how multiplexing would be supported, but candidate mechanisms are the use of the client’s VLAN tag (if present), application of a carrier VLAN tag at the UNI-N, or other tags. This is an active area of discussion in IEEE, ITU-T and IETF. With Ethernet virtual services, it will be possible to reuse an existing SONET/SDH pipe for future Ethernet connections as long as bandwidth is available. The network will be able to respond dynamically to new demands, either creating a new optical connection or reusing existing optical connections as needed.

3.2 Signaling Extensions for Multi-Layer Networking Some of the more interesting problems that needed to be solved for multi-layer networking included the following:

• How to correlate signaling at client and server layers: Since connections were established first at the server layer (SDH) and then at the client layer (Ethernet), there needed to be a way to correlate signaling at multiple layers, so that, for example, the edge switches correctly identified which SDH timeslots were to be used, and were able to exchange signaling directly between them. For the testing, a mechanism called LSP (label switched paths) Hierarchy [10] was used, which involves the addition of fields in the RSVP signaling to identify the signaling addresses of the edge switches and the creation of virtual interfaces corresponding to the server layer connections

. • How to translate the Ethernet bandwidth request into the required SDH components:

Since Ethernet bandwidth is expressed in terms of bits per second required and burst rates supported, while SDH bandwidth is expressed in terms of the size of the signal (STS-1, VC-4, VC-4-4v), there needed to be a mapping from the Ethernet bandwidth request over the UNI to the SDH bandwidth requirement at the E-NNI. In practice, such a translation would be a matter of policy determined by the service provider, since it is affected by the guarantees offered by the service provider as well as their core infrastructure. For the Demo, a mapping table was used to unambiguously map one layer to the other.

• Routing across multiple layers: In theory, the addressing used for clients and network

elements at one layer may be independent of that used at another, so that the routing of the connection may involve translation from addresses in the client layer to addresses used in the server layer. For example, the destination (or in OIF the TNA (Transport Network Assigned) address) for the Ethernet client must be translated to some associated endpoint in the SDH network for SDH path computation. For the Demo, a one-to-one correspondence was assumed, where in real networks a more complex translation may be required.

3.3 Control Plane Support of Virtual Concatenation VCAT is an inverse multiplexing capability defined in ITU-T [3] that allows multiple SONET/SDH channels to be bound into a single higher rate VCAT group (VCG). For the demonstration, separate connections were set up for each component of the group, in order to create higher survivability for the group as a whole. LCAS allows failure of individual connections to be treated as reduced bandwidth in the group without actually causing failure of the entire group. A VCAT group consisting of multiple connections in the server layer was created using multiple call setups, therefore allowing each connection to follow a different path based on its individual path computation. An example of call setup for VCAT is shown in Figure 6. A coordination mechanism was supported to synchronize the establishment of the supporting VCAT connections and the client layer call. Both parallel and sequential strategies of setting up VCAT connections were considered.

UNI 2.0 UNI 2.0

E-NNI

E-NNI E-NNI

Ethernet Client Layer Connection

VCAT Component Connections

Figure 6 Setup of diverse VCAT connections

3.4 Additional Transport plane Testing Additional transport plane-only Ethernet testing was done, based on Ethernet service specifications developed in ITU-T and the Metro Ethernet Forum (MEF). These tests demonstrated interoperability in the transport plane for Ethernet Virtual Services, where multiple Ethernet services were transported using the same SONET/SDH VCGs. The virtual services demonstrated included: Ethernet Virtual Private Line, Ethernet Virtual Private LAN and Internet Access/Virtual Trunk. These provide a complementary aspect to the control plane testing, which focused on Ethernet Private Line. For Ethernet Virtual Private Line service, for example, individual client flows were tagged at the UNI-N, aggregated into a single transport link and separated at the destination based on the values of the VLAN tags. VLAN tags as defined in IEEE 802.1Q were used to identify an individual service. Testing of Ethernet Virtual services was based on work being done in ITU-T and MEF, especially ITU-T Recommendations G.8011.1 and .2 [11], and MEF 6 [12].

3.5 Future Work Findings from the interoperability testing have been compiled and provided as input to the various standards bodies active in optical control plane specification, to identify any areas of potential confusion or omission in the standards. Future testing work may be aimed at more complex services and topologies such as dynamic control of Ethernet virtual services, as these are incorporated into optical control plane standards work.

4 Conclusions The 2005 OIF demonstration was the first time Ethernet adaptation and distributed optical control planes were brought together in an integrated fashion, and it was done on a global scale. The call/connection control of the UNI-N devices was the most important technical innovation demonstrated, in two respects. First, the UNI-N provides inter-layer control plane coordination as the client signal enters the network. The UNI-N is responsible for accepting the client connection request, initiating calls in the server layer, and completing the client layer call once the server layer is set up. Second, the UNI-N triggers and controls the Ethernet adaptation function and mapping of client layer signals into server layer containers. This architecture minimizes the overall network impact since the client Ethernet devices and core SONET/SDH devices are only concerned with a single layer.

The OIF World Interoperability Demonstration is an essential step in the evolution process of the optical control plane, helping to make it suitable for deployment in carrier networks. The testing demonstrated multi-vendor support of a distributed optical control plane, its ability to control multi-layer end-to-end services and the overall commitment to the technology by both vendors and carriers. The demonstration utilized real network elements from vendors whose market presence accounts for 64% of the 2004 worldwide revenue in the optical networking switching and routing markets (source: RHK). The equipment was hosted in technology evaluation labs of top-tier carriers in North America, Europe and Asia. While this event provided solutions for a number of technical issues, it also revealed others that need to be addressed. The knowledge gained from this interoperability demonstration is being applied to the OIF UNI 2.0 and E-NNI 2.0 signaling specifications. The experience also benefits carriers in planning migrations to distributed optical control planes and anticipating the operational considerations for multi-vendor networks. The authors would like to acknowledge all the people in the carrier labs for the tremendous efforts in putting the demonstration together and shepherding it through its stages, the staff from the participating vendors for the relentless work in accomplishing the interoperability, and the support of the OIF leadership in getting it all together.

5 References 1. OIF-UNI 1.0 Release 2, “OIF-UNI-01.0-R2-Common - User Network Interface (UNI) 1.0

Signaling Specification, Release 2: Common Part” and “OIF-UNI-01.0-R2-RSVP - RSVP Extensions for User Network Interface (UNI) 1.0 Signaling, Release 2”.

2. ITU-T G.7041, “Generic Framing Procedure (GFP)”. 3. ITU-T G.707, “Network Node Interface for the Synchronous Digital Hierarchy (SDH)”. 4. ITU-T G.7042, “Link Capacity Adjustment Scheme (LCAS)”. 5. OIF UNI 2.0 Signaling, “”, oif2003.293. 6. OIF E-NNI 1.0 Signaling, “OIF-E-NNI-Sig-01.0 - Intra-Carrier E-NNI Signaling

Specification”. 7. ITU-T G.8080/Y.1304, “Architecture for the Automatically Switched Optical Network

(ASON)”. 8. ITU-T G.7713.2, “Distributed Call and Connection Management: Signalling mechanism

using GMPLS RSVP-TE”. 9. ITU-T G.7715.1, “ASON Routing Architecture and Requirements for Link State Protocols”. 10. draft-ietf-mpls-lsp-hierarchy-08.txt, “LSP Hierarchy with Generalized MPLS TE” 11. ITU-T G.8011/Y1307, “Ethernet over Transport – Ethernet services framework”

ITU-T G.8011.1/Y1307.1, “Ethernet private line service” ITU-T G.8011.2/Y1307.2, “Ethernet Virtual Private Line Service”

12. MEF 6, “Ethernet Services Definitions - Phase I”.