Nanog55.Talk43.Christracy Nanog55 100g Jun2012

7/27/2019 Nanog55.Talk43.Christracy Nanog55 100g Jun2012

1/62

100G Deployment

Challenges & Lessons Learnedfrom the ANI Prototype & SC11

Chris Tracy, Network Engineer

ESnet Engineering Group

June 5, 2012

Version 1.2


2/62

Lawrence Berkeley National Laboratory U.S. Department of Energy | Office of Science

06/05/12

Title Misnomer

Talk title references a prototype 100G network

- This is no longer just a prototype

Rollout of a nationwide, production-quality IP/MPLSbackbone running at 100Gbps is underway

- Scheduled to transition to full production by Nov 2012

2


3/62


06/05/12

Outline

ANI (100G Prototype) and ESnet5 (100G Production)

100Gbps optical transmission over the WAN

100Gigabit Ethernet Transmission

Pluggable optics

Testing, measurement, debugging, fault isolation

Interoperability

Case Study

References

3


4/62


06/05/12

Advanced Networking Initiative (ANI)

4

100GigE prototype circuits


5/62


06/05/12


4

~13,000mi dark fiber


6/62


06/05/12


4

experimental testbed


7/62


06/05/12

Advanced Networking Initiative (ANI)Seven 100GigE Circuits for SC11

5

optical add/drops and/or regens

* Geography is only representational


8/62


06/05/12

Advanced Networking Initiative (ANI)Seven 100GigE Circuits for SC11

5

optical add/drop and router



9/62


06/05/12

ESnet5 - Optical Network

6

~13,000mi lit fiber



10/62


06/05/12


6

wave capacity shared with Internet2



11/62


06/05/12


6* Geography is only representational

over 200 amp sites


12/62


06/05/12



over 60 optical add/drop sites15 "directionless" add/drop


13/62


06/05/12



23 100G s (metro & wide-area)


14/62


06/05/12



46 100G add/drop transponders22 100G re-gens across wide-area


15/62


06/05/12 7* Geography is only representational

ESnet5 - Routed NetworkScheduled for Production by November 2012


16/62




20 routers with 100G interfaces


17/62




52 layer-3 100GigE interfaces


18/62




customer-owned 100G routers


19/62




100G exchange point interconnects


20/62


06/05/12

100G Optical Transmission over the WAN

System Perspective

100G Ethernet client signals - IEEE 802.3ba [2]

- on core-facing andcustomer/peer-facing edge ports

- weve standardized on 10x10-10km & 10x10-2km CFPs

10x10 MSA [5]

non-IEEE standard

100G client signals mapped into Optical Transport Unit 4 (OTU4)

- ITU-T G.709 [3] for encapsulating 100GigE into OTU4

- includes mandatory Forward Error Correction (FEC)

transported using dual polarization-quadrature phase shift keying(DP-QPSK) technology with coherent detection [4]

8


21/62


06/05/12


Dual polarization-quadrature phase shift keying (DP-QPSK)

DP: two independent optical signals, same frequency, orthogonal

- single transmit laser, each signal carries half of the data

- two polarizations ! lower modulation rate ! reduce optical bandwidth

- allows 100G payload (plus overhead) to fit into 50GHz of spectrum

9source: [1] and [4]

phase-amplitude constellation2 bits per symbol

Q

I

signals added together to form QPSK signal


22/62


06/05/12


Dual polarization-quadrature phase shift keying (DP-QPSK)

QPSK: encode data by changing the phase of the optical carrier

- compare to on-off keying (OOK), intensity modulation ! 0=off, 1=on

- further reduces the symbol rate by half, sends twice as much data

Together, DP and QPSK reduce required rate by a factor of 4

10source: [1] and [4]

phase-amplitude constellation2 bits per symbol

signals added together to form QPSK signal

Q

I


23/62


06/05/12

Spectral Efficiency: 10 vs 40 vs 100Gb/s

11

0.15nm/div0.4nm! 50GHz

1551.

68

1551.

83

1551.53

1551.

98

1552.

13

1552.

28

1552.

43

1552.58

1552.73

1552.

88

1553.

03

nm

Source: [1] Roberts, Beckett, Boertjes, Berthold and Laperle (2010, July)


24/62


06/05/12


11


20GHz

14GBaudeach

10G single-polarization single-carrier 40G coherent dual-polarization single-carrier

100G coherent dual-polarization dual-carrier

1551.

68

1551.

83

1551.53

1551.

98

1552.

13

1552.

28

1552.

43

1552.58

1552.73

1552.

88

1553.

03

nm



25/62


06/05/12


11


20GHz

14GBaudeach

10G single-polarization single-carrier 40G coherent dual-polarization single-carrier

100G coherent dual-polarization dual-carrier

1551.

68

1551.

83

1551.53

1551.

98

1552.

13

1552.

28

1552.

43

1552.58

1552.73

1552.

88

1553.

03

nm

1551

.72

1552.

12

155

2.

52

155

2.

93

50GHz



26/62


06/05/12


Coherent Detection (on the receiver) - Breakthrough Technology

Technology originally developed circa 1980s - [1] and [7]

- advances in digital signal processing (# of gates, operatingfrequency) allowed coherent detection to emerge as disruptive

technology in optical communications Offers significant improvement in noise tolerance over conventional

direct detection schemes

Able to compensate for propagation impairments such as chromaticdispersion (CD) and polarization mode dispersion (PMD)

- dispersion compensating fibers are no longer needed in long-haulapplications

high-speed A/D samples incoming analog components

- DSP ASIC to apply numerical adaptations, recover signal

12


27/62


06/05/12


Coherent Detection - Another Breakthrough

works like a radio receiver - source: [1], [4], and [7]

- use a strong local oscillator tuned to the frequency of interest

has provided a breakthrough in optical filtering capabilities

- tunable optical filters exist but have never been cost-effective

important impacts of this technology:

- colorless ROADMs are [finally] becoming available by usingcoherent optical filtering - complete software reprogrammability

- no longer need fixed channel filters (arrayed waveguide gratings)

- no longer important to stay aligned to a 50 or 100 GHz ITU grid

- spectrum can be used more efficiently, system is more flexible

- wavelength selective elements would need to become flexible

13


28/62


06/05/12


Amp placement and fiber span length has become very important

100G transmission for a given channel of optical spectrum isapproaching Shannon limit

- going higher than 100Gb/s, at the same distance, in same amount of

spectrum, requires improvement of SNR (or just use more spectrum) Even if we just want to stay at 100Gb/s, reach is limited by OSNR

- EDFAs contribute noise - Amplified Spontaneous Emission (ASE)

- on a long segment consisting of cascaded EDFAs, a few very long(high-loss) fiber runs degrades the systems SNR

- could break up long spans and place more amplifiers for less lossbetween amps, or possibly deploy Raman, to improve SNR

- otherwise, extra 100G re-gens could be required to get the desiredreach - might not get advertised reach if amp spacing is not ideal

14


29/62


06/05/12

Flexible Grid Concept

15Source: [8] Gringeri, Basch, Shukla, Egorov and Xia (2010, July)

At rates >100Gb/s, 50GHz spacing would require high SNR, will limit reach

Future-proof network by allowing channels to occupy more spectrum


30/62


06/05/12

100G Ethernet Transmission(IEEE 802.3ba)

16Source: [11] Drolet and Duplessis (2010, July)

Unlike 1G or 10G, 100G has embraced parallelism throughout its design


31/62


06/05/12

100G Ethernet Transmission(IEEE 802.3ba)


Unlike 1G or 10G, 100G has embraced parallelism throughout its design

physical coding sublayer (PCS)provides reordering & realignment


32/62


06/05/12

100G Ethernet Transmission(specific to 100GBASE-LR4)


LR4


33/62


06/05/12



physical medium attachment (PMA)(often referred to as a gearbox)

LR4


34/62


06/05/12




impact of gearbox on data ordering

LR4


35/62


06/05/12




possibility of skew between wavelengths

impact of gearbox on data ordering

LR4


36/62


06/05/12

CFP Pluggable Optics - LR4 or 10x10 ( )?

18

Interface Wavelength Cost Fiber Type Connector Reach *Link

BudgetComment

100GBASE-LR44 x 25G WDM lanes1294.53-1310.19nm $$$ SMF SC / LC 10 km 7.3 dB IEEE Standard [2]

10x10-2km10x10-10km

10 x 10G WDM lanes1521-1597nm

$$ SMF SC / LC2 km

10 km5.1 dB8.0 dB

non-IEEE standard, no 10:4gearbox, 10x10 MSA [5]

100GBASE-SR1010 x 10G parallel MMF

840-860nm$

parallelMMFs

MPO / MTP100 / 150 mOM3/OM4 MMF

8.3 dB IEEE Standard [2]

10x10 LR10 optics are gaining popularity and vendor support

Does your equipment support third-party optics?

Similar to 1G & 10G pluggables, vendors sell certified optics

3rd party may not work (or be supported) by your equipment

Digital Diagnostic Monitoring (optical power monitoring) may or may not be supported, especially with uncertified optics

if supported - may be aggregate, per-lane, or both

* reach is highly dependent on link budget. some modules may have higher-than-average launch power.

10x10LR10CFP

akaLR10


37/62


06/05/12

CFP Pluggable Optics - LR4 or 10x10 ( )?

18

Interface Wavelength Cost Fiber Type Connector Reach *Link

BudgetComment

100GBASE-LR44 x 25G WDM lanes1294.53-1310.19nm $$$ SMF SC / LC 10 km 7.3 dB IEEE Standard [2]

10x10-2km10x10-10km

10 x 10G WDM lanes1521-1597nm

$$ SMF SC / LC2 km

10 km5.1 dB8.0 dB

non-IEEE standard, no 10:4gearbox, 10x10 MSA [5]

100GBASE-SR1010 x 10G parallel MMF

840-860nm$

parallelMMFs

MPO / MTP100 / 150 mOM3/OM4 MMF

8.3 dB IEEE Standard [2]

10x10 LR10 optics are gaining popularity and vendor support

Does your equipment support third-party optics?

Similar to 1G & 10G pluggables, vendors sell certified optics

3rd party may not work (or be supported) by your equipment

Digital Diagnostic Monitoring (optical power monitoring) may or may not be supported, especially with uncertified optics

if supported - may be aggregate, per-lane, or both

* reach is highly dependent on link budget. some modules may have higher-than-average launch power.

10x10LR10CFP

akaLR10


38/62


06/05/12

Pluggable Optics - Optical Power

19

Interface Lanes Average Launch PowerEach Lane (min / max) Total Average Launch Power(max) Source

100GBASE-LR4 4 -4.3 to 4.5 dBm +10.5 dBm [2]

10x10-2km10x10-10km

10-6.9 to 3.0 dBm (2km)

-5.8 to 3.0 dBm (10km)+13 dBm [5]

Field Testing of 100G Transceivers

CFPs will appear to launch hot* with standard power meters

testing spectrum compliance and per-lane optical power withparallel optics on SMF requires use of an Optical Spectrum Analyzer

some power meters may be tunable to measure different "s(consider passband width, marginal for spectrum compliance)

every CFP will have a slightly different Tx lane transmit profile

- maximum reach over dark fiber may vary slightly depending onthis profile

* we are referring to optical launch power, but weve found CFPs also run very hot, temperature-wise


39/62


06/05/12

Pluggable Optics - Other Considerations

20

100G Fiber Patches

10x10 LR10: can generally* be connected back-to-back without attenuation

SR10: MPO-terminated parallel MMF for short patches (we havent tried this)

- going through patch panels would be messy

- break-out using 1xMPO to 10xSC octopus cables?

CXPs

high-density, targeting MMF connections

to be interoperable with CFPs

active optical cables

* check the data sheets for your transceivers or talk to your vendor first, of course

source: [6]


40/62


06/05/12

Testing and Measurement

October 3rd, 2011, 14:07 Pacific

First pings across transcontinental 100GigE between two routers

41 days left before the start of SC11

- Want to give users as much time as possible to tune their

applications

Ping is a good first step, need more testing before hand-off to users

- Transport system shows clean FEC, low BER, optical layer clean

- We want to drive links at 100% utilization and measure 0 drops

- Validate QoS, throughput, latency, loss, interoperability, etc.

How are we going to test seven 100GigE circuits?

- At this point, work on building/deploying hosts capable of sourcing>10Gbps of traffic had begun, still preliminary

- Do we need a 100GigE hardware tester?21


41/62


06/05/12

Testing and Measurement - UDP flows

Could we leverage the 10Gbps systems we already have deployed?

22

SUNN-ANI

perfSONAR

SALT-ANI100G

SUNN-SDN210G

ESnet4

ANI

2x10G


42/62


06/05/12



22

SUNN-ANI

perfSONAR

SALT-ANI100G

SUNN-SDN210G

ESnet4

ANI

2x10G

lets have some fun with routing loops

static-route 10.10.10.0/30next-hop [sunn-ani]

static-route 10.10.10.0/30

next-hop [salt-ani]

static-route 10.10.10.0/30

next-hop [sunn-ani]


43/62


06/05/12



22

2x2Gbps UDP iperfTTL=52 to loop 50X SUNN-ANI

perfSONAR

SALT-ANI100G

SUNN-SDN210G

ESnet4

ANI

2x10G

lets have some fun with routing loops

hop 1

hop 2,4,6,...,52 hop 3,5,...,51

iperf -c 10.10.10.1 -u -b 2G -l8800 -i1 -t600 -T52

iperf -c 10.10.10.2 -u -b 2G -l8800 -i1 -t600 -T52

and carefully chosen TTLs

loop 50x


44/62


06/05/12


This was a quick solution to saturate the link, can we improve it?

23

ANI

SALT-ANI STAR-ANI AOFA-ANI

ANL-ANI


45/62


06/05/12


This was a quick solution to saturate the link, can we improve it?

23

use policy-based routing for more complex loops - still testing UDP

firewall ACLs for counting packets / bytes to measure loss

ANI

SALT-ANI STAR-ANI AOFA-ANI

ANL-ANI

ip-filter applied at ingress:

match src-ip [perfSONAR IP]

count number of packets and bytes

action forward next-hop [anl-ani]

ip-filter applied at ingress:

match src-ip [perfSONAR IP]

count number of packets and bytes

action forward next-hop [salt-ani]

test flows


46/62


06/05/12

Testing and Measurement - TCP flows

Deployed systems on ANI experimental testbed - source/sink 4x10Gbps

24

48.6ms RTT, 97.9Gbps aggregate TCP throughput with 10 TCP streams

Thanks to Eric Pouyoul, Brian Tierney, and many others

nersc-ani

nersc-diskpt-1

eth2

eth3

eth4

eth5

nersc-diskpt-2

eth2

eth3

eth4

eth5

nersc-diskpt-3

eth2

eth3

eth4

eth5

anl-mempt-1

eth2

eth3

eth4

eth5

anl-mempt-2

eth2

eth3

eth4

eth5

anl-mempt-3

eth2

eth3

eth4

eth5

9/1/3

9/1/5

9/1/1

9/1/4

10/1/4

10/1/3

10/1/5

10/1/7

10/1/9

10/1/8

10/1/10

10/1/6

1/1/1

anl-ani

9/1/1

9/1/2

9/1/3

9/1/4

10/1/3

10/1/4

10/1/5

10/1/6

10/1/7

10/1/8

10/1/9

10/1/10

1/1/1

VPLS

10

VPLS

40

VPLS

20

VPLS

30

NERSC Argonne


47/62


06/05/12

Testing and Measurement - TCP flows

Deployed systems on ANI experimental testbed - source/sink 4x10Gbps

24

48.6ms RTT, 97.9Gbps aggregate TCP throughput with 10 TCP streams

Thanks to Eric Pouyoul, Brian Tierney, and many others

nersc-ani

nersc-diskpt-1

eth2

eth3

eth4

eth5

nersc-diskpt-2

eth2

eth3

eth4

eth5

nersc-diskpt-3

eth2

eth3

eth4

eth5

anl-mempt-1

eth2

eth3

eth4

eth5

anl-mempt-2

eth2

eth3

eth4

eth5

anl-mempt-3

eth2

eth3

eth4

eth5

9/1/3

9/1/5

9/1/1

9/1/4

10/1/4

10/1/3

10/1/5

10/1/7

10/1/9

10/1/8

10/1/10

10/1/6

1/1/1

anl-ani

9/1/1

9/1/2

9/1/3

9/1/4

10/1/3

10/1/4

10/1/5

10/1/6

10/1/7

10/1/8

10/1/9

10/1/10

1/1/1

VPLS

10

VPLS

40

VPLS

20

VPLS

30

NERSC Argonne


48/62


06/05/12

Testing and Measurement

25

Advantages Disadvantages

Hardware

Testers

stable & robust, tests many protocols 40/100Gbps interface speeds test single streams > 10Gbps

accurate measurement of loss/reordering very precise control of packet content can source/sink high bandwidth flows

consisting of small packets

TCP implementation at high-BW often notstateful (e.g., no congestion controlalgorithm), not a good indicator of how an

actual end host would perform generally cannot run user applications

that interface to the test equipment relatively expensive

PC-based

Testers

run user applications (data transfer suchas GridFTP, bbFTP, scp)

real-world TCP implementation- pluggable TCP congestion control algo. choice of various measurement and

analysis applications depending on exact configuration, packet

capture capabilities

at 100G, requires careful build and tuning host issues due to NIC driver, kernel,

interfering user/system processes

line-rate flows will have a limit onsmallest packet size

possibility of bugs in measurementapplications

accurate measurement problematic- implement counting on hardware


49/62


06/05/12

Debugging and Fault Isolation

Tracking down loss

Relatively easy to track down if there are accurate counters

- and the ability to filter on some particular test traffic (e.g. 5-tuple)

Difficult when there is background traffic and equipment does not

allow packet counting via ACLs, etc.

Important to understand where allof the Drop counters are

Tracking down re-ordering

More difficult (especially if you dont have a hardware tester)

Some open-source test tools can report on sequence errors or mis-orders, but there is room for improvement

Can capture TCP packet dumps, analyze with tcptrace

26


50/62


06/05/12

Debugging and Fault Isolation

Using loopbacks

Loopbacks can be very useful in certain situations

Bring up facility loops facing a layer-3 router on transport equipment

- ping broadcast address of layer-3 interface

- check for obvious signs of trouble

- helpful to verify patching

If wide-area link is down, bring up loops at termination points toverify patching between layer-3 and transport gear

- Place loops at re-gen points, check for link up/down on router

Facility loops always face fiber (client side or network side)

- e.g., client facility loop on transport layer CFP facing a router

Terminal loops face internals of client board close to client laser

27


51/62


06/05/12

Interoperability

Between Layer-2/3 Gear and Optical Transport Equipment

no issues have arisen

transport layer is expected to be transparent, so far, it has been

Between Layer-2/3 Equipment and other Layer-2/3 Equipment

Remember, 100GigE is still quite new

Vendors have chosen slightly different ways of supporting 100G

- not necessarily optimizing for one single line-rate flow

- in some cases these have led to challenges in achieving desiredperformance

- re-working the way the system is logically configured or physicallycabled can potentially have a huge impact

Important to understand exactly how flows transit your equipment

- How exactly do line cards (and fabric) interface to the backplane?28


52/62


06/05/12

Interoperability

Testing with both 100GBASE-LR4 and 10x10 MSA LR10 CFPs

Basic interoperability* demonstrated between:

- Juniper T1600

- Juniper MX

- Brocade MLX- Ciena OME6500

- Ciena 5410

- Alcatel-Lucent 7750 SR

Support foroptical power monitoring varied, hardware dependencies

- per-lane power, aggregate, both per-lane & aggregate, or neither Over short distances, 10x10-2km interoperates with 10x10-10km

Thanks to SCinet, StarLight, and Indiana University engineers for their test results.

29* basic interoperability meaning we could pass packets and links ran error-free, not that every feature worked


53/62


06/05/12

Case study:Transport of 2x100G "s to SC11

30

Inter-domain, multi-vendor collaboration

Presented to SCinet WAN team as alienwaves at the Westin building in Seattle

100Gbps connection to ANI and Internet2

100Gbps connection to CANARIE

Supported numerous 40/100Gbpsdemonstrations on ANI and Internet2- https://my.es.net/topology/sc11/overview

#1180mi/1900km


54/62


06/05/12


30






#1180mi/1900km OEO conversion

100GigE--OTU4at SC11 & SALT


55/62


06/05/12


30






#1180mi/1900km regen 2X

OTU4 / DP-QPSK


56/62


06/05/12


30






#1180mi/1900km all-optical

pass-through


57/62


06/05/12


30





Supported numerous 40/100Gbpsdemonstrations on ANI and Internet2- https://my.es.net/topology/sc11/overview managed by

SCinet WAN

team

#1180mi/1900km


58/62


06/05/12


30






#1180mi/1900km hand-off as

alien wavesOTU4 / DP-QPSK


59/62


06/05/12

Future Work

Near-term: Mixing 10G NRZ & 40G/100G DP-QPSK "s in metro rings

These technologies do not necessarily play nice together [9]

Kevin McGrattans talk [10] has a lot of detailed background on thistopic

Understand impact in terms of:

- reach penalty on 100G channels (how is OSNR affected?)

- maximum channel count

- colorless ROADM deployment (with non-coherent wavelengths)

Deployment of Colorless ROADM components

Long-term: Flexible Grid?

Flexible Wavelength Selectable Switch (WSS) components

Super channels at 1Tb/s which occupy >100GHz of spectrum31


60/62


06/05/12

Acknowledgements

This work would have been possible without the help of the rest ofthe ESnet staff as well as our partners:

Internet2

GlobalNOC / Indiana University

Alcatel-Lucent

Ciena

Level3

SCinet Routing, WAN, and Fiber Teams

32


61/62


06/05/12

References

[1] Roberts, K., Beckett, D., Boertjes, D., Berthold, J., Laperle, C. (2010, July). 100G and Beyond with Digital Coherent SignalProcessing. Communications Magazine, IEEE, 48(7), 62-69. Retrieved January 16, 2011, from IEEE Xplore database.

[2] IEEE 802.3ba. http://standards.ieee.org/getieee802/download/802.3ba-2010.pdf

[3] ITU-T G.709. http://www.itu.int/rec/T-REC-G.709/en

[4] OIF-FD-100G-DWDM-01.0 - 100G Ultra Long Haul DWDM Framework Document (June 2009). http://www.oiforum.com/public/documents/OIF-FD-100G-DWDM-01.0.pdf

[5] 10x10 MSA Revision 2.4. http://www.10x10msa.org/documents/MSA%20Technical%20Rev2-4.pdf

[6] Finisar C.wire. http://finisar.com/products/active-cables/C.wire

[7] Digital Coherent Receiver Technology for 100-Gb/s Optical Transport Systems. http://www.fujitsu.com/downloads/MAG/vol46-1/paper18.pdf

[8] Gringeri, S., Basch, B., Shukla, V., Egorov, R., Xia, T. (2010, July). Flexible Architectures for Optical Transport Nodes andNetworks. Communications Magazine, IEEE, 48(7), 40-50. Retrieved January 16, 2011, from IEEE Xplore database.

[9] Magill, P. (2010, September). 100G Coherent trials and deployments: AT&T plans and perspective. ECOC2010, 36th EuropeanConference and Exhibition on Optical Communication. Retrieved January 16, 2011, from IEEE Xplore database.

[10] McGrattan, K. Considerations in Migrating DWDM Networks to 100G. http://events.internet2.edu/2011/jt-clemson/agenda.cfm?go=session&id=10001587

[11] Drolet, P., Duplessis, L. (2010, July). 100G Ethernet and OTU4 Testing Challenges: From the Lab to the Field. CommunicationsMagazine, IEEE, 48(7), 40-50. Retrieved January 16, 2011, from IEEE Xplore database.

33


62/62


06/05/12 34

Questions?

Thanks!

Nanog55.Talk43.Christracy Nanog55 100g Jun2012

Documents

Transcript of Nanog55.Talk43.Christracy Nanog55 100g Jun2012