A WAN-in-LAB for A WAN-in-LAB for Protocol DevelopmentProtocol Development

Netlab, CaltechNetlab, Caltech

Lachlan Andrew, Lachlan Andrew, George Lee, George Lee,

Steven Low(PI), John Doyle, Harvey NewmanSteven Low(PI), John Doyle, Harvey Newman

OutlineOutline

What and why is WAN-in-Lab? What and why is WAN-in-Lab? What can I do with WiL?What can I do with WiL? Why would I use WiL?Why would I use WiL? How do I use WiL?How do I use WiL? Future plansFuture plans

What is WAN-in-Lab?What is WAN-in-Lab?

““Wide Area” Network in a laboratoryWide Area” Network in a laboratory• Real fibre delaysReal fibre delays• Carrier-class routers, switches, …Carrier-class routers, switches, …

Why -- Spectrum of toolsWhy -- Spectrum of toolscost

abstractionmathssimulationemulationlive netwk WANinLab

NS2SSFNetQualNetJavaSim

Mathis formulaOptimizationControl theoryNonlinear modelStocahstic model

DummyNetEmuLabModelNetWAIL

UltraLightPlanetLabAbileneNLRLHCNetCENICetc

?

All scales are important– WAN-in-Lab fills a gap

What can I do with WAN-in-Lab?What can I do with WAN-in-Lab?

Other groups’ interestsOther groups’ interests

Protocol developmentProtocol development• FAST, delay-basedFAST, delay-based• MaxNet, explicit signallingMaxNet, explicit signalling• ADPM, single-bit explicit signallingADPM, single-bit explicit signalling

Impact of small buffers (U. Pittsburgh)Impact of small buffers (U. Pittsburgh) Test automatic configuration of routers Test automatic configuration of routers

(MonALISA, Ultralight)(MonALISA, Ultralight) Test distributed file-system (MojaveFS)Test distributed file-system (MojaveFS)

TCP BenchmarkingTCP Benchmarking

Our current main directionOur current main direction Evaluating others’ protocols, not oursEvaluating others’ protocols, not ours

Web interfaceWeb interface• Submit kernel patchSubmit kernel patch• Standard tests automatically performedStandard tests automatically performed• Results mailed backResults mailed back

Explicit or implicit signalling protocolsExplicit or implicit signalling protocols

Physical topologyPhysical topology

RA

RB

RC

Svr9

Svr6

Svr13

Disk4

Disk3

Eth210.1.12.2

Disk2

Svr7

Disk1Eth??

10.1.13.2

RD

Link via copper GbE switch

Direct fibre link

Direct copper GbE linkWired in, but not yet configurable (network booted)

Svr11

Svr2

Svr1

Svr10

Svr14

eth210.1.13.2

Eth210.1.12.2

Eth210.3.11.2

SA:gi1/0/12Eth1

10.3.12.2

Eth210.3.12.17

Eth210.3.12.18

Eth210.2.11.2

Eth210.2.12.2

Eth210.2.21.2

Eth210.2.24.2

Gi2/210.2.22.1

Eth??10.1.51.2

POS1/110.0.12.1

POS1/110.0.12.2

POS2/110.0.23.2

POS9/110.0.24.2

POS9/110.0.13.3

POS9/110.0.13.1

POS1/110.0.24.4

POS1/110.0.23.3

Sonet (movable by MEMS)

POS1/110.0.24.4

Capabilities: DelayCapabilities: Delay

24 spools of 100km fibre, many loopbacks24 spools of 100km fibre, many loopbacks• Set delay by MEMS switching loops in/outSet delay by MEMS switching loops in/out

130ms physical delay130ms physical delay• more with IP loopbackmore with IP loopback

2 Dummynets: long delay for cross-traffic2 Dummynets: long delay for cross-traffic

125 ms, 1.8ms steps125 ms, 1.8ms steps

External connectionsExternal connections

Linked to Ultralight, 10Gbps Physics WANLinked to Ultralight, 10Gbps Physics WAN Smooth migration testing -> deploymentSmooth migration testing -> deployment

DelayDelay• longerlonger• jitterjitter

Cross trafficCross traffic

MonitorMonitordata routeddata routedthrough WiLthrough WiL

Why use WAN-in-Lab?Why use WAN-in-Lab?

Why use WiL?Why use WiL?

Complement other levels of abstraction, Complement other levels of abstraction, not replace themnot replace them

Different ways to use it: reasons for eachDifferent ways to use it: reasons for each

Standard platform for TCP benchmarkingStandard platform for TCP benchmarking• Easier to compare with others’ resultsEasier to compare with others’ results• No need to write your own test suiteNo need to write your own test suite

Artifacts of software delaysArtifacts of software delays

Packets sent on 1ms “ticks”Packets sent on 1ms “ticks” 1Gbps = 83,333 pk/s1Gbps = 83,333 pk/s

1ms

83 packets

How can I use WAN-in-Lab?How can I use WAN-in-Lab?

Management structureManagement structure

Wil-ns.cs.caltech.eduStart script, configure, compile

ServersNetwork bootRead-only FS virtual /etc scratch disk

Data plane

Time sharingTime sharing

Coarse switching between projectsCoarse switching between projects• Servers rebooted, routers reconfiguredServers rebooted, routers reconfigured

Switchover takes ~5 minutesSwitchover takes ~5 minutes Book in advanceBook in advance

• For longer bookings, book further in advanceFor longer bookings, book further in advance• Also “ad hoc” bookings for individual hostsAlso “ad hoc” bookings for individual hosts

Can log in while others have bookedCan log in while others have booked

Future plansFuture plans

Future plansFuture plans

Benchmarking infrastructureBenchmarking infrastructure• Standardise testsStandardise tests• Use it ourselvesUse it ourselves• Develop “indices” of TCP performanceDevelop “indices” of TCP performance

Better control over capacities and buffersBetter control over capacities and buffers Better cross-traffic generationBetter cross-traffic generation

• Currently HarpoonCurrently Harpoon Investigate differences from DummyNetInvestigate differences from DummyNet Integrate DAG cardsIntegrate DAG cards

ConclusionConclusion

WAN-in-Lab fills the gap between WAN-in-Lab fills the gap between emulation and live network experimentsemulation and live network experiments

Seeks to be as realistic as possibleSeeks to be as realistic as possible• Long links, simple topologyLong links, simple topology

Focus will be on TCP benchmarkingFocus will be on TCP benchmarking

We welcome people to use itWe welcome people to use it

<http://wil.cs.caltech.edu><http://wil.cs.caltech.edu>

Spare SlidesSpare Slides

Case Study: MaxNetCase Study: MaxNet

WAN in LabWAN in Lab• Capacity: 2.5 – 10 GbpsCapacity: 2.5 – 10 Gbps• Delay: 0 – 120 ms round tripDelay: 0 – 120 ms round trip

BreakableBreakable• Won’t take down live network Won’t take down live network

Flexible, active debuggingFlexible, active debugging• Passive monitoring, AQMPassive monitoring, AQM

Configurable & evolvableConfigurable & evolvable• Topology, rate, delays, routeTopology, rate, delays, route• Modular design stays up to dateModular design stays up to date

Integral part of R&A networksIntegral part of R&A networks• Transition from theory, implementation, demonstration, Transition from theory, implementation, demonstration,

deploymentdeployment• Transition from lab to marketplaceTransition from lab to marketplace

Global resourceGlobal resource• Part of global infrastructure UltraLight led by Harvey NewmanPart of global infrastructure UltraLight led by Harvey Newman

Aim: Wind Tunnel of NetworkingAim: Wind Tunnel of Networking

EquipmentEquipment

4 Cisco 7609 routers with OC48 line cards4 Cisco 7609 routers with OC48 line cards 6 Cisco ONS 15454 switches6 Cisco ONS 15454 switches A few dozen high speed servers A few dozen high speed servers 1G switch to routers/servers1G switch to routers/servers Calient switch for OC48Calient switch for OC48 2,400 kilometres of fibre, optical 2,400 kilometres of fibre, optical

amplifiers, dispersion compensation amplifiers, dispersion compensation modulesmodules

63ms aggregate RTT delay, in two hops63ms aggregate RTT delay, in two hops• 120ms using IP loopbacks120ms using IP loopbacks

AccountsAccounts

Mail wil at cs.caltech.eduMail wil at cs.caltech.edu Sudo access to “network” commandsSudo access to “network” commands

• Ifconfig/…/Ifconfig/…/• Custom commands to set topologiesCustom commands to set topologies

Login to routers if requiredLogin to routers if required

Separate accounts for “benchmark only”Separate accounts for “benchmark only”

Configuration -- DelaysConfiguration -- Delays

Want maximum delay from limited fibreWant maximum delay from limited fibre• Signals traverse fibre 16 timesSignals traverse fibre 16 times

4 WDM wavelengths4 WDM wavelengths 4 OC48 (2.5G) MUXed onto OC192 (10G)4 OC48 (2.5G) MUXed onto OC192 (10G)

Lots of transpondersLots of transponders• WDM amplifier joins 100km spools WDM amplifier joins 100km spools 200km 200km

Configuration – delaysConfiguration – delays

OC48 slot

Amp

-------WDM Wavelength--------

Bidirectional 100km

Bidirectional 100km

16x200km

Configuration – delaysConfiguration – delays

Delay varied by adjusting the number of Delay varied by adjusting the number of OC48 hops traversedOC48 hops traversed

Calient optical switch selects required Calient optical switch selects required hopshops

Hop lengths 200km up to 1600kmHop lengths 200km up to 1600km• Maximise granularity given limited switch portsMaximise granularity given limited switch ports

Switch

ProjectsProjects

TCP benchmarkingTCP benchmarking FASTFAST

• Delay-based congestion controlDelay-based congestion control MaxNetMaxNet

• Explicit signalling congestion controlExplicit signalling congestion control

MojaveFSMojaveFS• New distributed file systemNew distributed file system

University of PittsburghUniversity of Pittsburgh• TCP with small buffersTCP with small buffers

University of MelbourneUniversity of Melbourne• Single-bit congestion markingSingle-bit congestion marking

WAN-in-Lab testbedWAN-in-Lab testbed

Dummynet and simulation introduce artifactsDummynet and simulation introduce artifacts Also need to test on Also need to test on realreal equipment equipment WAN with real delays, located in a single roomWAN with real delays, located in a single room

• Connected to an external WAN (Ultralight)Connected to an external WAN (Ultralight) Open for the community to use for benchmarkingOpen for the community to use for benchmarking

OC-48

OC-48

WAN-in-Lab capabilitiesWAN-in-Lab capabilities

CurrentCurrent PlannedPlanned

Two 2.5G bottlenecksTwo 2.5G bottlenecks

Multiple 1G bottlenecksMultiple 1G bottlenecksSix 2.5G bottlenecksSix 2.5G bottlenecks

Two “real” delaysTwo “real” delays(Emulate cross traffic delay)(Emulate cross traffic delay)

Up to six “real” delaysUp to six “real” delays

End-to-end RTT, dropEnd-to-end RTT, drop Per-router delay, dropPer-router delay, drop(movable DAG cards)(movable DAG cards)

Configuration -- delaysConfiguration -- delays

OC48 slot

Amp

-------WDM Wavelength--------

Bidirectional 100km

Bidirectional 100km

Using WAN-in-LabUsing WAN-in-Lab

Contact me – lachlan at caltech . EduContact me – lachlan at caltech . Edu

Coarse timesharingCoarse timesharing• Some users set up experiments while others Some users set up experiments while others

run experimentsrun experiments

Software setup still being developedSoftware setup still being developed• Your chance to influence our directions to tailor Your chance to influence our directions to tailor

it to your needsit to your needs

Sample MaxNet resultsSample MaxNet results

Achieves realistic delay at 1Gbit/sAchieves realistic delay at 1Gbit/s