Evaluating Networked Systems

Robert Ricci September 2007

Network Evaluation

Network EvaluationSo, you've built a faster BitTorrent

Or built a new video-on-demandservice

Or improved TCP

But how can you be sure it's better?

And how can you convince others?

3

Network EvaluationMethodologies

Ad-hoc experimentation

Live-network experimentation

Simulation

Emulation

4

Ad-hoc Experimentation

5

Ad-hoc ExperimentationJust try it on some machines on aLAN

... but are the network conditionsbelievable?... and is it big enough?

Plus, you might need to do a lot ofwork to get it set up

And it wouldn't be very automated

6

Live Internet Experimentation

7

Live Internet ExperimentationGet access to machines sitting around theInternet

PlanetLab is a popular way to do this

Much more believable network conditions

... but network conditions are notcontrollable... and are not repeatable... and it can be hard to tell what's goingon out there

You might be able to get real users

8

PlanetLabLinux machines around the world

Over 300 working ones at over 160sitesShared among many usersMost at universities

People run real services on it

Coral - .nyud.net URLsCoDEEN content distribution

www.planet-lab.org9

Simulation

10

SimulationTotally different tactic - don't run on a realnetwork at all!

Simulate the behavior of yourapplication/protocol

Application-specific simulatorsGeneral network simulators (like NS)

Totally repeatable, controllable, andtransparent

... but you can miss out on a lot of things bynot running a real application, OS... and most developers would rather workwith something "real"

11

NS-2, the Network SimulatorPopular general-purpose networksimulator

Great for detailed examination ofprotocols

Several TCP (and other protocol)implementations

Traffic generators

NAM for visualizing network activity

12

Emulation

13

EmulationCombine the previous three strategies:

Run real application on a bunch ofmachines all in one placeBut emulate realistic networkcharacteristics between themProviding a controllable, repeatable,and transparent environment

Still not perfect, though

Network emulation is only as good asthe conditions you tell it to emulate

14

Emulab

What is Emulab?An emulator

... and a lot more

A full-service experimentation platform

Experiment management, collaboration tools,etc.

A lot of hardware

Over 360 PCs, over in the MEB

A lot of code

Software runs two dozen testbeds

Created here at the U by the Flux Group

16

Emulab Hardware - Front

17

Emulab Hardware - Back

18

Emulab BasicsYou describe a network

Network links and LANs, including BW,latency, etc.Host characteristics, including OSSoftware to install

Emulab builds your network for you

You get exclusive access to some PCsYou get an isolated network with thecharacteristics you asked forWithin a few minutes

19

What Do People Use It For?Evaluation of new/improved network protocols

Peer-to-PeerDistributed SystemsMulticast

Security Research

IDSThreat Response

Reliability Work

Operating System Research

Classes

20

Experimentation on Emulab

Demo Building a Topology With GUI

How To Design A GoodExperiment

Use the scientific method

Test hypothesis"BitDeluge is faster than BitTorrent underpacket loss"

Change only one thing at a timeJust client bandwidth, or packet loss, not both]

When comparing to an existing system, makesure to compare apples to apples

Automate as much as possible

Collect as much data as you can, and sanitycheck

23

Running Your ExperimentInteractive: Log in via SSH

Remember, it's "just a PC"Good for development/debugging

Scripted: Use "event system"

Can fire off programs, change networkconditionsBrings consistency and repeatability toyour experimentLet experiments run while you sleep

24

Demo Logging in And Running Simple Commands

How Emulab Builds Your Network

Setting Up Your NodesLoads disks with custom diskloader (Frisbee)

You get the nodes to yourself

You get full root access

Shared network-mounted homedirectory

Nodes get wiped when you'redone

27

Setting Up Your NetworkVLANs provide the links and LANs

Isolated form other experiments

Automatically configures IPaddresses and routing

Inserts delay nodes for trafficshaping

28

VLANs and IsolationVLAN: Virtual LAN

Switch technology that limitswhich ports can communicate

Provides the illusion of manyexclusive-use switches

Emulab switches provisioned toavoid performance artifacts

29

Emulating NetworkCharacteristics: Delay Nodes

Transparently inserted in your linksand LANs

'Bridge' packets between interfacesat Layer 2

Can be brought down to emulate linkfailure

Passes packets through dummynet todo traffic shaping

30

dummynet details

Packetsenter

Packetsleave

BandwidthQueue

DelayQueue

AvailablebandwidthPacket

Dropper

Bandwidth Queue: Bandwidth,Queuing and Transmission Delay

Delay Queue: Propagation Delay

Packet Dropper: Packet Loss31

The Future: GENI

GENINSF-Funded "Global Environment for NetworkInnovations"

Facility for designing the next generationInternet

Enable experimentation at the network layer

Improves on the Live Internet method bycontrolling its own backbone

Virtualization allows multiple networkarchitectures on the same substrate

Some architecture work, prototype being donehere at the U

33

Wrap Up

Network EvaluationMethodologies

Ad-hoc experimentation

Useful for development, not serious evaluation

Live-network experimentation

Real, but unpredictable

Simulation

Good for very detailed studies, not so much forreal applications

Emulation

Good balance - reasonably realistic, goodcontrol/repeatability

35

EmulabPlatform for simulation, liveexperimentation, emulation

But mostly emulation

Created here at the U

Available to students andresearchers

Apply for an account atwww.emulab.net

36

end