MED: The Monitor-Emulator-Debugger for Software-Defined ...weixu/Krvdro9c/...MED functions MED:a...
Transcript of MED: The Monitor-Emulator-Debugger for Software-Defined ...weixu/Krvdro9c/...MED functions MED:a...
![Page 1: MED: The Monitor-Emulator-Debugger for Software-Defined ...weixu/Krvdro9c/...MED functions MED:a useful tool to debug problems in SDN • Create an emulator that can be set to the](https://reader036.fdocuments.net/reader036/viewer/2022071516/6139fab00051793c8c00c8ba/html5/thumbnails/1.jpg)
MED:TheMonitor-Emulator-DebuggerforSoftware-DefinedNetworks
Quanquan Zhi andWeiXuInstitute for Interdisciplinary Information Sciences
Tsinghua University
![Page 2: MED: The Monitor-Emulator-Debugger for Software-Defined ...weixu/Krvdro9c/...MED functions MED:a useful tool to debug problems in SDN • Create an emulator that can be set to the](https://reader036.fdocuments.net/reader036/viewer/2022071516/6139fab00051793c8c00c8ba/html5/thumbnails/2.jpg)
Software-Defined Networks (SDN):promises and challenges
• SDN will simplify future network design and operation
• Bugs are common─ Controller─ Switch software─ Race conditions
• Network Ops -> SystemsDevOps─ Command line -> programs─ Lacking of tools─ Fast, repeatable
![Page 3: MED: The Monitor-Emulator-Debugger for Software-Defined ...weixu/Krvdro9c/...MED functions MED:a useful tool to debug problems in SDN • Create an emulator that can be set to the](https://reader036.fdocuments.net/reader036/viewer/2022071516/6139fab00051793c8c00c8ba/html5/thumbnails/3.jpg)
Monitor-Emulator-Debugger:A debug / testing tool forSDNDevOps
• A software Debugger─ fast, repeatable, automated tools
─ addresses concurrency bugs
• Tightly coupled with physical network
- Automatic physical network sync
![Page 4: MED: The Monitor-Emulator-Debugger for Software-Defined ...weixu/Krvdro9c/...MED functions MED:a useful tool to debug problems in SDN • Create an emulator that can be set to the](https://reader036.fdocuments.net/reader036/viewer/2022071516/6139fab00051793c8c00c8ba/html5/thumbnails/4.jpg)
MED architecture overview
Monitor Emulator Debugger
App
Controlmessages
App App
Controller
RealSDN
MED Agent (Monitor) MED(Emulator)
VirtualSDN
OVS
OVS
OVS
Datapackets
Packet Tracer
Loop and Reachability Checker
Table Checker
Race Conditions Detector
Debugger Controller
Debugger
![Page 5: MED: The Monitor-Emulator-Debugger for Software-Defined ...weixu/Krvdro9c/...MED functions MED:a useful tool to debug problems in SDN • Create an emulator that can be set to the](https://reader036.fdocuments.net/reader036/viewer/2022071516/6139fab00051793c8c00c8ba/html5/thumbnails/5.jpg)
• Snapshot (initialization)─ Physical network topology (LLDP)─ Initial forwarding table states
• Capture SDN state changes over time─ Openflowmessages to/from the SDN controller─ E.g. packets-in,packets-out,ruleinstallation/removal,andportsup/downevents
• Sample data packets─ Essential for replay/testing
The monitor
![Page 6: MED: The Monitor-Emulator-Debugger for Software-Defined ...weixu/Krvdro9c/...MED functions MED:a useful tool to debug problems in SDN • Create an emulator that can be set to the](https://reader036.fdocuments.net/reader036/viewer/2022071516/6139fab00051793c8c00c8ba/html5/thumbnails/6.jpg)
The emulator: key ideas• Thekeychallenge
─ Emulating a blackbox controller from physical SDN
• Solution─ Replay all Openflowmessages captured=>settoatime
• Question:In what order?App
Controlmessages
App App
Controller
Statemessages
RealSDN
Emulator Controller
VirtualSDN
OVS
OVS
OVS
Replayedmessages
DebuggerController
App App
Injectmessages
![Page 7: MED: The Monitor-Emulator-Debugger for Software-Defined ...weixu/Krvdro9c/...MED functions MED:a useful tool to debug problems in SDN • Create an emulator that can be set to the](https://reader036.fdocuments.net/reader036/viewer/2022071516/6139fab00051793c8c00c8ba/html5/thumbnails/7.jpg)
The emulator: operation• Online Operation- Trackingmode
• Offline Operation─ “Time Travel”
Initial setup
Set_to_current Trackingstate
Set_to_stable Specified state
Set_to_nondeterministic(t)State1 State2 StateN
Replay
Online
…
Offline
![Page 8: MED: The Monitor-Emulator-Debugger for Software-Defined ...weixu/Krvdro9c/...MED functions MED:a useful tool to debug problems in SDN • Create an emulator that can be set to the](https://reader036.fdocuments.net/reader036/viewer/2022071516/6139fab00051793c8c00c8ba/html5/thumbnails/8.jpg)
The emulator: offline operations• Set to a stable state at any time
• Emulateallpossibleorderingforconcurrentevents
Initial setup
Set_to_current Trackingstate
Set_to_stable Specified state
Set_to_nondeterministic(t)State1 State2 StateN
Replay
Online
…
Offline
![Page 9: MED: The Monitor-Emulator-Debugger for Software-Defined ...weixu/Krvdro9c/...MED functions MED:a useful tool to debug problems in SDN • Create an emulator that can be set to the](https://reader036.fdocuments.net/reader036/viewer/2022071516/6139fab00051793c8c00c8ba/html5/thumbnails/9.jpg)
The debugger
• A controller that injects messages into the replayedmessage stream
• “Apps” built on top of the emulator─ Settoaspecifictime─ Anexternalcontrollerinterface
• Example debugger apps─ Packettracer
─ Loopandreachabilitychecker
─ Forwarding tablechecker
─ Raceconditionsdetector
![Page 10: MED: The Monitor-Emulator-Debugger for Software-Defined ...weixu/Krvdro9c/...MED functions MED:a useful tool to debug problems in SDN • Create an emulator that can be set to the](https://reader036.fdocuments.net/reader036/viewer/2022071516/6139fab00051793c8c00c8ba/html5/thumbnails/10.jpg)
Emulator Controller
Replayedmessages
VirtualSDN
OVS
OVS
OVS
Exampledebuggerapp 1:PacketTracer(PT)
DebuggerController
PT
TO_CONTROLLERReplay:Packet_Out
Packet_InFlow_Status_Request
Flow_status_reply
PacketmatchesNormalEntryPacketmatchesTO_CONTROLLER
Outputs:1. A packet’s entire path through the network2. Whichforwardingruleisusedoneach hop
![Page 11: MED: The Monitor-Emulator-Debugger for Software-Defined ...weixu/Krvdro9c/...MED functions MED:a useful tool to debug problems in SDN • Create an emulator that can be set to the](https://reader036.fdocuments.net/reader036/viewer/2022071516/6139fab00051793c8c00c8ba/html5/thumbnails/11.jpg)
Exampledebuggerapp 2:LoopandReachabilityChecker(LRC)
DebuggerController
PT
LRC
Asserts:
• The packet forwarding has no loop
-- AND --
• The packet reaches the destination
• Worksonlineoroffline
![Page 12: MED: The Monitor-Emulator-Debugger for Software-Defined ...weixu/Krvdro9c/...MED functions MED:a useful tool to debug problems in SDN • Create an emulator that can be set to the](https://reader036.fdocuments.net/reader036/viewer/2022071516/6139fab00051793c8c00c8ba/html5/thumbnails/12.jpg)
Exampledebuggerapp 3:RaceConditionDetector(RCD)
Asserts:
• In ANY possible concurrent state, there is no loopor blackhole
Initial setup
Set_to_nondeterministic(t)State1 State2 StateN…
Offline
• Expensive?Cantriviallyruninparallelwithmultipleemulators
DebuggerController
PT
LRC
RCD
![Page 13: MED: The Monitor-Emulator-Debugger for Software-Defined ...weixu/Krvdro9c/...MED functions MED:a useful tool to debug problems in SDN • Create an emulator that can be set to the](https://reader036.fdocuments.net/reader036/viewer/2022071516/6139fab00051793c8c00c8ba/html5/thumbnails/13.jpg)
Exampledebuggerapp 4:TableChecker(TC)
Asserts:• The forwarding tables on physical switches are thesame as those in the emulator
Forwardingrules
Flow table
OpenFlow Switch
SDN
Forwardingrules
Flow table
OVS
EmulatorTable
Checker
Installrules
DebuggerController
PT
LRC
RCD
TC
![Page 14: MED: The Monitor-Emulator-Debugger for Software-Defined ...weixu/Krvdro9c/...MED functions MED:a useful tool to debug problems in SDN • Create an emulator that can be set to the](https://reader036.fdocuments.net/reader036/viewer/2022071516/6139fab00051793c8c00c8ba/html5/thumbnails/14.jpg)
Evaluation
• Performance- Emulatorinitialization
- PacketTracing (PT)performance
• Case studies- Bugsonphysicalswitchsoftware
- Racecondition analysis
![Page 15: MED: The Monitor-Emulator-Debugger for Software-Defined ...weixu/Krvdro9c/...MED functions MED:a useful tool to debug problems in SDN • Create an emulator that can be set to the](https://reader036.fdocuments.net/reader036/viewer/2022071516/6139fab00051793c8c00c8ba/html5/thumbnails/15.jpg)
Experimentsetup
• 20switchesnetwork,typicalDCNtopology─ Pica8P-3298
─ 30,000OpenFlow total(~1,500rulesperswitch)
![Page 16: MED: The Monitor-Emulator-Debugger for Software-Defined ...weixu/Krvdro9c/...MED functions MED:a useful tool to debug problems in SDN • Create an emulator that can be set to the](https://reader036.fdocuments.net/reader036/viewer/2022071516/6139fab00051793c8c00c8ba/html5/thumbnails/16.jpg)
Initial setup performance
Discoverphysicaltopo+setupemulator
topo
Dumpallflowtablesfromswitches
InstallallflowtablesentriestoEmulator
(30Krules)
4.9sec 0.54sec 12.2sec
State changed during the setup? Redo until done.
![Page 17: MED: The Monitor-Emulator-Debugger for Software-Defined ...weixu/Krvdro9c/...MED functions MED:a useful tool to debug problems in SDN • Create an emulator that can be set to the](https://reader036.fdocuments.net/reader036/viewer/2022071516/6139fab00051793c8c00c8ba/html5/thumbnails/17.jpg)
PacketTracing(PT) performance
• Random routing
• Performanceoftracingpathswithdifferentlengths
#hops 2 4 6 8 10
% oftestdata 10.6% 13.2% 57.9% 16.2% 2.1%
Timetaken(ms) 0.626 1.536 2.828 3.532 5.001
![Page 18: MED: The Monitor-Emulator-Debugger for Software-Defined ...weixu/Krvdro9c/...MED functions MED:a useful tool to debug problems in SDN • Create an emulator that can be set to the](https://reader036.fdocuments.net/reader036/viewer/2022071516/6139fab00051793c8c00c8ba/html5/thumbnails/18.jpg)
Realworldbuginswitch software
Pica8switchflowtable:
MEDOVSflowtable:
BuginPicOS-OVS2.3
“AGREportisinjectingARPrequestpacketsbacktothesameport.TheexpectedresultsistoforwardallpacketsexcepttheGREport.”
http://www.pica8.com/document/v2.3/html/release-notes-for-picos-2.3
![Page 19: MED: The Monitor-Emulator-Debugger for Software-Defined ...weixu/Krvdro9c/...MED functions MED:a useful tool to debug problems in SDN • Create an emulator that can be set to the](https://reader036.fdocuments.net/reader036/viewer/2022071516/6139fab00051793c8c00c8ba/html5/thumbnails/19.jpg)
Non-deterministicstatesinthenetworkduetoconcurrentmessages
Controller
• Whichswitchprocessedthemessagefirst?─ Sometimeswedonotknow
─ Canbeok,butcanmeanproblems
![Page 20: MED: The Monitor-Emulator-Debugger for Software-Defined ...weixu/Krvdro9c/...MED functions MED:a useful tool to debug problems in SDN • Create an emulator that can be set to the](https://reader036.fdocuments.net/reader036/viewer/2022071516/6139fab00051793c8c00c8ba/html5/thumbnails/20.jpg)
Raceconditionexample
r:in_port=1->Port2
r:in_port=1->Port3r:in_port=3->Port1
Should we enforce the ordering?
Are we enforcing them correctly?
[1] XinJin,Hongqiang HarryLiu,RohanGandhi,Srikanth Kandula,Ratul Mahajan,MingZhang, JenniferRexford,RogerWattenhofer, DynamicSchedulingofNetworkUpdates, SIGCOMM, 2014
AB
C
![Page 21: MED: The Monitor-Emulator-Debugger for Software-Defined ...weixu/Krvdro9c/...MED functions MED:a useful tool to debug problems in SDN • Create an emulator that can be set to the](https://reader036.fdocuments.net/reader036/viewer/2022071516/6139fab00051793c8c00c8ba/html5/thumbnails/21.jpg)
Racecondition detectorexample(cont’d)
![Page 22: MED: The Monitor-Emulator-Debugger for Software-Defined ...weixu/Krvdro9c/...MED functions MED:a useful tool to debug problems in SDN • Create an emulator that can be set to the](https://reader036.fdocuments.net/reader036/viewer/2022071516/6139fab00051793c8c00c8ba/html5/thumbnails/22.jpg)
Conclusion
• A step bring in the software testing / debugging tools toSDN• Fast, reproducible
• Single step tracingwith packets
• Debugging concurrencyproblems
• Emulates physical network
• Evaluation on an SDN with 20-switches
Wei Xu <[email protected]>
![Page 23: MED: The Monitor-Emulator-Debugger for Software-Defined ...weixu/Krvdro9c/...MED functions MED:a useful tool to debug problems in SDN • Create an emulator that can be set to the](https://reader036.fdocuments.net/reader036/viewer/2022071516/6139fab00051793c8c00c8ba/html5/thumbnails/23.jpg)
Backup slides
![Page 24: MED: The Monitor-Emulator-Debugger for Software-Defined ...weixu/Krvdro9c/...MED functions MED:a useful tool to debug problems in SDN • Create an emulator that can be set to the](https://reader036.fdocuments.net/reader036/viewer/2022071516/6139fab00051793c8c00c8ba/html5/thumbnails/24.jpg)
MEDfunctions
MED: ausefultooltodebugproblemsinSDN
• Createanemulatorthatcanbesettothenetworkstateatanygivenpointoftime
• Tracetheforwardingpathsandtheflowtableentriesusedalongthepath, foreachindividualdatapackets
• CaptureandfindthecauseofcommonSDNproblems:Loop,Reachability failure and RaceConditions
![Page 25: MED: The Monitor-Emulator-Debugger for Software-Defined ...weixu/Krvdro9c/...MED functions MED:a useful tool to debug problems in SDN • Create an emulator that can be set to the](https://reader036.fdocuments.net/reader036/viewer/2022071516/6139fab00051793c8c00c8ba/html5/thumbnails/25.jpg)
Performance:insertingrules