Building Gigabit-rate Routers with the NetFPGA: EuroSys Tutorial at University of Glasgow
-
Upload
gervaise-odin -
Category
Documents
-
view
70 -
download
3
description
Transcript of Building Gigabit-rate Routers with the NetFPGA: EuroSys Tutorial at University of Glasgow
EuroSys 2008 - NetFPGA Tutorial 1 S T A N F O R D U N I V E R S I T Y
Presented by: John W. Lockwood & G. Adam Covington
(Stanford University)Andrew Moore
(University of Cambridge)
Monday March 31, 2008
9am-5pmGlasgow, Scotlandhttp://NetFPGA.org
Building Gigabit-rate Routerswith the NetFPGA: EuroSys Tutorial
at University of Glasgow
EuroSys 2008 - NetFPGA Tutorial 2 S T A N F O R D U N I V E R S I T Y
FPGA
Memory
1GE
1GE
1GE
1GE
What is the NetFPGA?
PCI
CPU Memory
NetFPGA Board
PC with NetFPGA
NetworkingSoftwarerunning on a standard PC
A hardware acceleratorbuilt with Field Programmable Gate Arraydriving Gigabit network links
EuroSys 2008 - NetFPGA Tutorial 3 S T A N F O R D U N I V E R S I T Y
Introduction
Who uses the NetFPGA• Teachers• Students• Researchers
How they use the NetFPGA1. To run the Router Kit2. To build modular reference designs
• IPv4 router• 4-port NIC• Ethernet switch, …
3. To create new networking systems
EuroSys 2008 - NetFPGA Tutorial 4 S T A N F O R D U N I V E R S I T Y
FPGA
Memory
1GE
1GE
1GE
1GE
Running the Router Kit
User-space development, 4x1GE line-rate forwarding
PCI
CPU MemoryOSPF BGP
My Protocoluser
kernelRouting
Table
Usage #1
IPv4Router
1GE
1GE
1GE
1GE
FwdingTable
PacketBuffer
“Mirror”
EuroSys 2008 - NetFPGA Tutorial 5 S T A N F O R D U N I V E R S I T Y
FPGA
Memory
1GE
1GE
1GE
1GE
Building Modular Router Modules
PCI
CPU Memory
Usage #2
NetFPGA Driver
Java GUIFront Panel(Extensible)
PW-OSPF
In QMgmt
IPLookup
L2Parse
L3Parse
Out QMgmt
1GE
1GE
1GE
1GEVerilog modules interconnected by FIFO interfaces
MyBlock
VerilogEDA Tools
(Xilinx, Mentor, etc.)
1. Design2. Simulate3. Synthesize4. Download
EuroSys 2008 - NetFPGA Tutorial 6 S T A N F O R D U N I V E R S I T Y
FPGA
Memory
1GE
1GE
1GE
1GE
Creating new systems
PCI
CPU Memory
Usage #3
NetFPGA Driver
1GE
1GE
1GE
1GE
My Design
(1GE MAC is soft/replaceable)
VerilogEDA Tools
(Xilinx, Mentor, etc.)
1. Design2. Simulate3. Synthesize4. Download
EuroSys 2008 - NetFPGA Tutorial 7 S T A N F O R D U N I V E R S I T Y
Tutorial Outline• Background
– Basics of an IP Router (Andrew)– The NetFPGA Platform (John)
• The Stanford Base Reference Router– Demo1 : Reference Router running on the NetFPGA (Adam)– Inside the NetFPGA hardware (John)– Breakneck introduction to Verilog (John)– Exercise 1: Build your own Reference Router (Adam)
• The Enhanced Reference Router– Motivation: Understanding buffer size requirements in a router (Andrew)– Demo 2: Observing and controlling the queue size (Adam)– Exercise 2: Enhancing the Reference Router (Adam)
• The Life of a Packet Through the NetFPGA– Hardware Datapath (Adam)– Interface to software: Exceptions and Host I/O (John) – Exercise 3: Drop 1 in N Packets (Adam)
• Concluding Remarks– Using NetFPGA for research and teaching (John)– Group Discussion (All)– Survey (You)
EuroSys 2008 - NetFPGA Tutorial 8 S T A N F O R D U N I V E R S I T Y
Basic Operation of an IP RouterR3
A
B
C
R1
R2
R4 D
E
FR5
R5FR3ER3DNext HopDestination
D
EuroSys 2008 - NetFPGA Tutorial 9 S T A N F O R D U N I V E R S I T Y
What does a router do?R3
A
B
C
R1
R2
R4 D
E
FR5
R5FR3ER3DNext HopDestination
D
16 3241
DataOptions (if any)
Destination AddressSource Address
Header ChecksumProtocolTTLFragment OffsetFlagsFragment ID
Total Packet LengthT.ServiceHLenVer
20 b
ytes
EuroSys 2008 - NetFPGA Tutorial 10 S T A N F O R D U N I V E R S I T Y
What does a router do?
A
B
C
R1
R2
R3
R4 D
E
FR5
EuroSys 2008 - NetFPGA Tutorial 11 S T A N F O R D U N I V E R S I T Y
Basic Components of an IP Router
Control Plane
Datapathper-packet processing
SwitchingForwardingTable
Routing Table
Routing Protocols
Management& CLI
Softw
areH
ardware
EuroSys 2008 - NetFPGA Tutorial 12 S T A N F O R D U N I V E R S I T Y
Per-packet processing in an IP Router
1. Accept packet arriving on an incoming link.2. Lookup packet destination address in the
forwarding table, to identify outgoing port(s).
3. Manipulate IP header: e.g., decrement TTL, update header checksum.
5. Buffer packet in the output queue.6. Transmit packet onto outgoing link.
EuroSys 2008 - NetFPGA Tutorial 13 S T A N F O R D U N I V E R S I T Y
Generic Datapath Architecture
LookupIP Address
UpdateHeader
Header ProcessingData Hdr Data Hdr
ForwardingTable
IP Address Next Hop
QueuePacket
BufferMemory
EuroSys 2008 - NetFPGA Tutorial 14 S T A N F O R D U N I V E R S I T Y
CIDR and Longest Prefix Matches The IP address space is broken into line segments. Each line segment is described by a prefix. A prefix is of the form x/y where x indicates the prefix of all
addresses in the line segment, and y indicates the length of the segment.
e.g. The prefix 128.9/16 represents the line segment containing addresses in the range: 128.9.0.0 … 128.9.255.255.
0 232-1
128.9/16
128.9.0.0
216
142.12/1965/8
128.9.16.14
EuroSys 2008 - NetFPGA Tutorial 15 S T A N F O R D U N I V E R S I T Y
Classless Interdomain Routing (CIDR)
0 232-1128.9/16
128.9.16.14
128.9.16/20128.9.176/20
128.9.19/24128.9.25/24
Most specific route = “longest matching prefix”
EuroSys 2008 - NetFPGA Tutorial 16 S T A N F O R D U N I V E R S I T Y
Techniques for LPM in hardware• Linear search• Direct lookup
– Currently requires too much memory– Updating a prefix leads to many changes
• Tries– Deterministic lookup time– Easily pipelined– But requires multiple memories/references
• TCAM (Ternary CAM)– Simple and widely used– But low-density, high-power– Gradually being replaced by new algorithms
EuroSys 2008 - NetFPGA Tutorial 17 S T A N F O R D U N I V E R S I T Y
An IP Router on NetFPGA
SwitchingForwardingTable
Routing Table
Routing Protocols
Management& CLI
Softw
areH
ardware
Linux user-levelprocesses
Verilog on NetFPGA PCI board
ExceptionProcessing
EuroSys 2008 - NetFPGA Tutorial 18 S T A N F O R D U N I V E R S I T Y
NetFPGA Router
Function – 4 Gigabit Ethernet ports
Fully programmable– FPGA hardware
Low cost
Open-source FPGA hardware – Verilog base design
Open-souce Software– Drivers in C and C++
EuroSys 2008 - NetFPGA Tutorial 19 S T A N F O R D U N I V E R S I T Y
NetFPGA PlatformMajor Components
– Interfaces• 4 Gigabit Ethernet Ports• PCI Host Interface
– Memories• 36Mbits Static RAM• 512Mbits DDR2 Dynamic RAM
– FPGA Resources• Block RAMs• Configurable Logic Block (CLBs)• Memory Mapped Registers
EuroSys 2008 - NetFPGA Tutorial 20 S T A N F O R D U N I V E R S I T Y
NetFPGA System
User Space
Linux Kernel
NIC
GE
PCI-ePCI
Browser& VideoClient
MonitorSoftware
GE
GE
GE
GE
GE
CADTools
NetFPGA RouterHardware
VI VI VI VI
Packet Forwarding Table
(eth1 .. 2)(nf2c0 .. 3)
Web &VideoServer
EuroSys 2008 - NetFPGA Tutorial 21 S T A N F O R D U N I V E R S I T Y
NetFPGA Hardware
EuroSys 2008 - NetFPGA Tutorial 22 S T A N F O R D U N I V E R S I T Y
NetFPGA System Implementation• NetFPGA Blocks
– Virtex-2 Pro FPGA– 4.5MB ZBT SRAM– 64MB DDR2 DRAM – PCI Host Interface– 4 Gigabit Ethernet ports
• Intranet Test Ports – Dual Gigabit
Ethernets on PCI-e
• Internet – Gigabit Ethernet
on Motherboard
• Processor – Quad-Core CPU
• Operating System– Linux CentOS 4.4
EuroSys 2008 - NetFPGA Tutorial 23 S T A N F O R D U N I V E R S I T Y
Dell 2950: Another support NetFPGA host
NetFPGA properlyInserts in PCI / PCI-X slot
Dell 2950 with PCI-X and PCI-Express Slots
Thanks: Brian Cashman for providing machine
EuroSys 2008 - NetFPGA Tutorial 24 S T A N F O R D U N I V E R S I T Y
NetFPGA Lab Setup
(eth1 .. 2)
Nf2c3 : Adj. ServerCPU x2 Net-FPGA
Dual NICGEPC
I-ePC
I
Client
NetFPGAControl SW
GE
GE
GE
GE
GECAD Tools
InternetRouter
Hardware
Eth2 : Server
ServerEth1 : Local host
Nf2c1 : Adjacent
Nf2c2 : Local Host
Nf2c0 : Adjacent
EuroSys 2008 - NetFPGA Tutorial 25 S T A N F O R D U N I V E R S I T Y
Net-FPGA
NetFPGA Hardware Set for Demo #1
Net-FPGA
GE
GE
GE
GE
InternetRouter
Hardware
CPU x2
Net-FPGA
NICGE
PCI-e
PCIVideo
Display
GE
GE
GE
GE
GE
CAD Tools
GE
GE
GE
GE
InternetRouter
Hardware
InternetRouter
Hardware
CPU x2
PCI-e
PCI
VideoServer
NICGE
PCI-e GE
…
Server deliversstreaming HD videothrough a chain of NetFPGA Routers
EuroSys 2008 - NetFPGA Tutorial 26 S T A N F O R D U N I V E R S I T Y
Tutorial Outline• Background
– Basics of an IP Router (Andrew)– The NetFPGA Platform (John)
• The Stanford Base Reference Router– Demo1 : Reference Router running on the NetFPGA (Adam)– Inside the NetFPGA hardware (John)– Breakneck introduction to Verilog (John)– Exercise 1: Build your own Reference Router (Adam)
• The Enhanced Reference Router– Motivation: Understanding buffer size requirements in a router (Andrew)– Demo 2: Observing and controlling the queue size (Adam)– Exercise 2: Enhancing the Reference Router (Adam)
• The Life of a Packet Through the NetFPGA– Hardware Datapath– Interface to software– Exercise 3: Drop 1 in N Packets
• Concluding Remarks– Using NetFPGA for research and teaching (John)– Group Discussion– Survey
EuroSys 2008 - NetFPGA Tutorial 27 S T A N F O R D U N I V E R S I T Y
HDDisplay
Topology of NetFPGA Routers
VideoServer
Demo 1
EuroSys 2008 - NetFPGA Tutorial 28 S T A N F O R D U N I V E R S I T Y
Setup for the Reference Router
Each NetFPGA card has four ports
Port 2 connected to Client / Server
Ports 0 and 3 connected to adjacent NetFPGA cards
NetFPGA
Video Client
Demo 1
NetFPGA
NetFPGA
Video Server
EuroSys 2008 - NetFPGA Tutorial 29 S T A N F O R D U N I V E R S I T Y
Subnet Configuration
.1.1
.1.2.3.1
.30.2
.4.1
.4.2
.6.1.3.2
.7.1
.7.2
.9.1.6.2
.10.1
.10.2
.12.1.9.2
.13.1
.13.2
.15.1.12.2
.16.1
.16.2.15.2
.28.1
.28.2 .27.1
.30.1
.25.1
.25.2.24.1
.27.2
.22.1
.22.2.21.1
.24.2
.19.1
.19.2
.17.1
.21.2.18.2
.5.1 .8.1 .11.1 .14.1 .18.1
.20.1.23.1.26.1
.29.1
.2.1
Video ClientShortest Path
Video Server
Demo 1
EuroSys 2008 - NetFPGA Tutorial 30 S T A N F O R D U N I V E R S I T Y
Physical Configuration
• Qnty = N * – NetFPGA
• Mrg Part#: NETFPGA from DigilentInc.com– Ethernet Adapter
• Intel Pro/1000 Dual-port Gigabit PCI-Express x4 NIC• Mfg Part#: EXPI9402PT from Buy.com
– Category 5E Gigabit Ethernet Cables• Loopback (Red, 1’ = 30cm)• Network (Green, 3’ = 1m)• Adjacent Host (Blue, 3’ = 1m)• Configuration (Yellow, 9’ = 3m)
• Additional Parts– RJ45 Cable connectors, Qnty=2
EuroSys 2008 - NetFPGA Tutorial 31 S T A N F O R D U N I V E R S I T Y
3210
12
nf2c
eth
12
3210
12
nf2c
eth
11
3210
12
nf2c
eth
9
3210
12
nf2c
eth
8
3210
12
nf2c
eth
7
3210
12
nf2c
eth
6
3210
12
nf2c
eth
5
3210
12
nf2c
eth
4
3210
12
nf2c
eth
3
3210
12
nf2c
eth
2
Cable Configuration• NetFPGA Gigabit Ethernet Interfaces
– nf2c3 : Left neighbor in network – nf2c2 : Local host interface– nf2c1 : Direct connection to adjacent server– nf2c0 : Right neighbor in network
• Host Ethernet Interfaces – eth1 : Local host interface– eth2 : Direct connection for adjacent client
EuroSys 2008 - NetFPGA Tutorial 32 S T A N F O R D U N I V E R S I T Y
Photos of Machines During Setup
EuroSys 2008 - NetFPGA Tutorial 33 S T A N F O R D U N I V E R S I T Y
Working IP Router
• Objectives – Become familiar with
Stanford Reference Router
– Observe PW-OSPF re-routing traffic around a failure
Demo 1
EuroSys 2008 - NetFPGA Tutorial 34 S T A N F O R D U N I V E R S I T Y
Streaming Video through the NetFPGA• Video server
– Source files/var/www/html/video
– Network URL : http://192.168.Net.Host/video
• Video client– Windows Media Player– Linux mplayer
• Video traffic– MPEG2 HDTV (35 Mbps)– MPEG2 TV (9 Mbps)– DVI (3 Mbps)– WMF (1.7 Mbps)
Demo 1
EuroSys 2008 - NetFPGA Tutorial 35 S T A N F O R D U N I V E R S I T Y
Step 1 – Observe the Routing TablesThe router is already
configured and running on your machines
The routing table has converged to the routing decisions with minimum number of hops
Next, break a link …
Demo 1
EuroSys 2008 - NetFPGA Tutorial 36 S T A N F O R D U N I V E R S I T Y
Step 2 - Dynamic Re-routingBreak the link
between video server and video client
Routers re-route traffic around the broken link and video continues playing
.1.1
.1.2.3.1
.30.2
.4.1
.4.2
.6.1.3.2
.7.1
.7.2
.9.1
.6.2
.10.1
.10.2
.12.1
.9.2
.13.1
.13.2
.15.1
.12.2
.16.1
.16.2.15.2
.28.1
.28.2.27.1
.30.1
.25.1
.25.2.24.1
.27.2
.22.1
.22.2.21.1
.24.2
.19.1
.19.2
.17.1
.21.2.18.2
.5.1 .8.1 .11.1 .14.1
.18.1
.20.1
.23.1.26.1
.29.1
.2.1
Demo 1
EuroSys 2008 - NetFPGA Tutorial 37 S T A N F O R D U N I V E R S I T Y
Tutorial Outline• Background
– Basics of an IP Router (Andrew)– The NetFPGA Platform (John)
• The Stanford Base Reference Router– Demo1 : Reference Router running on the NetFPGA (Adam)– Inside the NetFPGA hardware (John)– Breakneck introduction to Verilog (John)– Exercise 1: Build your own Reference Router (Adam)
• The Enhanced Reference Router– Motivation: Understanding buffer size requirements in a router (Andrew)– Demo 2: Observing and controlling the queue size (Adam)– Exercise 2: Enhancing the Reference Router (Adam)
• The Life of a Packet Through the NetFPGA– Hardware Datapath (Adam)– Interface to software: Exceptions and Host I/O (John) – Exercise 3: Drop 1 in N Packets (Adam)
• Concluding Remarks– Using NetFPGA for research and teaching (John)– Group Discussion (All)– Survey (You)
EuroSys 2008 - NetFPGA Tutorial 38 S T A N F O R D U N I V E R S I T Y
Integrated Circuit TechnologyFull-custom Design
– Complementary Metal Oxide Semiconductor (CMOS)
Semi-custom ASIC Design – Gate array– Standard cell
Programmable Logic Device– Programmable Array Logic– Field Programmable Gate Arrays
Processors
EuroSys 2008 - NetFPGA Tutorial 39 S T A N F O R D U N I V E R S I T Y
Combinatorial Logic
AB
CD
Z
Look-Up TablesCombinatorial logic is stored
in Look-Up Tables (LUTs) – Also called
Function Generators (FGs)– Capacity is limited only by
number of inputs, not complexity– Delay through the LUT is constant
A B C D Z0 0 0 0 00 0 0 1 00 0 1 0 00 0 1 1 10 1 0 0 10 1 0 1 1
. . .1 1 0 0 01 1 0 1 01 1 1 0 01 1 1 1 1
Diagram From: Xilinx, Inc
EuroSys 2008 - NetFPGA Tutorial 40 S T A N F O R D U N I V E R S I T Y
Slice 0
LUT Carry
LUT Carry D QCE
PRE
CLR
DQCE
PRE
CLR
Xilinx CLB StructureEach slice has four outputs
– Two registered outputs, two non-registered outputs
– Two BUFTs associated with each CLB, accessible by all 16 CLB outputs
Carry logic run vertically – Signals run upwards– Two independent
carry chains per CLB
Diagram From: Xilinx, Inc (Courtesy Jeff Weintraub)
EuroSys 2008 - NetFPGA Tutorial 41 S T A N F O R D U N I V E R S I T Y
Field Programmable Gate ArraysCLB
– Primitive element of FPGA
Routing Module– Global routing– Local interconnect
Macro Blocks– Block Memories– Microprocessor
I/O Block
4 LU T
G4G3G2G1
G
4 LU T
F4F3F2F1
F
3 LUT
H
S
R
D Q
S
R
D Q
H1
Din Clk
YQ
Y
XQ
X
M
M
M
M
CLB
GRMLocal R outing
CLB PIP
...
...
...
... ...
......
3rd Generation LUT-based FPGA
Pad Routing CLB M atrix I/O
M acroBlock(uP,M em)
EuroSys 2008 - NetFPGA Tutorial 42 S T A N F O R D U N I V E R S I T Y
NetFPGA Block Diagram
NetFPGA platform
1GE
M
AC
1GE
M
AC
1GE
M
AC
1GE
M
AC
Virtex II-Pro 50 FPGA withuser-defined network logic* Hardware specified with
- Verilog source code- Pre-generated cores
* Software written for - Embedded PowerPCs - Soft core processors
(Microblaze, LEON ..)
18Mb
SRAM
1GE
PH
Y1G
E
PHY
1GE
PH
Y1G
E
PHY
FPGA w/provided infrastructure64M
BD
DR
2SD
RAM
Linux OS - NetFPGA Kernel driverHostcomputer
User-defined software networking applications
Four Gigabit E
thernet Interfaces
18Mb
SRAM
FIFOpacketbuffers
Control, PCIInterface
3 Gb
SATA
Board-B
oard Interconnect
EuroSys 2008 - NetFPGA Tutorial 43 S T A N F O R D U N I V E R S I T Y
Details of NetFPGA
• Fits into Standard PCI slot– Standard Bus : 32 bits, 33 MHz
• Provides Interfaces for processing network packets– 4 Gigabit Ethernet Ports
• Allows hardware-accelerated processing – Implemented with Field Programmable Gate Array (FPGA) Logic
EuroSys 2008 - NetFPGA Tutorial 44 S T A N F O R D U N I V E R S I T Y
Tutorial Outline• Background
– Basics of an IP Router (Andrew)– The NetFPGA Platform (John)
• The Stanford Base Reference Router– Demo1 : Reference Router running on the NetFPGA (Adam)– Inside the NetFPGA hardware (John)– Breakneck introduction to Verilog (John)– Exercise 1: Build your own Reference Router (Adam)
• The Enhanced Reference Router– Motivation: Understanding buffer size requirements in a router (Andrew)– Demo 2: Observing and controlling the queue size (Adam)– Exercise 2: Enhancing the Reference Router (Adam)
• The Life of a Packet Through the NetFPGA– Hardware Datapath (Adam)– Interface to software: Exceptions and Host I/O (John) – Exercise 3: Drop 1 in N Packets (Adam)
• Concluding Remarks– Using NetFPGA for research and teaching (John)– Group Discussion (All)– Survey (You)
EuroSys 2008 - NetFPGA Tutorial 45 S T A N F O R D U N I V E R S I T Y
Hardware Description Languages• Concurrent
– By Default, Verilog statements evaluated concurrently
• Express fine grain parallelism– Allows gate-level parallelism
• Provides Precise Description– Eliminates ambiguity about operation
• Synthesizable– Generates hardware from description
EuroSys 2008 - NetFPGA Tutorial 46 S T A N F O R D U N I V E R S I T Y
Verilog Data Typesreg [7:0] A; // 8-bit register, MSB to LSB
// (Preferred bit order for NetFPGA)reg [0:15] B; // 16-bit register, LSB to MSB
B = {A[7:0],A[0:7]}; // Assignment of bits
reg [31:0] Mem [0:1023]; // 1K Word Memory
integer Count; // simple signed 32-bit integerinteger K[1:64]; // an array of 64 integerstime Start, Stop; // Two 64-bit time variables
From: CSCI 320 Computer ArchitectureHandbook on Verilog HDL, by Dr. Daniel C. Hyde :
http://eesun.free.fr/DOC/VERILOG/verilog-manual.html
EuroSys 2008 - NetFPGA Tutorial 47 S T A N F O R D U N I V E R S I T Y
Signal Multiplexers
From: http://eesun.free.fr/DOC/VERILOG/synvlg.html
Two input multiplexer (using if / else) reg y; always @*
if (select) y = a; else y = b;
Two input multiplexer (using ternary operator ?:)
wire t = (select ? a : b);
EuroSys 2008 - NetFPGA Tutorial 48 S T A N F O R D U N I V E R S I T Y
Larger MultiplexersThree input multiplexer
reg s; always @* begin case (select2)
2'b00: s = a; 2'b01: s = b; default: s = c; endcase end
From: http://eesun.free.fr/DOC/VERILOG/synvlg.html
EuroSys 2008 - NetFPGA Tutorial 49 S T A N F O R D U N I V E R S I T Y
Synchronous Storage Elements• Values change at
times governed by clock
Clock Transition
t=0 t=1 t=201Clock
time
Clock Transition
S0Doutt=0
A B
A B CDint=0
Clock
Din DoutQD
– Clock• Input to circuit
– Clock Event• Example: Rising edge
– Flip/Flop• Transfers Value From Din
to Dout on Clock event
EuroSys 2008 - NetFPGA Tutorial 50 S T A N F O R D U N I V E R S I T Y
Finite State Machines
Copyrigh t 2001, John W. Lockwood, All Rights Reserved
Combinational Logic
Inputs (X)
S(t) S(t+1)=(X,S(t))
Outputs (Z)
StateNext
State Storage
...
[Moore](S(t))
[Mealy](X,S(t))
-or-
Q D
Q D
EuroSys 2008 - NetFPGA Tutorial 51 S T A N F O R D U N I V E R S I T Y
Synthesizable Verilog : Delay Flip/Flops
From: http://eesun.free.fr/DOC/VERILOG/synvlg.html
D type flip flop with data enablereg q; always @ (posedge clk)
if (enable) q <= d;
D-type flip flop reg q; always @ (posedge clk)
q <= d;
EuroSys 2008 - NetFPGA Tutorial 52 S T A N F O R D U N I V E R S I T Y
Tutorial Outline• Background
– Basics of an IP Router (Andrew)– The NetFPGA Platform (John)
• The Stanford Base Reference Router– Demo1 : Reference Router running on the NetFPGA (Adam)– Inside the NetFPGA hardware (John)– Breakneck introduction to Verilog (John)– Exercise 1: Build your own Reference Router (Adam)
• The Enhanced Reference Router– Motivation: Understanding buffer size requirements in a router (Andrew)– Demo 2: Observing and controlling the queue size (Adam)– Exercise 2: Enhancing the Reference Router (Adam)
• The Life of a Packet Through the NetFPGA– Hardware Datapath (Adam)– Interface to software: Exceptions and Host I/O (John) – Exercise 3: Drop 1 in N Packets (Adam)
• Concluding Remarks– Using NetFPGA for research and teaching (John)– Group Discussion (All)– Survey (You)
Eurosys 2008 - NetFPGA Tutorial 53 S T A N F O R D U N I V E R S I T Y
Reference Router Pipeline• Five stages
– Input– Input Arbitration– Routing Decision and
packet modification– Output Queuing– Output
• Packet-based module interface
• Pluggable design
Exerci
se 1
MACRxQ
CPURxQ
MACRxQ
CPURxQ
MACRxQ
CPURxQ
MACRxQ
CPURxQ
Input Arbiter
Output Port Lookup
MACTxQ
CPUTxQ
MACTxQ
CPUTxQ
MACTxQ
CPUTxQ
MACTxQ
CPUTxQ
Output Queues
EuroSys 2008 - NetFPGA Tutorial 54 S T A N F O R D U N I V E R S I T Y
Make your own router
Objectives: – Learn how to build hardware– Run the software– Explore router architecture
Execution– Start synthesis– Rerun the GUI with the new hardware– Test connectivity and statistics with pings– Explore pipeline in the details page– Explore detailed statistics in the details page
Exerci
se 1
Eurosys 2008 - NetFPGA Tutorial 55 S T A N F O R D U N I V E R S I T Y
Step 1 - Build the hardware
Start terminal, cd to “NF2/projects/tutorial_router/synth”
Run “make clean”
Start synthesis with “make”
Exerci
se 1
EuroSys 2008 - NetFPGA Tutorial 56 S T A N F O R D U N I V E R S I T Y
First Breakon 5th Floor
(while hardware compiles)
Eurosys 2008 - NetFPGA Tutorial 57 S T A N F O R D U N I V E R S I T Y
Step 2 - Run Homemade Router
cd to “NF2/projects/tutorial_router/sw”
Type: “tutorial_router_gui.pl” to
use the just built router hardware
The same interface should start again
Exerci
se 1
Eurosys 2008 - NetFPGA Tutorial 58 S T A N F O R D U N I V E R S I T Y
Step 4 - Connectivity and StatisticsPing any addresses
192.168.x.y where x is from 1-20 and y is 1 or 2
Open the statistics tab in the Quickstart window to see some statistics
Explore more statistics in modules under the details tab
Exerci
se 1
EuroSys 2008 - NetFPGA Tutorial 59 S T A N F O R D U N I V E R S I T Y
Step 5 - Explore Router Architecture
Click the Details tab of the Quickstart window
This is the reference router pipeline – a canonical, simple to understand, modular router pipeline
Exerci
se 1
Eurosys 2008 - NetFPGA Tutorial 60 S T A N F O R D U N I V E R S I T Y
Step 6 - Explore Output QueuesClick on the Output
Queues module in the Details tab
The page gives configuration details
…and statistics
Exerci
se 1
EuroSys 2008 - NetFPGA Tutorial 61 S T A N F O R D U N I V E R S I T Y
Tutorial Outline• Background
– Basics of an IP Router (Andrew)– The NetFPGA Platform (John)
• The Stanford Base Reference Router– Demo1 : Reference Router running on the NetFPGA (Adam)– Inside the NetFPGA hardware (John)– Breakneck introduction to Verilog (John)– Exercise 1: Build your own Reference Router (Adam)
• The Enhanced Reference Router– Motivation: Understanding buffer size requirements in a router (Andrew)– Demo 2: Observing and controlling the queue size (Adam)– Exercise 2: Enhancing the Reference Router (Adam)
• The Life of a Packet Through the NetFPGA– Hardware Datapath (Adam)– Interface to software: Exceptions and Host I/O (John) – Exercise 3: Drop 1 in N Packets (Adam)
• Concluding Remarks– Using NetFPGA for research and teaching (John)– Group Discussion (All)– Survey (You)
Eurosys 2008 - NetFPGA Tutorial 62 S T A N F O R D U N I V E R S I T Y
Buffer Requirements in a RouterBuffer size matters:
– Small queues reduce delay– Large buffers are expensive
Theoretical tools predict requirements– Queuing theory– Large deviation theory– Mean field theory
Yet, there is no direct answer.– Flows have a closed-loop nature– Question arises on whether focus should be on
equilibrium state or transient state..
Eurosys 2008 - NetFPGA Tutorial 63 S T A N F O R D U N I V E R S I T Y
• Universally applied rule-of-thumb:– A router needs a buffer size:– 2T is the two-way propagation delay (or just 250ms)– C is capacity of bottleneck link
• Context– Mandated in backbone and edge routers.– Appears in RFPs and IETF architectural guidelines.– Already known by inventors of TCP
• [Van Jacobson, 1988]– Has major consequences for router design
CTB 2
CRouterSource Destination
2T
Rule-of-thumb
Eurosys 2008 - NetFPGA Tutorial 64 S T A N F O R D U N I V E R S I T Y
The Story So Far
)(logO22 )2()1( Wn
CTCT
10,000 20# packetsat 10Gb/s 1,000,000
(1) Assume: Large number of desynchronized flows; 100% utilization(2) Assume: Large number of desynchronized flows; <100% utilization
Eurosys 2008 - NetFPGA Tutorial 65 S T A N F O R D U N I V E R S I T Y
Using NetFPGA to explore buffer size• Need to reduce buffer size and measure
occupancy• Alas, not possible in commercial routers• So, we will use NetFPGA instead
Objective:– Use NetFPGA to understand how large a buffer
we need for a single TCP flow.
Eurosys 2008 - NetFPGA Tutorial 66 S T A N F O R D U N I V E R S I T Y
Rule for adjusting W– If an ACK is received: W ← W+1/W– If a packet is lost: W ← W/2
Why 2TxC for a single TCP Flow?
Only W packets may be outstanding
Eurosys 2008 - NetFPGA Tutorial 67 S T A N F O R D U N I V E R S I T Y
Time evolution of a single TCP flow through a router. Buffer is < 2T*C
Time Evolution of a Single TCP Flow
Time evolution of a single TCP flow through a router. Buffer is 2T*C
EuroSys 2008 - NetFPGA Tutorial 68 S T A N F O R D U N I V E R S I T Y
NetFPGA Hardware Set for Demo #2
CPU x2PC
I-eVideoServer
NICGE
PCI-e GE
Net-FPGA
CPU x2
NICGE
PCI-e
PCIVideo
Client
GE
GE
GE
GE
GE
InternetRouter
Hardware
…
Server deliversstreaming HD videoto adjacent client
EuroSys 2008 - NetFPGA Tutorial 69 S T A N F O R D U N I V E R S I T Y
Tutorial Outline• Background
– Basics of an IP Router (Andrew)– The NetFPGA Platform (John)
• The Stanford Base Reference Router– Demo1 : Reference Router running on the NetFPGA (Adam)– Inside the NetFPGA hardware (John)– Breakneck introduction to Verilog (John)– Exercise 1: Build your own Reference Router (Adam)
• The Enhanced Reference Router– Motivation: Understanding buffer size requirements in a router (Andrew)– Demo 2: Observing and controlling the queue size (Adam)– Exercise 2: Enhancing the Reference Router (Adam)
• The Life of a Packet Through the NetFPGA– Hardware Datapath (Adam)– Interface to software: Exceptions and Host I/O (John) – Exercise 3: Drop 1 in N Packets (Adam)
• Concluding Remarks– Using NetFPGA for research and teaching (John)– Group Discussion (All)– Survey (You)
EuroSys 2008 - NetFPGA Tutorial 70 S T A N F O R D U N I V E R S I T Y
Setup for the Demo 2
Each NetFPGA card has four ports
Port 2 connected to Local Host
Port 3 connected to adjacent Server
LocalHost
Demo 2
NetFPGA
AdjacentServer
Eurosys 2008 - NetFPGA Tutorial 71 S T A N F O R D U N I V E R S I T Y
Topology for Second DemonstrationRouters connected point-to-point topologyPort 3 connects to local hostPort 1 connects to adjacent neighborPorts 0 and 2 unused
.1.1 .1.2
.4.1
.4.2
.7.1
.7.2
.10.1
.10.2
.13.1
.13.2
.16.1.16.2
.28.1
.28.2
.25.1
.25.2
.22.1
.22.2
.19.1
.19.2
Demo 2
.2.2
.2.1
.5.2
.5.1
.8.2
.8.1
.11.2
.11.1
.14.2
.14.1.17.2
.17.1
.20.2
.20.1
.23.2
.23.1
.26.2
.26.1
.29.2
.29.1
EuroSys 2008 - NetFPGA Tutorial 72 S T A N F O R D U N I V E R S I T Y
Enhanced Router
Objectives – Observe router with new modules– New modules: rate limiting, delay, event capture
Execution– Run event capture router– Look at routing tables– Explore details pane– Start tcp transfer, look at queue occupancy– Change rate/delay, look at queue occupancy
Demo 2
Eurosys 2008 - NetFPGA Tutorial 73 S T A N F O R D U N I V E R S I T Y
Step 1 - Run Pre-made Enhanced Router
Start terminal and cd to “NF2/projects/tutorial_router/sw/”
Type “./tut_adv_router_gui.pl”
A familiar GUI should start
Demo 2
Eurosys 2008 - NetFPGA Tutorial 74 S T A N F O R D U N I V E R S I T Y
Step 3 - Explore Enhanced Router
Click on the Details tab
A similar Pipeline to the one seen previously shows with some additions
Demo 2
Eurosys 2008 - NetFPGA Tutorial 75 S T A N F O R D U N I V E R S I T Y
Enhanced Router PipelineTwo modules added
1. Event Capture to capture output queue events (writes, reads, drops)
2. Rate Limiter to create a bottleneck
Demo 2
MACRxQ
CPURxQ
MACRxQ
CPURxQ
MACRxQ
CPURxQ
MACRxQ
CPURxQ
Input Arbiter
Output Port Lookup
MACTxQ
CPUTxQ
MACTxQ
CPUTxQ
MACTxQ
CPUTxQ
MACTxQ
CPUTxQ
Output Queues
RateLimiter
Event Capture
Eurosys 2008 - NetFPGA Tutorial 76 S T A N F O R D U N I V E R S I T Y
Step 4 - Decrease the Link RateTo create bottleneck and
show the TCP “sawtooth”, link-rate is decreased.
In the Details tab click the “Rate Limit” module
Check Enable
Set link rate to 1.953Mbps
Demo 2
Eurosys 2008 - NetFPGA Tutorial 77 S T A N F O R D U N I V E R S I T Y
Step 5 – Decrease Queue Size
Go back to the Details Panel and click on “Output Queues”.
Select the “Output Queue 2” tab.
Change the output queues size in packets slider to 16
Demo 2
Eurosys 2008 - NetFPGA Tutorial 78 S T A N F O R D U N I V E R S I T Y
Step 6 - Start Event Capture
Click on the Event Capture module under the Details tab
This should start the configuration page
Demo 2
Eurosys 2008 - NetFPGA Tutorial 79 S T A N F O R D U N I V E R S I T Y
Step 7 - Configure Event Capture
Check Send to local host to receive events on the local host
Check Monitor Queue 2 to monitor output queue of MAC port1
Check “Enable Capture” to start Event capture
Demo 2
Eurosys 2008 - NetFPGA Tutorial 80 S T A N F O R D U N I V E R S I T Y
Step 8 - Start TCP Transfer
We will use iperf to run a large TCP transfer and look at queue evolution
Start a terminal and cd to“NF2/projects/tutorial_router/sw”
type “iperf.sh”
Demo 2
Eurosys 2008 - NetFPGA Tutorial 81 S T A N F O R D U N I V E R S I T Y
Step 9 - Look at Event Capture Results
Click on the Event Capture module under the details tab.
The sawtooth pattern should now be visible.
Demo 2
Eurosys 2008 - NetFPGA Tutorial 82 S T A N F O R D U N I V E R S I T Y
Queue Occupancy Charts
EuroSys 2008 - NetFPGA Tutorial 83 S T A N F O R D U N I V E R S I T Y
Tutorial Outline• Background
– Basics of an IP Router (Andrew)– The NetFPGA Platform (John)
• The Stanford Base Reference Router– Demo1 : Reference Router running on the NetFPGA (Adam)– Inside the NetFPGA hardware (John)– Breakneck introduction to Verilog (John)– Exercise 1: Build your own Reference Router (Adam)
• The Enhanced Reference Router– Motivation: Understanding buffer size requirements in a router (Andrew)– Demo 2: Observing and controlling the queue size (Adam)– Exercise 2: Enhancing the Reference Router (Adam)
• The Life of a Packet Through the NetFPGA– Hardware Datapath (Adam)– Interface to software: Exceptions and Host I/O (John) – Exercise 3: Drop 1 in N Packets (Adam)
• Concluding Remarks– Using NetFPGA for research and teaching (John)– Group Discussion (All)– Survey (You)
EuroSys 2008 - NetFPGA Tutorial 84 S T A N F O R D U N I V E R S I T Y
Enhance Your Router
Objectives – Add new modules to datapath– Synthesize and test router
Execution– Open user_datapath.v, uncomment
delay/rate/event capture modules– Synthesize– After synthesis, test the new system.
Exerci
se 2
EuroSys 2008 - NetFPGA Tutorial 85 S T A N F O R D U N I V E R S I T Y
An aside: xemacs TipsWe will be modifying the Verilog source codeSlides show xemacs, but vim also available.xemacs:
– To undo, use ctrl+shift+'-'– To cancel a multi-keystroke command, just type ctrl+g– To select lines, hold shift and press the arrow keys.– To comment some selected lines, type ctrl+c+c– To uncomment a commented block, move the cursor to one
of the lines inside the commented block and type ctrl+c+u– To save type ctrl+x+s– To search, type ctrl+s search_pattern
Eurosys 2008 - NetFPGA Tutorial 86 S T A N F O R D U N I V E R S I T Y
Step 1 - Open the SourceWe will modify the Verilog
source code to add event capture, rate limiter, and delay modules
We will simply comment and uncomment existing code
Open terminalType “xemacs
NF2/projects/tutorial_router/src/user_data_path.v
Exerci
se 2
Eurosys 2008 - NetFPGA Tutorial 87 S T A N F O R D U N I V E R S I T Y
Step 2 - Add wires
Now we need to add wires to connect the new modules
Search for “new wires” (ctrl+s new wires) then press Enter
Uncomment the wires (ctrl+c+u)
Exerci
se 2
Eurosys 2008 - NetFPGA Tutorial 88 S T A N F O R D U N I V E R S I T Y
Step 3a - Connect Event Capture
Search for opl_output (ctrl+s opl_output) then press Enter
Comment the four lines above (up, shift + up + up + up + up, ctrl+c+c)
Uncomment the block below to connect the outputs (ctrl+s opl_out, ctrl+c+u)
Exerci
se 2
Eurosys 2008 - NetFPGA Tutorial 89 S T A N F O R D U N I V E R S I T Y
Step 3b - Connect the Output Queue Registers
Search for opl_output (ctrl+s opl_output, Enter)
Comment the 6 lines (select the six lines by using shift+arrow keys, then type ctrl+c+c)
Uncomment the commented block by scrolling down into the block and typing ctrl+c+u
Exerci
se 2
Eurosys 2008 - NetFPGA Tutorial 90 S T A N F O R D U N I V E R S I T Y
Step 4 - Add the Event Capture Module
Search for evt_capture_top (ctrl+s evt_capture_top) then press Enter
Uncomment the block (ctrl+c+u)
Exerci
se 2
Eurosys 2008 - NetFPGA Tutorial 91 S T A N F O R D U N I V E R S I T Y
Step 5 - Add the Drop Nth Module
Search for drop_nth_packet (ctrl+s drop_nth_packet) then press Enter
Uncomment the block (ctrl+c+u)
Exerci
se 2
Eurosys 2008 - NetFPGA Tutorial 92 S T A N F O R D U N I V E R S I T Y
Step 6 - Connect the Output Queue to the Rate Limiter
Search for port_outputs (ctrl+s port_outputs, Enter)
Comment the 4 lines above (select the four lines by using shift+arrow keys, then type ctrl+c+c)
Uncomment the commented block by scrolling down into the block and typing ctrl+c+u
Exerci
se 2
Eurosys 2008 - NetFPGA Tutorial 93 S T A N F O R D U N I V E R S I T Y
Step 7 - Connect the Registers
Search for port_outputs (ctrl+s port_outputs, Enter)
Comment the 6 lines (select the six lines by using shift+arrow keys, then type ctrl+c+c)
Uncomment the commented block by scrolling down into the block and typing ctrl+c+u
Exerci
se 2
six
Eurosys 2008 - NetFPGA Tutorial 94 S T A N F O R D U N I V E R S I T Y
Step 8 - Add Rate Limiter
Scroll down until you reach the next “Excluded” block
Uncomment the block containing the rate limiter instantiations. (scroll into the block and type ctrl+c+u)
Save (ctrl+x+s)
Exerci
se 2
Eurosys 2008 - NetFPGA Tutorial 95 S T A N F O R D U N I V E R S I T Y
Step 9 - Build the hardware
Start terminal, cd to “NF2/projects/tutorial_router/synth”
Run “make clean”
Start synthesis with “make”
Exerci
se 2
EuroSys 2008 - NetFPGA Tutorial 96 S T A N F O R D U N I V E R S I T Y
Second BreakLunch on 5th Floor
(while hardware compiles)
EuroSys 2008 - NetFPGA Tutorial 97 S T A N F O R D U N I V E R S I T Y
Tutorial Outline• Background
– Basics of an IP Router (Andrew)– The NetFPGA Platform (John)
• The Stanford Base Reference Router– Demo1 : Reference Router running on the NetFPGA (Adam)– Inside the NetFPGA hardware (John)– Breakneck introduction to Verilog (John)– Exercise 1: Build your own Reference Router (Adam)
• The Enhanced Reference Router– Motivation: Understanding buffer size requirements in a router (Andrew)– Demo 2: Observing and controlling the queue size (Adam)– Exercise 2: Enhancing the Reference Router (Adam)
• The Life of a Packet Through the NetFPGA– Hardware Datapath (Adam)– Interface to software: Exceptions and Host I/O (John) – Exercise 3: Drop 1 in N Packets (Adam)
• Concluding Remarks– Using NetFPGA for research and teaching (John)– Group Discussion (All)– Survey (You)
EuroSys 2008 - NetFPGA Tutorial 98 S T A N F O R D U N I V E R S I T Y
Full System Components
Software
PCI Bus
NetFPGA
CPURxQ
CPUTxQ
nf2_reg_grp
user data path
nf2c0 nf2c1 nf2c2 nf2c3 ioctl
MACTxQ
MACRxQ
Ethernet
CPURxQ
CPUTxQ
CPURxQ
CPUTxQ
CPURxQ
CPUTxQ
MACTxQ
MACRxQ
MACTxQ
MACRxQ
MACTxQ
MACRxQ
EuroSys 2008 - NetFPGA Tutorial 99 S T A N F O R D U N I V E R S I T Y
port0 port2 192.168.2.y192.168.1.x
Life of a Packet through the hardware
EuroSys 2008 - NetFPGA Tutorial 100 S T A N F O R D U N I V E R S I T Y
Router Stages Again
MACRxQ
CPURxQ
MACRxQ
CPURxQ
MACRxQ
CPURxQ
MACRxQ
CPURxQ
Input Arbiter
Output Port Lookup
MACTxQ
CPUTxQ
MACTxQ
CPUTxQ
MACTxQ
CPUTxQ
MACTxQ
CPUTxQ
Output Queues
EuroSys 2008 - NetFPGA Tutorial 101 S T A N F O R D U N I V E R S I T Y
Inter-module Communication
Using “Module Headers”:
IP HdrEth Hdr
…00
0Last word of packet0x10
Last Module Hdry……
Module Hdrx Contain information such as packet length, input port, output port, …
Data Word(64 bits)
Ctrl Word(8 bits)
EuroSys 2008 - NetFPGA Tutorial 102 S T A N F O R D U N I V E R S I T Y
data
Inter-module Communication
ctrlwr
rdy
EuroSys 2008 - NetFPGA Tutorial 103 S T A N F O R D U N I V E R S I T Y
MAC Rx Queue
EuroSys 2008 - NetFPGA Tutorial 104 S T A N F O R D U N I V E R S I T Y
Rx Queue
IP Hdr:IP Dst: 192.168.2.3,
TTL: 64, Csum:0x3ab4
Eth Hdr:Dst MAC = port 0,
Ethertype = IP
Data
0
0
0
Pkt length,input port = 00xff
EuroSys 2008 - NetFPGA Tutorial 105 S T A N F O R D U N I V E R S I T Y
Input Arbiter
Pkt
Pkt
Pkt
EuroSys 2008 - NetFPGA Tutorial 106 S T A N F O R D U N I V E R S I T Y
Output Port Lookup
EuroSys 2008 - NetFPGA Tutorial 107 S T A N F O R D U N I V E R S I T Y
IP Hdr:IP Dst: 192.168.2.3,
TTL: 64, Csum:0x3ab4
IP Hdr:IP Dst: 192.168.2.3,
TTL: 63, Csum:0x3ac2
Output Port Lookup
EthHdr: Dst MAC = 0Src MAC = x,
Ethertype = IP
Data
0
0
0
Pkt length,input port = 00xff
output port = 40x04
1- Check input port matches
Dst MAC
2- Check TTL, checksum
3- Lookup next hop IP & output port
(LPM)
4- Lookup next hop MAC address (ARP)
5- Add output port module
6- Modify MAC Dst and Src addresses
7-Decrement TTL and update
checksum
EthHdr: Dst MAC = nextHop Src MAC = port 4,
Ethertype = IP
EuroSys 2008 - NetFPGA Tutorial 108 S T A N F O R D U N I V E R S I T Y
Output Queues
OQ0
OQ4
OQ7
EuroSys 2008 - NetFPGA Tutorial 109 S T A N F O R D U N I V E R S I T Y
MAC Tx Queue
EuroSys 2008 - NetFPGA Tutorial 110 S T A N F O R D U N I V E R S I T Y
MAC Tx Queue
IP Hdr:IP Dst: 192.168.2.3,
TTL: 64, Csum:0x3ab4
IP Hdr:IP Dst: 192.168.2.3,
TTL: 63, Csum:0x3ac2
EthHdr: Dst MAC = nextHop Src MAC = port 4,
Ethertype = IP
Data
0
0
0
Pkt length,input port = 00xff
output port = 40x04
EuroSys 2008 - NetFPGA Tutorial 111 S T A N F O R D U N I V E R S I T Y
Exception Packet• Example: TTL = 0 or TTL = 1• Packet has to be sent to the CPU which will
generate an ICMP packet as a response• Difference starts at the Output Port lookup
stage
EuroSys 2008 - NetFPGA Tutorial 112 S T A N F O R D U N I V E R S I T Y
Exception Packet Path
Software
PCI Bus
NetFPGA
CPURxQ
CPUTxQ
CPURxQ
CPUTxQ
CPURxQ
CPUTxQ
CPURxQ
CPUTxQ
nf2_reg_grp
user data path
nf2c0 nf2c1 nf2c2 nf2c3 ioctl
MACTxQ
MACRxQ
MACTxQ
MACRxQ
MACTxQ
MACRxQ
MACTxQ
MACRxQ
Ethernet
EuroSys 2008 - NetFPGA Tutorial 113 S T A N F O R D U N I V E R S I T Y
IP Hdr:IP Dst: 192.168.2.3,
TTL: 1, Csum:0x3ab4
Output Port Lookup
EthHdr: Dst MAC = 0,Src MAC = x,
Ethertype = IP
Data
0
0
0
Pkt length,input port = 00xff
output port = 10x04
1- Check input port matches
Dst MAC
2- Check TTL, checksum – EXCEPTION!
3- Add output port module
EuroSys 2008 - NetFPGA Tutorial 114 S T A N F O R D U N I V E R S I T Y
Output Queues
OQ0
OQ1
OQ2
OQ7
EuroSys 2008 - NetFPGA Tutorial 115 S T A N F O R D U N I V E R S I T Y
CPU Tx Queue
EuroSys 2008 - NetFPGA Tutorial 116 S T A N F O R D U N I V E R S I T Y
CPU Tx Queue
IP Hdr:IP Dst: 192.168.2.3,
TTL: 1, Csum:0x3ab4
EthHdr: Dst MAC = 0, Src MAC = x,
Ethertype = IP
Data
0
0
0
Pkt length,input port = 00xff
output port = 10x04
EuroSys 2008 - NetFPGA Tutorial 117 S T A N F O R D U N I V E R S I T Y
ICMP Packet• For the ICMP packet, the packet arrives at
the CPU Rx Queue from the PCI Bus• Follows the same path as a packet from the
MAC until the Output Port Lookup.• The OPL module seeing the packet is from
the CPU Rx Queue 1, sets the output port directly to 0.
• The packet then continues on the same path as the non-exception packet to the Output Queues and then MAC Tx queue 0.
EuroSys 2008 - NetFPGA Tutorial 118 S T A N F O R D U N I V E R S I T Y
ICMP Packet Path
Software
PCI Bus
NetFPGA
CPURxQ
CPUTxQ
CPURxQ
CPUTxQ
CPURxQ
CPUTxQ
CPURxQ
CPUTxQ
nf2_reg_grp
user data path
nf2c0 nf2c1 nf2c2 nf2c3 ioctl
MACTxQ
MACRxQ
MACTxQ
MACRxQ
MACTxQ
MACRxQ
MACTxQ
MACRxQ
Ethernet
EuroSys 2008 - NetFPGA Tutorial 119 S T A N F O R D U N I V E R S I T Y
NetFPGA-Host Interaction• Linux driver interfaces with hardware
– Packet interface via standard Linux network stack
– Register reads/writes via ioctl system call (with convenience wrapper functions)
• readReg(nf2device *dev, int address, unsigned *rd_data)• writeReg(nf2device *dev, int address, unsigned *wr_data)
eg:readReg(&nf2, OQ_NUM_PKTS_STORED_0, &val);
EuroSys 2008 - NetFPGA Tutorial 120 S T A N F O R D U N I V E R S I T Y
NetFPGA-Host Interaction
NetFPGA to host packet transfer
PCI B
us
2. Interrupt notifies driver of packet arrival
3. Driver sets up and initiates DMA transfer
1. Packet arrives – forwarding table sends to CPU queue
EuroSys 2008 - NetFPGA Tutorial 121 S T A N F O R D U N I V E R S I T Y
NetFPGA-Host Interaction
NetFPGA to host packet transfer (cont)
PCI B
us
4. NetFPGA transfers packet via DMA
5. Interrupt signals completion of DMA
6. Driver passes packet to network stack
EuroSys 2008 - NetFPGA Tutorial 122 S T A N F O R D U N I V E R S I T Y
NetFPGA-Host Interaction
Host to NetFPGA packet transfers
PCI B
us
3. Interrupt signals completion of DMA
1. Software sends packet via network sockets. Packet delivered to driver.
2. Driver sets up and initiates DMA transfer
EuroSys 2008 - NetFPGA Tutorial 123 S T A N F O R D U N I V E R S I T Y
NetFPGA-Host Interaction
Register access
PCI B
us1. Software makes ioctl call on network socket. ioctl passed to driver.
2. Driver performs PCI memory read/write
EuroSys 2008 - NetFPGA Tutorial 124 S T A N F O R D U N I V E R S I T Y
NetFPGA-Host Interaction• Packet transfers shown using DMA
interface
• Alternative: use programmed IO to transfer packets via register reads/writes– slower but eliminates the need to deal with
network sockets
EuroSys 2008 - NetFPGA Tutorial 125 S T A N F O R D U N I V E R S I T Y
Step 10 – Perfect the Router
If interested, go back to “Demo 2: Step 1” after synthesis is done and redo the steps with your own router.
You can also change the bandwidth and queue size settings to see how that effects the queue occupancy evolution.
To run your router:1- cd NF2/projects/tutorial_router/sw2- type “./tut_adv_router_gui.pl --use_bin
../../../bitfiles/tutorial_router.bit”
Exerci
se 2
EuroSys 2008 - NetFPGA Tutorial 126 S T A N F O R D U N I V E R S I T Y
Drop 1 in N Packets
Objectives – Add counter and FSM to the code– Synthesize and test router
Execution– Open drop_nth_packet.v – Insert counter code – Synthesize– After synthesis, test the new system.
Exerci
se 3
EuroSys 2008 - NetFPGA Tutorial 127 S T A N F O R D U N I V E R S I T Y
Step 1 - Open the Source
We will modify the Verilogsource code to add acounter to the drop_nth_packet module
Open terminalType “xemacs
NF2/projects/tutorial_router/src/drop_nth_packet.v
Exerci
se 3
EuroSys 2008 - NetFPGA Tutorial 128 S T A N F O R D U N I V E R S I T Y
Step 2 - Add Counter to Module
Add Counter using the following signals:counter – (16 bit signal)rst_counter – resetinc_counter – increment
Search for insert counter (ctrl+s insert counter, Enter)
Insert Counter and Save(ctrl+x+s)
Exerci
se 3
EuroSys 2008 - NetFPGA Tutorial 129 S T A N F O R D U N I V E R S I T Y
Step 3 - Build the hardware
Start terminal, cd to “NF2/projects/tutorial_router/synth”
Run “make clean”
Start synthesis with “make”
Exerci
se 3
EuroSys 2008 - NetFPGA Tutorial 130 S T A N F O R D U N I V E R S I T Y
Third Breakon 5th Floor
(while hardware compiles)
EuroSys 2008 - NetFPGA Tutorial 131 S T A N F O R D U N I V E R S I T Y
Step 4 – Test the RouterYou can watch the number of received and sent packets to watch the
module drop every Nth packet. Ping a local machine (i.e. 192.168.7.1) and watch for missing pings.
To run your router:1- cd NF2/projects/tutorial_router/sw2- type “./tut_adv_router_gui.pl --use_bin ../../../bitfiles/tutorial_router.bit”
To set the value of N (which packet to drop)type “regwrite 0x2000704 0xN” – replace N with a number
To enable packet dropping: To disable packet dropping:type “regwrite 0x2000700 0x1” type “regwrite 0x2000700 0x0”
Exerci
se 3
EuroSys 2008 - NetFPGA Tutorial 132 S T A N F O R D U N I V E R S I T Y
Using the NetFPGA in the Classroom
EuroSys 2008 - NetFPGA Tutorial 133 S T A N F O R D U N I V E R S I T Y
NetFPGA in the Classroom• Stanford University
– EE109 “Build an Ethernet Switch”• Undergraduate Course for all EE students• http://www.stanford.edu/class/ee109/
– CS344 “Building an Internet Router”• Quarter-Long Course • Taught Spr’05, Spr’07, Spr’08• http://cs344.stanford.edu
• Rice University– Nework Systems Architecture
• Spr’08• http://comp519.cs.rice.edu/
EuroSys 2008 - NetFPGA Tutorial 134 S T A N F O R D U N I V E R S I T Y
Components of NetFPGA Course• Documentation
– System Design– Implementation Plan
• Deliverables– Hardware Circuits– System Software – Milestones
• Testing– Proof of Correctness– Integrated Testing– Interoperabilty
• Post Mortem– Lessons Learned
EuroSys 2008 - NetFPGA Tutorial 135 S T A N F O R D U N I V E R S I T Y
NetFPGA in the Classroom• Stanford CS344: “Build an Internet Router”
– Courseware available on-line– Students work in teams of three
• 1-2 software• 1-2 hardware
– Design and implement router in 8 weeks– Write software for CLI and PW-OSPF– Show interoperability with other groups – Add new features in remaining two weeks
• Firewall, NAT, DRR, Packet capture, Data generator, …
EuroSys 2008 - NetFPGA Tutorial 136 S T A N F O R D U N I V E R S I T Y
softwarehardware
SwitchingForwardingTable
Routing Table
Routing Protocols
Management& CLI
ExceptionProcessing
InteroperabilityBuild basic router Routing Protocol(PWOSPF)
Integrate with H/W
Emulatedh/w in VNS
Routing Table
Routing Protocols
Management& CLI
ExceptionProcessing
Emulatedh/w in VNS
Routing Table
Routing Protocols
Management& CLI
ExceptionProcessing
Emulatedh/w in VNS
Routing Table
Routing Protocols
Management& CLI
ExceptionProcessing
Command Line Interface
1 2 3 4 5 6
• Innovate and add!• Presentations• Judges
4-port non-learningswitch
4-port learningswitch
IPv4 routerforwarding path
Integrate with S/W Interoperability
SwitchingForwardingTable
Learning Environment Modular design
Testing
CS344 MilestonesFinal Project
EuroSys 2008 - NetFPGA Tutorial 137 S T A N F O R D U N I V E R S I T Y
Typical NetFPGA Course PlanWeek Software Hardware Deliver
1 Verify Software Tools Verify CAD Tools Write Design Document
2 Build Software Router Build non-learning switch Run Software Router
3 Cmd. Line Interface Build learning switch Run basic Switch
4 Router Protocols Output Queues Run Learning Switch
5 Implement Protocol Forwarding path Interface SW & HW
6 Control Hardware Hardware Registers HW/SW Test
7 Interoperate Software & Hardware Router Submission
8 Plan new Advanced Feature Project Design Plan
9 Show new Advanced Feature Demonstartion
EuroSys 2008 - NetFPGA Tutorial 138 S T A N F O R D U N I V E R S I T Y
CS344 Project Presentations
http://cs344.stanford.edu
EuroSys 2008 - NetFPGA Tutorial 139 S T A N F O R D U N I V E R S I T Y
Network Systems Architecture at Rice
http://comp519.cs.rice.edu/
EuroSys 2008 - NetFPGA Tutorial 140 S T A N F O R D U N I V E R S I T Y
Networked FPGAs in Research1. RCP: Congestion control
• New module for parsing and overwriting new packet• New software to calculate explicit rates
2. Packet Monitoring (ICSI)• Network Shunt
3. Deep Packet Inspection (FPX)• TCP/IP Flow Reconstruction• Regular Expression Matching• Bloom Filters
4. Ethane: Network security• New switch (“managed flow-table”) deployed
5. Buffer Sizing• Reduce buffer size and measure effect on network
performance.• Need a way to set buffer size, and measure buffer
occupancy.
EuroSys 2008 - NetFPGA Tutorial 141 S T A N F O R D U N I V E R S I T Y
The NetFPGA.org Community
http://netfpga.org/
EuroSys 2008 - NetFPGA Tutorial 142 S T A N F O R D U N I V E R S I T Y
Group Discussion• Your plans for using the NetFPGA
– Teaching– Research– Other
• Resources needed for your class– Source code– Courseware– Examples
• Your plans to contribute– Expertise – Capabilities– Collaboration Opportunities
EuroSys 2008 - NetFPGA Tutorial 143 S T A N F O R D U N I V E R S I T Y
Survey• How did you like this this tutorial?
– What did you find useful?– What should be improved?– What should be removed?– What should be added?
• Can we post the video from this event?– If not, please let us know.
• Complete On-line survey– http://netfpga.org/tutorial_survey.html
EuroSys 2008 - NetFPGA Tutorial 144 S T A N F O R D U N I V E R S I T Y
Acknowledgements
Jianying Luo, Glen Gibb, Nick McKeown, Greg Watson, Jim Weaver, Jad Naous, Ramanan Raghuraman,
Paul Hartke, John LockwoodAlso (not shown): Adam Covington, David Erickson,
Brandon Heller, Neda Beheshti, Sara Bolouki
David Miller, University of Cambridge
NetFPGA Team at Stanford University: January 2007
EuroSys 2008 - NetFPGA Tutorial 145 S T A N F O R D U N I V E R S I T Y
Major support for this Tutorial
Previous Tutorial Hosted By:
SIGMETRICS
Special thanks to Joe Sventek
EuroSys 2008 - NetFPGA Tutorial 146 S T A N F O R D U N I V E R S I T Y
Acknowledgements• Support for the NetFPGA project has been provided
by the following companies and institutions
Disclaimer: Any opinions, findings, conclusions, or recommendations expressed in this material do not necessarily reflect the views of the National Science Foundation or of any other sponsors supporting this project.
EuroSys 2008 - NetFPGA Tutorial 147 S T A N F O R D U N I V E R S I T Y
Reference on the Web
NetFPGA homepage
http://NetFPGA.org