GETB – a Gigabit Ethernet Application Platform Its Use in the ATLAS TDAQ Network

1

GETB – a Gigabit Ethernet Application Platform

Its Use in the ATLAS TDAQ Network

Matei Ciobotaru1,4

Micheal LeVine3

Stefan Stancu2

Brian Martin1

1CERN2University of California, Irvine

3Brookhaven National Laboratory4“Politehnica” University of Bucharest

10-June-2005Matei Ciobotaru – The GETB Platform – Real Time 2005,

Stockholm 2

Outline Motivation

The ATLAS Network and its requirements

The Gigabit Ethernet Testbed (GETB) Hardware: FPGA-based PCI card Firmware: Handel-C language Software: Python-based control system

Applications and sample results The Network Tester The ATLAS ROB Emulator


Stockholm 3

ATLAS TDAQ Network ATLAS TDAQ

Network Gigabit Ethernet Layer 2 only

Only switches ~ 1000 end-nodes

Sustained rates 5 Gb/s on average

Requirements Minimal packet loss Minimal latency High Performance

Switches Try before you buy


Stockholm 4

Motivation What we needed

A system to evaluate switches for the ATLAS TDAQ network

A development platform for other networking applications

The market offered the commercial testers Examples: Ixia, Smartbits Built for standard RFC benchmarks Not easily customizable Too many features we don’t need Expensive

Finally a new platform was created


Stockholm 5

GETB Platform – Hardware GETB = Gigabit Ethernet Testbed PCI card with an FPGA and 2 Ethernet ports

GPS Clock Input

2 x Gigabit Ethernet

3.3V PCI

Configuration Flash

SDRAM 2 x 64Mb

SRAM 2 x 512Kb

Altera Stratix EP1S25 FPGA


Stockholm 6

GETB Platform – Firmware Commercial IP cores (IP = Intellectual Property)

Gigabit Ethernet MAC , PCI Controller Logic utilization ~ 85 - 90%

Single FPGA controls 2 ports

90% Handel-C, 10% VHDL FPGA advantage massive parallelism Code shared among projects


Stockholm 7

GETB Platform – Control

Based on the Python scripting language Distributed system – 64 cards in 15 hosts (128 ports in total) Common control infrastructure for all applications


Stockholm 8

Network Tester – Features Traffic Generator and Measurement

System 1 Gigabit/sec for all packet sizes (send

and receive simultaneously)

User-defined traffic patterns Deterministic (packet descriptors) Client-Server (request-response)

Real-time measurements Throughput, packet loss, latency and

inter-packet time (IPG) Histograms for latency, IPG, packet size

ATLAS Specific Requirements and test procedures

documented Pre-defined scripts to run all tests

Python Test Description

(configure tester and DUT)

Run Analyze ReportConnect Configure


Stockholm 9

Network Tester – ATLAS Traffic Performance for ATLAS Traffic

(funnel-shaped traffic pattern) Depends on the number of buffers

Determine the maximum load the clients can receive without loss


Stockholm 10

ATLAS ROB Emulator – Context

ROB = Read Out Buffer Intelligent event fragment buffering card

Data from detectors enter on fiber links TDAQ requests event fragments from the

ROB to do the event filtering

Connected to TDAQ via GE ports or PCI

10% of total system needed for tests Production ROB card not available in sufficient

quantity Need a ROB Emulator to test candidate

switches with the TDAQ software applications


Stockholm 11

ATLAS ROB Emulator – Features

One GETB 2 x ROB emulators

Fully compliant with the protocols used by TDAQ applications Seamless integration in a TDAQ testbed

Real-time access to relevant statistics

Ready to be installed in the pre-series testbed


Stockholm 12

Summary GETB = Flexible development platform

FPGA-based Gigabit line-speed performance

Applications Network Tester

Test switches according to ATLAS requirements

ATLAS ROB Emulator Can be integrated in the full ATLAS TDAQ infrastructure

Work ongoing on other applications (Network Emulator)

The GETB is currently being used to evaluate network equipment for the ATLAS TDAQ network


Stockholm 13

End of Presentation

Backup slides


Stockholm 14

Network Tester – Packet Generation Independent Generators

Packet descriptors Modify MAC addresses, packet size,

inter-packet time, IP headers, etc. Wide range of traffic patterns

Ethernet and IP packets, VLANs

Client Server ATLAS-like request-reply traffic Clients send data request packets to

servers; Servers send back replies Clients use a token-based system to

control network load


Stockholm 15

Network Tester – Full-Mesh Traffic Fully-meshed traffic performance

Maximum loss rate ~ 1% Maximum loss rate > 20%

Switch A Switch B


Stockholm 16

Quality of Service

8 transmitters, each sending traffic with a different priority

The switch uses Weighted Round Robin Scheduling

Result obtained using data collected by one GETB tester port

Switch


Stockholm 17

Buffer Sizes

Custom method for measuring buffer sizes

3 devices 3 different memory management techniques

Size of buffers important for the ATLAS traffic pattern


Stockholm 18

MAC Address Table

Switch advertises 16000 entries in the MAC table

Measurement reveals problems for more than 5000 addresses


Stockholm 19

Latency Measurement Latency for a fully meshed traffic pattern


Stockholm 20

Features Packets can be classified

according to source or VLAN ID Latency, IPG and packet size

can be histogrammed User defined resolution and

histogram window (start offset, length)

Sample Histogram Example of histogram of Inter-packet Time

for a Negative Exponential distribution


Stockholm 21

Handel-C Hardware description language

Like VHDL, but with syntax similar to C The result of the compilation is the

description of an electrical circuit

Contains built-in parallel constructs

Special features Arbitrary widths on variables Enhanced bit manipulation operators

Simple timing model Each assignment is one clock cycle

Support for hardware constructs Multiple clock domains, on-chip

memories, external interfaces

Synchronization primitives: channels, semaphores

// 3 Clock Cycles {

a=1;b=2;c=3;

}

Sequential Block

Parallel Block

// 1 Clock Cycle par{

a=1;b=2;c=3;

}


Stockholm 22

Python Language Features

Object oriented Easy to learn, read, use Extremely portable Extensible (new modules)

class Stack: "A well-known data structure" def __init__(self): # constructor self.items = [] def push(self, x): self.items.append(x) def pop(self): x = self.items[-1] del self.items[-1] return x def empty(self): return len(self.items) == 0

What is it used for? rapid prototyping scientific applications extension language web programming

GETB – a Gigabit Ethernet Application Platform Its Use in the ATLAS TDAQ Network

Documents

Transcript of GETB – a Gigabit Ethernet Application Platform Its Use in the ATLAS TDAQ Network