The TrainBuilder ATCA Data Acquisition Board for the European XFEL The TrainBuilder ATCA Data...

The TrainBuilder ATCA Data Acquisition Board for the European XFEL

The TrainBuilder ATCA Data Acquisition Boardfor the European-XFEL

John Coughlan, Chris Day, James Edwards, Ed Freeman, Senerath Galagedera and Rob Halsall

Science & Technology Facilities CouncilRutherford Appleton Laboratory Oxfordshire, United Kingdom

E-mail: [email protected]

Presented by John Coughlan. TWEPP Oxford. September 2012.


2

European-XFEL DAQ TrainBuilder demonstrator board FPGA Firmware System Applications


Topics


XFEL DAQ Data Flow

• 3 Large 2D Pixel Detectors• 1M Pixel, 2B/Pixel, 512 Images @10Hz (X-Ray pulse Train)• 10 Gbyte per second per Megapixel rate• 10G Ethernet using UDP protocol• Modularity : 16 x 10G Links off detector per Megapixel• Scalable• Full Data Rate to PC Layer

3

Train BuilderDetector 2D Camera 10G Switch 10G serversPC Layer

10G 10G 10G

Common Systems XFEL DAQ



Image Train Building

4Presented by John Coughlan. TWEPP Oxford. September 2012.

FEMs

Train Builder

FEMs

PCSWITCH PC

PCSWITCH PC

Train Builder

N x 512N x 512

10Gbps 10Gbps

Train BuilderDetector 2D Camera

10G serversPC Layer

Partial Images 128 KB Full Images 2 MB

Detector

SpecificCommon

XFEL DAQ

N= 16

Deep Buffers

Switch

10G Links


TrainBuilder Demonstrator


• Train Builder Demonstrator• ATCA form factor• 5 x Xilinx V5FXT100 FPGAs• Analogue X-point 80x80

• 4 x FMCs dual SFP+ 10Gb• 8 x 2GB DDR2 SODIMM VLP

Modules• 4 x QDRII SRAMs

• Links Configured as Input or Output by software

• Connects to eitherDetectors or PC Farm


TrainBuilder FE x 4


Virtex5 FX100T, Dual PPC

DESY FMC2 x 10Gbps SFP+

2 x DDR2 VLP SODIMMs2 GByte

QDRII SRAM 8 MByte

SRAM PPC code


TrainBuilder FE x 4




QDRII SRAM 8 MByte

QDRII SRAMPixel reordering

DDR2Partial Images train buffer

To Xpoint10G Inputs fromDetector

Configured as INPUT : Data from Detectors



TrainBuilder FE x 4




QDRII SRAM 8 MByte

10G Outputs to Farm

QDRII SRAM Image Tiling

DDR2Full Images train buffer

From Xpoint

Configured as OUTPUT : Data to PC Farm



TrainBuilder 10G link FMC


FPGA Mezzanine Card FMCANSI/VITA 57

Developed by DESY Electronics Group

Dual 10 Gbps SFP+

Vitesse 10G PHYs

XAUI 4 x 3.125 Gbps


TrainBuilder Demonstrator


Master FPGA

Boot MMCSpartan3AN FPGA

Xpoint • Crosspoint Switch• Mindspeed 80x80 6.5 Gbps• 64 Tx & Rx @ 2.5 Gbps• Switch changes @ ~ 10 Hz

• Master FPGA Virtex5 FXT100• Manages data switching• PPC GbE TCP/IP server• Python scripts on client PC

FPGA Configuration CF card

• Boot Spartan3 with FLASH

GbE Ctrl

GbE Ctrl RJ45


TrainBuilder ATCA Interfaces


ATCA Zone 3Cable/Rear Transition Module

ATCAZone 2GbE CtrlPCIe

ATCAZone 1JTAG

JTAG

• Zone 3 Custom RTM• 32 Tx & Rx data lanes

to Crosspoint+ clock&controls

• Zone 2 • Std interfaces

• Zone 1• Power 48V• MMC 3V3

ATCA 48V DC-DC


TrainBuilder Data Rates


• 10G

XAUI 4 x 3.125 Gbps

> 1 GB/sec

• Crosspoint• Aurora 4 lanes 64B/66B @ 2.5 Gbps

> 1 GB/sec

• DDR2 (1 per link) 2 GB• Dual PPC Hard IP Embedded

~ 1 GB/sec Read & Write (Long bursts)

• QDRII SRAM (1 per 2 links) 8 MB• 32 bit R & W @ 250 MHz

~ 1 GB/sec per link (2 word burst)

• Detector (512 bunches/train)

~ 640 MB/sec per 10G LINK

• Total B/W per TB board 8 Links

~ 8 GB/sec


13

Firmware



FE FPGA Firmware


AuroraCore

0

XPOINT

4 TX PAIRS

4 RX PAIRS

4 TX PAIRS

4 RX PAIRS

AuroraCore

1

XAUICore

0

XAUICore

1

Power PCMemory

ControllerSubsystem

4 TX PAIRS

4 RX PAIRS

4 TX PAIRS

4 RX PAIRS

XTALPLL

XTALPLL

DUAL10G PHY

FMC

PH

Y

SFP+

PH

Y

SFP+

TB FPGA

10GUDPCore

1

Power PCMemory

ControllerSubsystem

10GUDPCore

0

‘XCTRL’

DDR2 SDRAMImage BuffersStorage

DDR2 SDRAM

QDRII SRAMImageManipulation

QDRII SRAM

Aurora 64B/66BXAUI

QDRIIInterface

QDRIIInterface

LOCAL LINK Protocol


FE FPGA Firmware


AuroraCore

0

XPOINT

4 TX PAIRS

4 RX PAIRS

4 TX PAIRS

4 RX PAIRS

AuroraCore

1

XAUICore

0

XAUICore

1

Power PCMemory

ControllerSubsystem

4 TX PAIRS

4 RX PAIRS

4 TX PAIRS

4 RX PAIRS

XTALPLL

XTALPLL

DUAL10G PHY

FMC

PH

Y

SFP+

PH

Y

SFP+

TB FPGA

10GUDPCore

1

Power PCMemory

ControllerSubsystem

10GUDPCore

0

‘XCTRL’


DDR2 SDRAM


QDRII SRAM

Aurora 64/66BXAUI

QDRIIInterface

QDRIIInterface

LOCAL LINK Protocol

QDRII SRAMPixel reordering

DDR2Partial Images train buffer

To Xpoint10G Inputs fromDetector


FE FPGA Firmware


AuroraCore

0

XPOINT

4 TX PAIRS

4 RX PAIRS

4 TX PAIRS

4 RX PAIRS

AuroraCore

1

XAUICore

0

XAUICore

1

Power PCMemory

ControllerSubsystem

4 TX PAIRS

4 RX PAIRS

4 TX PAIRS

4 RX PAIRS

XTALPLL

XTALPLL

DUAL10G PHY

FMC

PH

Y

SFP+

PH

Y

SFP+

TB FPGA

10GUDPCore

1

Power PCMemory

ControllerSubsystem

10GUDPCore

0

‘XCTRL’


DDR2 SDRAM


QDRII SRAM

Aurora 64/66BXAUI

QDRIIInterface

QDRIIInterface

LOCAL LINK Protocol

10G Outputs to Farm

QDRII SRAM Image Tiling

DDR2Full Images train buffer

From Xpoint


TrainBuilder Firmware System

17

PHY XAUISFP UDP PPC AUR


FPGA 1

XP

RAM

RAM



FPGA 2RAM

RAM



FPGA 3RAM

FPGA

GIGE

RAM



FPGA 4RAM

RAM

PHYXAUI SFPUDPPPCAUR


FPGA 5RAM

RAM



FPGA 6RAM

RAM


PHYXAUI SFPPEPPCAUR

FPGA 7RAM

RAM

PHYXAUIPEPPCAUR

PHYXAUI SFPPEPPCAUR

FPGA 8RAM

RAM

SFPPHYXAUIUDPPPC

RAM

ASICRX

SFPPHYXAUIUDPPPC

RAM

TB

ASICRX

FEM

SFP

FPGA

FPGA


TrainBuilderFEMs


18

Applications



Status

•First ATCA boards manufactured in Q1/2012

• 4 Boards all passed Boundary SCAN first time.

• Basic Tests working•10G links Tx and Rx•DDR2•QDRII•Crosspoint•GbE controls

• Performance testing at RAL. Data Rates 1 GB/s/link? Soak Tests.

• Board in Hamburg for XFEL DAQ PC Farm tests since July.



First Detector Application

20

1 x TB configured as4 Inputs and 4 Outputs


x4

Large Pixel Detector Prototype Quadrant. 4 FEMs 10G links


2 x TBs : ½ Mpixel


2 x TB System : 8 Links½ Mpixel

PassiveCablesTx & Rx

x8

INPUT OUTPUT


FUTURE 4 x TBs : 1 Mpixel 2013


1 Mpixel with 4 x ATCA TB Demonstrators. 16 LINKs

x16

Parallel Opto LinksSNAP 12 Tx / Rx

Scaleable to larger systems

NEW ATCA Switch Board

NEW Optical RTMs

@ 2.5 Gbps


Summary & Plans

• Developed an ATCA board for common XFEL DAQ• Firmware infrastructure already operating incl. 10G UDP• 2 x Demonstrator ATCA can instrument ½ Mpixel today• Baseline is 1 Mpixel. Scheme exists to scale to larger detectors.

•Plans•Complete Performance Tests on prototypes•Develop Optical RTM and Switch card 2013•Integrate with 1 Mpixel detectors 2014•Install XFEL in 2015•Data taking at XEFL in 2016

Acknowledgements:M. Zimmer, I. Sheviakov (DESY Electronics) C. Youngman (XFEL DAQ)



24

Questions



25

Spare Slides



European-XFEL Hamburg


2D Camera captures 512 Images per TrainTrain repetition rate = 10 Hz

~5,000 fps

Xray Free Electron Laser


Crosspoint Switch Operation


Switch Protocol : Xilinx Aurora 64/66B @ 2.5 GbpsXpoint Switch changes only @ 10 Hz

Partial Images 128 KBTime Ordered

Full Image Trains512 x 2 MB

A->Y

B->Z

C->W

D->X

A->X

B->Y

C->Z

D->W

A->Z

B->W

C->X

D->Y

A->W

B->X

C->Y

D->Z

A->Y

B->Z

C->W

D->X

A->X

B->Y

C->Z

D->W

A->Z

B->W

C->X

D->Y

A->W

B->X

C->Y

D->Z

1234A

EXT RAM

FPGA FIFO

X1234B

FIFO

1234C

RAM

FIFO

1234

1234

1234

1234

1234D

RAM

FIFO

A1D1C1B1 W

RAM

FIFO A1B1C1D1

B2A2D2C2

FIFO A2B2C2D2

C3B3A3D3 Y

RAM

FIFO A3B3C3D3

D4C4B4A4 Z

RAM

FIFO A4B4C4D4

CrosspointSwitch64 x 64Bi-dir links

8 Lanes@ ~2.5 Gbps

1 2 3 4

IIIIIIIV IIIIIIIV

2 x 10 Gbps2 x 10 GbLinks 1234A

EXT RAM

FPGA FIFO

X1234B

FIFO

1234C

RAM

FIFO

1234

1234

1234

1234

1234D

RAM

FIFO

A1D1C1B1 W

RAM

FIFO A1B1C1D1

B2A2D2C2

FIFO A2B2C2D2

C3B3A3D3 Y

RAM

FIFO A3B3C3D3

D4C4B4A4 Z

RAM

FIFO A4B4C4D4

CrosspointSwitch64 x 64Bi-dir links

8 Lanes@ ~2.5 Gbps

1 2 3 4

IIIIIIIV IIIIIIIV

2 x 10 Gbps2 x 10 GbLinks

A

C

B

D

A

C

B

D

REARINPUTS

FRONTOUTPUTS

FEEs PC Farm

DDR2 DDR2


10G UDP

28

LL TX PKT

SPLIT

LL RXPKT

COMB

UDP TXBlock

UDP RXBlock

XAUICORE

Local Link Tx

Local Link Rx

64D8C

64D8C

RDMA RDMAREG

4 TX4RX312.5MHz 156.25 MHz

50 MHz

MDIO

XAUIPHY

GA0,GA1,SCLSDA

Ctrl & Mon

TX/RX ALARMRESET

ROM

• LL FRAME (64 bit) <-> UDP PKT STREAM• FPGA <-> FPGA - FPGA <-> PC

• Tx : MAC, IP ADDR & PORT LUT (LL Frame Header) *• Rx : IP, PORTS, MAC, Filter for PLAYBACK mode• Software Programmable : Packet Length (Jumbo), IFG ...etc

*Plan to add Resends on Tx for “Reliable UDP” (TB to PC Farm)

10G PHYFPGA


Local Link


PPC DDR2 Memory Controller


DDR2SODIMM

64 bit

2 GB

V5 FX100T 2 x PPC440 Hard IP blocksEach with 4 DMA engines Tx&Rx

DDR2 controller500 MT/s

Local Link Interface 32 bit @ 250 MHz

~ 1 GB/sec Concurrent Read & Write

LL Write

LL Read

Embedded C code manages DMA engines/buffers


TrainBuilder Next Generation


• TB• V7/K7

• ≥8 x QSFP+

• >8 x DDR3 SODIMM

• XP 144 x 144 @10G

• XP -> V7/K7

• QDR Bandwidth?

• Bigger Packages

• No FMC

The TrainBuilder ATCA Data Acquisition Board for the European XFEL The TrainBuilder ATCA Data...

Documents

Transcript of The TrainBuilder ATCA Data Acquisition Board for the European XFEL The TrainBuilder ATCA Data...