ELI: Petawatt nebo Petabyte – kdo z koho?

Pavel Bastl, Jaroslav Charfreitag

CESNET Community Fórum 2014 - Zpracování a ukládání velkých dat

ELI

Local DAQ

E L I – B L – 4 4 4 2 – P R E – 0 0 0 0 0 0 0 0 - A

Based on rack mount computer with PCIe interface Amount of PCIe x8 interfaces should be available

Optical fiber based NIC for DAQ network At the moment 4x SFP+ 10Gb/s NIC card (40 Gb/s overall throughput) Some SFP+ cages may be connected locally during local control More than one NIC may be inserted when needed and PCIe slots are available

Digitizers Other “standard” measurement cards are available on the market

Big amount of cheap DDR RAM is available Front access to devices in PCIe slots must be provided

Projekt ELIExtreme Light Infrastructure (ELI)

3 + 1 laser = ELIELI rovná se tři laserová centra sdružená pod jednou hlavičkou

První centrum (ELI Beamlines) bude umístěna v České republice a bude vytvářet novou generaci sekundárních zdrojů pro mezioborovéaplikace (fyzika, medicína, biologii, materiálové inženýrství…)

Druhé centrum (ELI Attosecond) se staví v Maďarsku a má se věnovat fyzice ultrakrátkých optických pulsů v řádu attosekund.

Třetí centrum (ELI Nuclear Physics) orientované na fotonukleární fyziku, by mělo sídlit v Rumunsku.

Čtvrté infrastruktury se v současné době jedná.

Projekt ELI – Beamlines CentrumDolní Břežany

První centrum (ELI Beamlines)

Projekt ELI – Plán umístění systémů

1. patroPodpůrné technologie

PřízemíLaserové haly

PodzemíKompresorová halaDistribuce svazků ve vakuu Specializované experimentální haly (E1 – E6)

Projekt ELI – Technologie a zdroje dat

Obory a oblasti

StavebníStrojníLaserovéExperimentálníŘídící (SCADA, kontrolní systémy)Zpracování dat (DAQ systémy) – největší zdrojOstatní – HR, finance, evidence, běžný IT provoz

Zpracování dat (DAQ systémy)

DAQ = Data Acquisition

data z experimentů, měření, kontrolních systémůřídicích systému

Řádově stovky TB dat z jednoho měření !!!

ELI = možnost až 6 experimentů současněELI = 4 laserové dráhy různého typu

ELI = zdroj dat v éře „BigData“

Agenda

DAQ system

Data acquisition introduction Data acquisition devices Data acquisition server structure Data acquisition online processing Global level data acquisition Local level data acquisition

Data acquisition introduction

Data acquisition - DAQ

Instrumentation

Data acquisition introduction

Data acquisition - DAQ

Instrumentation

Frame rate up to ~kHz level High volume of data High speed throughput is required Data tame-stamping

Data acquisition structure

Data acquisition input devices

Matrix detectors CCD CMOS

Scalar detectors Digitizers …

Network based interfaces GigE Visision 10GigE Visison (formally GigE

Vision ver 2.0)

Local bus interfaces USB 3.0

Metallic cable Optical fiber

PCIe CoaXPress CameraLink

PXI/PXIe CameraLink

TCA

Data acquisition structure

Data acquisition input devices

Matrix detectors CCD CMOS

Scalar detectors Digitizers …

Network based interfaces GigE Visision 10GigE Visison (formally GigE

Vision ver 2.0)

Local bus interfaces USB 3.0

Metallic cable Optical fiber

PCIe CoaXPress CameraLink

PXI/PXIe CameraLink

TCA

Global level DAQ structure – hierarchical data storage

Online data view NFS/CIFS mounts

DAQ server

Hierarchical data storage

High speed disks - Tier 1 Low speed disks - Tier 2 Tape library – Tier 3

DAQ server

Hierarchical data storage

High speed disks - Tier 1 Low speed disks - Tier 2 Tape library – Tier 3

DAQ server

Hierarchical data storage

High speed disks - Tier 1 Low speed disks - Tier 2 Tape library – Tier 3

Multinode DAQ server

Shared DRAM - Tier0

DAQ server

Hierarchical data storage

High speed disks - Tier 1 Low speed disks - Tier 2 Tape library – Tier 3

Multinode DAQ server

Shared DRAM - Tier0

Network interface

At minimum 4xSFP+ 40Gb interface

DAQ server – FPGA & Time-stamping

FPGA online processing FPGA card in PCIe DDR RAM buffer 10Gb SFP+ interfaces

FPGA IP cores & SW OpenCL support DDR memory IP core 10Gb ethernet IP core PCIe IP core

DAQ server – FPGA & Time-stamping

FPGA online processing FPGA card in PCIe DDR RAM buffer 10Gb SFP+ interfaces

FPGA IP cores & SW OpenCL support DDR memory IP core 10Gb ethernet IP core PCIe IP core

Time stamping support Timing system based card

Available only for facility level devices

PTP support Suitable for user computer or

devices

Global level DAQ - interfaces

LC connector Full duplex two fibers Standard in data networks and in ELI

MPO/MTP connector Twelve fibers in one connector Used in service HUBs

FC connector Standard for instrumentation

Local DAQ

Based on rack mount computer with PCIe interface Amount of PCIe x8 interfaces should be available

Optical fiber based NIC for DAQ network At the moment 4x SFP+ 10Gb/s NIC card (40 Gb/s overall throughput) Some SFP+ cages may be connected locally during local control More than one NIC may be inserted when needed and PCIe slots are available

Digitizers Other “standard” measurement cards are available on the market

Big amount of cheap DDR RAM is available Front access to devices in PCIe slots must be provided

Local DAQ - interfaces

PCIe Standard interface Data acquisition cards available Optical network is supported USB 3.0 is supported Timing and triggering not supported by the bus

PXI/PXIe NI proprietary interface – open? Data acquisition and general measurement cards available

General measurement cards CPU cards are very expensive and have not enough memory for high speed data storage Optical network not supported USB 3.0 interface not supported Backplane supports timing and triggering

μTCA Advanced telecommunication architecture – open, very flexible, complicated Data acquisition and general cards available Optical network is supported USB 3.0 is supported Backplane supports timing and triggering Becomes to be popular in facilities (DESY, Elettra, SLAC etc.)

Local DAQ - interfaces

Based on rack mount computer with PCIe interface Amount of PCIe x8 interfaces should be available

Optical fiber based NIC for DAQ network At the moment 4x SFP+ 10Gb/s NIC card (40 Gb/s overall throughput) Some SFP+ cages may be connected locally during local control More than one NIC may be inserted when needed and PCIe slots are available

Digitizers Other “standard” measurement cards are available on the market

Big amount of cheap DDR RAM is available Front access to devices in PCIe slots must be provided

These requirements lead to server based computer

ELI – vlastní datové uložiště (vývoj)

Komerční uložiště X Vlastní uložištěcena, rozšiřitelnost, morální udržitelnost, doba dodání

- 4U/5U velikost pro montáž do rack skříně, duální napájení- Možnosti – klasické disky/SSD disky/kombinace – dle požadavku na IOPs- 45 disků 3,5“, hrubá kapacita až 270 TB (450 TB na přelomu 2015/2016)- Možnost FC, iSCSI a InfiniBand (FDR/QDR) rozhraní dle potřeb

Děkujeme za pozornost a pozvánísdružení

Institute of Physics ASCR, v. v. i.Na Slovance 2

182 21 Prague 8Czech Republic

eli-cz@fzu.czwww.eli-beams.eu

Kontakt: Fyzikální ústav AV ČR v.v.i., Na Slovance 2, 182 21 Praha 8OC Harfa, Českomoravská 2420/15, 190 00 Praha 9 (do září 2015)

Pavel Bastle, Pavel.Bastl@eli-beams.euJaroslav Charfreitag, Jaroslav.Charfreitag@eli-beams.eu