High Data Throughput Detectors - esrf.fr

High Data Throughput Detectors

A. Homs-Puron, ESRFUP WP-10 Soft Workshop, Grenoble, 13-14 Jan 2011

Alejandro Homs Puron on behalf of the

ISDD Software Beamline Control Unit(former BLISS)

Talk outline

• Introduction to Espia card

� ESRF-developed detectors

� CameraLink detectors

• Data saving performance

� Local disks


� Local disks

� Network storage

• Latest developments

� Recently acquired detectors

� Prepare for next generation

� Designed to interface the FReLoN 2k� Collaboration � ESRF + SECAD, S.A.� 2 Gbps fiber optic link (> 100 m)� PCI 64 bits / 66 MHz

Espia card


� PCI 64 bits / 66 MHz� ~180 MB/s maximum data rate� Driver developed in-house

New FReLoN Generation

FReLoN 2k (ATMEL):� 2048 x 2048 – 14 bit – 4 ADCs� Frame Transfer Mode� 2048 x 1024 @ 16 fps � 66 MB/s


FReLoN 4M (KODAK):� High dynamic range

FReLoN HD (PSB-2):� 2048 x 1024 @ 32 fps � ~125 MB/s

• Connects to the Espia• 2 x Camera Link connectors:• Multiple pixel packing formats• Test image generator @ ~180 MB/s

Camera Link: FOCLA


• Test image generator @ ~180 MB/s

� Frame transfer technology� 1024 x 1024 @ 60 fps � 120 MB/s� 1 x Camera Link [Base] Connector

Dalsa Pantera 1M60


� 1 x Camera Link [Base] Connector� Compact!� Image not reconstructed� FOCLA does reconstruction

� 512 x 512 @ 300 fps � 150 MB/s� 512 x 128 @ 500 fps � 125 MB/s� 2 x 8 ADCs – 12 bit� 2 x Camera Link [Base] Connectors� Separate RS-232 serial line

Sarnoff CAM512


� Separate RS-232 serial line

� Tomography in 0.5 sec @ 500 fps

Medipix2 / Maxipix

� Pixel detector � photon counting� 256 x 256 – 13 bit� 0.3 ms readout time� 1000 fps � 130 MB/s� Max: 1400 fps � 180 MB/s


� Chipboard: University of Geneve� PRIAM: ISG/ESRF� Interfaced to the Espia

Espia

Multichip Maxipix• Medipix2 array• 5 x 1 � 1280 x 256• 2 x 2 � 512 x 512

� 1000 fps � 650 MB/s� PRIAM multiple FO links


� PRIAM multiple FO links� Future configs: 5 x 2

Sarnoff @ ID15

� 512 x 512 12 bits @ 300 fps � 150 MB/s� Write to local disks: 360 MB/s� Write to remote disks: 300 MB/s


Central Data Server

10 Gbps Ethernet

“Bestia”� Dell PowerEdge 2900� Similar to Ecrin/Trenton + 16 GB RAM� 4 + 6 = 10 disks � 530 MB/s (800 MB/s raw)� Super Sarnoff: 2x FOCLA test image� Acquisition rate: ~360 MB/s


Espia applications40 detectors:

ESRF, ANKA & DESSY


LIMA: Library for IMage Acquisition

� Reuse of common code � generic procedures + interfaces� Software “features” fallback if hardware has limited capabilities:

�RoI, Binning, Frame Accumulation� Basic processing and data reduction:

�Centroid, RoI statistics, background subtraction, parallel saving


�Centroid, RoI statistics, background subtraction, parallel saving�Sinogram

� Low level: C++ (multi-threaded)� High level: Python (SIP)� Generic TANGO interface + detector specific configuration� Currently integrated: Frelon, Pilatus, Multichip Maxipix� Next in the queue: PCO.dimax � Specific data reduction algorithms through “plug-ins”

Espia Next Generation

� PCI-Express 8x lanes� 4 fiber optic links @ 250 MB/s �1 GB/s� 100% register backward compatibility� 64-bit DMA support


Turbo Bestia� 4 + 6 + 8 = 18 disks � Single RAID 0 array � 840 MB/s raw� 3 RAID 0 arrays + parallel write � 1.5 GB/s raw

4x


4x

6x

8x

Solid State Storage� Current capacities: 300 GB� Future roadmap: 1.2 TB� Reading: 1.5 GB/s� Writing: 1.2 GB/s


EspiaExpress first tests

� 4 FOCLA data generators @ 180 MB/s � 720 MB/s� 4 SSD SATA disks @ 190 MB/s � 760 MB/s� Preliminary acquisition: 20 s � 14 GB


EspiaExpress first tests

� 4 FOCLA data generators @ 180 MB/s � 720 MB/s� 4 SSD SATA disks @ 190 MB/s � 760 MB/s� Preliminary acquisition: 20 s � 14 GB� Intensive frame-to-high-memory copy


Photron CMOS results @ ID15 (2009)


Photron CMOS results @ ID15 (2009)

• 40,000 fps !!


• 40,000 fps !!

J. Synch. Rad. (2009) 16, 432-434

pco.dimax

• CMOS detector• 2016 x 2016 – 12 bit• 1279 fps � 9 GB/s• 36 GB internal RAM• GigE: 90 MB/s


• Camera Link:• Standard 16-bit packing: 160 MB/s• Fast 2x12-bit packing: 240 MB/s

Windows 7 workstation

� Local 4 disks RAID-0 � 400 MB/s

� NAS + 10 Gigabit � 240 MB/s

1 Gbps Ethernet


Sneakernet10 Gbps Ethernet

Thecus 7700Pro NAS

XPCS GPU workstation @ ID10

� Fast data retrieval for online data analysis �75 MB/s


1 Gbps Ethernet

Tomography GPU workstation @ ID19

� Data replication into GPU workstation:�10 Gbps + 4 disks + NFS4 � 300 MB/s !

�Online slice reconstruction (~6 s)


1 Gbps Ethernet 10 Gbps Ethernet

Dectris Pilatus 300K – 1M – 6M

• Pixel detector � photon counting• 487 x 619 @ 200 fps • 981 x 1043 @ 30 fps • 2463 x 2527 @ 120 fps

• Effective speed � 75 MB/s


• Effective speed � 75 MB/s• TVX control software � socket interface

� Included in LIMA�Definitive saving configuration under design

Next generation PC

• E4/Supermicro vs. Ecrin/Trenton• Dual six core• Latest Intel chipset • Up to 8 disks• Possibility of GPU included


• Possibility of GPU included• Evaluation arrived yesterday …

Fast DAQ infrastructure

• Generate 10 GB/s in the near future• 10 (40) Gbps Ethernet is the common choice• External PCI-Express?


• Common hardware architecture• Flexible, scalable, software friendly• Looking for (EU) resources (P. Fajardo)• XNAP, MEDIPIX3• EIGER

Conclusions

� Significant number of supported 2D detectors� This number keeps raising …� Experience in high speed acquisitions� New detectors demand high performance control


� New detectors demand high performance control� A unified hardware infrastructure is desirable …� Common LIMA library is helping!

Acknowledgements

ISDD Ctl. Elect/Detectors:� J.C. Labiche� E. Collet� L. Siron� J.J. Thevenin

Computing Services:� L. Garçon� F. Andrei� D. Gervaise� P. Meritto

ISDD Software BCU:� M. Papillon� S. Petitdemange� S. Ohlsson� L. Claustre


SECAD, S.A.� F. Hereson

� J.J. Thevenin� C. Ponchut� J.M. Rigal� P. Pinel� J. Clement� P. Fajardo

� P. Meritto� B. Rousselle� F. Calvelo� B. Lebayle

� L. Claustre

Already leaved:� D. Fernandez� P. Makijarvi

Thank you!


Thank you!

High Data Throughput Detectors - esrf.fr

Documents

Transcript of High Data Throughput Detectors - esrf.fr