SACLA/XFEL におけるイベント駆動型 データ収集システム
25
SACLA/XFEL ににににににににににに ににににににににに にに に JASRI/SPring-8 ににににににににににににににに にににに ににににににににに ＠ 2012/1/28
description
SACLA/XFEL におけるイベント駆動型 データ収集システム. 大端 通 JASRI/SPring-8. Contents. Glance over SACLA/XFEL Overview of DAQ system DAQ Front-end Future perspective Summary. DAQ for Large Image Sensors : > 5Gbps. CPU board. “K Supercomputer” 10 Pflops, 2012.4~. > 5 Gbps in total. Custom-made VME - PowerPoint PPT Presentation
Transcript of SACLA/XFEL におけるイベント駆動型 データ収集システム
SACLA/XFEL におけるイベント駆動型データ収集システム大端 通 JASRI/SPring-8
次世代データ収集システム研究会＠大阪大学 核物理研究センター 2012/1/28
Contents
• Glance over• SACLA/XFEL• Overview of DAQ system• DAQ Front-end• Future perspective• Summary
次世代データ収集システム研究会＠大阪大学 核物理研究センター 2012/1/28
DAQ for Large Image Sensors : >5Gbps
Camera Linkカード
PrPMC
FPGAデータ圧縮
Camera Linkカード
PrPMC
FPGAデータ圧縮
Camera Linkカード
PrPMC
FPGAデータ圧縮
Camera Linkカード
PrPMC
FPGAデータ圧縮
Camera Linkカード
PrPMC
FPGAデータ圧縮
Camera Linkカード
PrPMC
FPGAデータ圧縮
Camera Linkカード
PrPMC
FPGAデータ圧縮
Camera Grabber
PrPMCLinux
FPGALossless
Compression
>5 Gbpsin total
Custom-madeVMECamera Link Board
10 Gbps Ethernet
“K Supercomputer”10 Pflops, 2012.4~
10 Gbps Ethernet
CPU board
CCD array4M pixels(8 sensor modules)60Hz
High SpeedStorage
Data-handlingServers
CameraLink> 2Gbps/connection
Reduced data foron-line monitor
controlPC ClusterPreprocess10Tflops
次世代データ収集システム研究会＠大阪大学 核物理研究センター 2012/1/28
SACLA : XFEL facility at SPring-8 (SPring-8 Angstrom Compact free electron LAser)
• Began operation in February 2011.– Aiming for < 1Å of wavelength of FEL by 8-GeV linac.
• First lasing achieved on June 2011 at 1.2Å .• User experiment will begin on April 2012.
SPring-8
SACLA~700m
e-
ExperimentalHall
次世代データ収集システム研究会＠大阪大学 核物理研究センター 2012/1/28
5
Anticipating a lot of Type of Experiments
XFELAtomic-level imaging for non-crystalline objects
Probing of ultrafast chemical reaction & phase transition
Generation of extreme state
High peak brilliancex109
Ultrafast pulse < 100 fs
Spatial coherence
100%
Courtesy of M. Yabashi
phage flagella, cilia
次世代データ収集システム研究会＠大阪大学 核物理研究センター 2012/1/28
Coherent X-ra
ys
Sample
Unstained Human Chromosome
Fourier Transform
Phase Retrieval
Coherent X-ray Diffraction
Sample ImageReconstruction
without the aid of Lenses
22 )(KFPrdd
e
rrK rK deF i )()(
Structure Factor
Differential Cross Section
(r) : Electron Density Distribution
e.g. Coherent X-ray Diffraction Imaging
Courtesy of Y. Nishino
A lot of images must be taken by a large pixel sensor to reconstruct the original structure.
次世代データ収集システム研究会＠大阪大学 核物理研究センター 2012/1/28
Overview of DAQ system
Requirements for Data Acquisition(DAQ)
• Shot-by-shot data acquisition in synchronization with the beam operation cycle.– In order to correlate the beam characteristics with the data. – 60Hz at present, higher for future.
• Accumulate data of large image sensors into storage without data loss.
⇒ >5 Gbps of data rate with 6M pixels at 60Hz .– Data compression to reduce the bandwidth and storage size.
• On-line data-quality monitor for efficient data collection.• Any type of users instrument specific to their
experiment can be attached to our DAQ to work with.– Like a Plug-and-play device.
次世代データ収集システム研究会＠大阪大学 核物理研究センター 2012/1/28
• Network distributed system.• MADOCA as the control framework.
Schematic of DAQ
Front-endElectronics
Front-end Computer
1G/10G Ethernet
DataBuffering
VME : Solaris PC : Linux
DataBuffering
Sybase
MySQL
On-lineDatabase
AcceleratorStatusDatabase
DAQ Control
GUI
On-lineData Monitor
GUI
VME : Solaris PC : Linux
DataBuffering
PC farmfor
Analysis
京 (K)Super
ComputerImage SensorsCCD cameras
Beam lineInstruments Monitors
UsersExperimentalInstruments Detectors
HighSpeed
Storage
Windows LabVIEW TCP socket
client vi 次世代データ収集システム研究会＠大阪大学 核物理研究センター 2012/1/28
• Data transfer via TCP/IP over 1- and 10-Gbps Ethernet. • FIFO buffering for non-real-time components.
Schematic of DAQ
Front-endElectronics
Front-end Computer
1G/10G Ethernet
DataBuffering
VME : Solaris PC : Linux
Sybase
MySQL
On-lineDatabase
AcceleratorStatusDatabase
DAQ Control
GUI
On-lineData Monitor
GUI
VME : Solaris PC : Linux
PC farmfor
Analysis
京 (K)Super
Computer
TCP/IP
Image SensorsCCD cameras
UsersExperimentalInstruments Detectors
HighSpeed
Storage
DataBuffering
Beam lineInstruments Monitors Data
Buffering
Windows LabVIEW TCP socket
client vi 次世代データ収集システム研究会＠大阪大学 核物理研究センター 2012/1/28
Event Synchronization: Tag Number• To reconstruct the data set in the same beam shot,
a “tag number” is recorded in every data to identify which one is related to which beam shot.– Sequential number of master trigger pulse.– Counted by the trigger-counter module at each station.
• Delivered to users instruments as well.– Encoded serial pulse-train, parallel bit-pattern.
MasterTrigger
BeamlineComponents
ExperimentalDetectors
Trigger Counter
AcceleratorComponents
MonitorSystemGun
Trigger Counter
Trigger Counter
Trigger Counter
Trigger Counter
Trigger Counter
Trigger Counter
Trigger Counter
Trigger Counter
60Hz Pulse
1,2,3,.. 1,2,3,..
UsersInstruments
Trigger Counter
1,2,3,..
1,2,3,..
1,2,3,.. 1,2,3,.. 1,2,3,..
次世代データ収集システム研究会＠大阪大学 核物理研究センター 2012/1/28
DAQ Front-end
DAQ front-end
DAQ Front-end
# of pixel 500k pixelDepth 16 bitRepetition rate 60 HzTotal bit rate 480 Mbps
Cameralink (base configuration) < 2.04 Gbps
Gigabit Ethernet << 1Gbps- Protocol overhead- Congestion control
Interface with CCD camera and transfer the data to storage system
Trigger
次世代データ収集システム研究会＠大阪大学 核物理研究センター 2012/1/28
Safety margin < 300 Mbps
On-the-fly lossless compression
Trigger tag counter
• general purpose FPGA board with custom user logic executable.
FPGA: VIRTEX-5 (XC5VFX70T)
• CPU board with GbE I/F CPU: PowerQUICC III• Cameralink board.
Designed for a RF-beam monitor system and an x-ray diamond monitor system of SPring-8
Implementation of DAQ front-end
次世代データ収集システム研究会＠大阪大学 核物理研究センター 2012/1/28
Implementation
Cameralink
CPU board
Bandwidth Limit: 760Mbps Repetition limit is ~90Hz 次世代データ収集システム研究会＠大阪大学 核物理研究センター 2012/1/28
Implementation
IP cores on FPGA 次世代データ収集システム研究会＠大阪大学 核物理研究センター 2012/1/28
Cam
eraL
ink
Data file servers
Data Direct Network S2A9900
Quantum StorNextfile system
> 5Gbps in total
SAN>100Gbps
High Speed Storage : DDN + StorNext• Parallel-writing to a single file system
– Achieve over 5 Gbps in total.– Convenient to handle data files in a centralized repository.
~180TB>GB/sec
Single file system
Data ofCCD sensor #1
#2 #3 #4 #5 #6 #7 #8 #9 #10 #11 #12
additionalsensor
1G/10G Ethernet
次世代データ収集システム研究会＠大阪大学 核物理研究センター 2012/1/28
June 7, 2011First lasing at SACLABeam energy: 7GeVWavelength: 1.2ÅMPCCD image
DAQ System Worked as Expected
1 mm
• Intensively used during beam tuning of SACLA to view the beam profile.
• Ready for user experiments.
次世代データ収集システム研究会＠大阪大学 核物理研究センター 2012/1/28
Future perspective
Upgrade plan In near future:Requirement for next gen. DAQ system• Image sensor: 6M 76M pixels⇒• Beam repetition: 60Hz 120Hz ⇒ 300Hz⇒
⇒ Data rate ~ x26 : ~150Gbps !~ x65 : ~380Gbps !
Target specification of DAQ front-end
次世代データ収集システム研究会＠大阪大学 核物理研究センター 2012/1/28
Number of Pixel 1.92MDepth 24bit+metadataRepitation 60HzTotal bit rate 2.77Gbps※
Number of Pixel 1.92MDepth 24bit+metadataRepitation 300HzTotal bit rate 13.85Gbps※
1st phase Final target
Overview of DAQ system
次世代データ収集システム研究会＠大阪大学 核物理研究センター 2012/1/28
Latest action and future plan
Cameralink Cameralink HS CoaXPress 10GigE VisionMax speed / single 2.04Gbps(Base) 3.125Gbps 6.25Gbps 10GbpsMax speed / several 5.44Gbps(Full) 3.125Gbps x N 6.25Gbps x N 40Gbps (4 lanes)Max length 7~10m 15m 40~130m 37~100mConnector/Cable SDR CX4 BNC/coaxial RJ45/Cat6A-Trigger Yes Yes Yes ?Optical option Yes not yet ? YesStandardization done not yet done not yet
- Features of several VISION technologies -
Because progress of vision technology is very fast, we can wait until a market is growing up and specification will be settled down.
次世代データ収集システム研究会＠大阪大学 核物理研究センター 2012/1/28
Latest action and future plan
XAUI (10GbE) SerDes(RocketIO) sRIO PCIe
Max speed / single 6~10Gbps 3.125~6.5Gbps※ 6.25Gbps 5~40Gbps
Max speed / several - 3.125~6.5Gbps x N 6.25Gbps x N -
Max length several 100 m / optical fiber
Connector/Cable SFP+ (Q)SFP+ SFP+ (Q)SFP+Trigger Yes Yes Yes Yes
Optical option Yes Yes Yes YesStandardization Yes Yes Yes YesCost of IP core Free Free Some Some
- Features of several generic serial link technologies -
※ depend on transceiver We started preparing test stand for XAUI and RocketIO, and will finish evaluation within this fiscal year.
次世代データ収集システム研究会＠大阪大学 核物理研究センター 2012/1/28
Latest action and future plan- Platform design -
FMC (FPGA Mezzanine Card) separate a logic and I/O from a platform. change suitable I/F enough performance. ~20Gbps
Study itemsActual bandwidth between FMCs .
Actual bandwidth of DMA transfer at PCIe
Hardware implementation
FPGA Logic implementation
次世代データ収集システム研究会＠大阪大学 核物理研究センター 2012/1/28
Summary• We have constructed an event-synchronized data
acquisition system for >5 Gbps of data rate for the SACLA experiment.– Shot-by-shot data acquisition up to 60Hz .– Expandable for user instruments.
⇒ Ready for user experiments.
• Further upgrades for larger detector size and higher repetition rate will be necessary in future.– Faster data transfer and storage system.– Effective data compression / suppression.
⇒ Start fighting them.
次世代データ収集システム研究会＠大阪大学 核物理研究センター 2012/1/28
