LHCb DAQ Protocol Independent Performance Evaluator: DAQPIPE
description
Transcript of LHCb DAQ Protocol Independent Performance Evaluator: DAQPIPE
![Page 1: LHCb DAQ Protocol Independent Performance Evaluator: DAQPIPE](https://reader036.fdocuments.net/reader036/viewer/2022062410/568161d9550346895dd1e1a0/html5/thumbnails/1.jpg)
LHCb DAQ Protocol Independent Performance
Evaluator: DAQPIPE
Daniel Hugo Cámpora Pérez LHCb DAQ workshop, 27-05-2013
1
![Page 2: LHCb DAQ Protocol Independent Performance Evaluator: DAQPIPE](https://reader036.fdocuments.net/reader036/viewer/2022062410/568161d9550346895dd1e1a0/html5/thumbnails/2.jpg)
What is it, anyway
● DAQPIPE is an evaluation tool designed to measure the performance of a generic DAQ system.
Daniel Hugo Cámpora Pérez LHCb DAQ workshop, 27-05-2013
2
EM
![Page 3: LHCb DAQ Protocol Independent Performance Evaluator: DAQPIPE](https://reader036.fdocuments.net/reader036/viewer/2022062410/568161d9550346895dd1e1a0/html5/thumbnails/3.jpg)
What we want to measure
● Network transfer protocols● Network topologies● Event-Building protocols
Daniel Hugo Cámpora Pérez LHCb DAQ workshop, 27-05-2013
3
![Page 4: LHCb DAQ Protocol Independent Performance Evaluator: DAQPIPE](https://reader036.fdocuments.net/reader036/viewer/2022062410/568161d9550346895dd1e1a0/html5/thumbnails/4.jpg)
Network transfer protocols
● Preliminary tests with TCP/UDP over 40G.● Preparing IB FDR tests.● Testing
– 40G Ethernet UDP– iWarp– ROCE– IB OpenMPI (full hardware support)– IB RDMA
Daniel Hugo Cámpora Pérez LHCb DAQ workshop, 27-05-2013
4
InfiniBand? Ethernet? Upcoming ones?
![Page 5: LHCb DAQ Protocol Independent Performance Evaluator: DAQPIPE](https://reader036.fdocuments.net/reader036/viewer/2022062410/568161d9550346895dd1e1a0/html5/thumbnails/5.jpg)
Network topologies (recap)
Unidirectional flow
● Simple
● Current solution, scale up
Daniel Hugo Cámpora Pérez LHCb DAQ workshop, 27-05-2013
5
![Page 6: LHCb DAQ Protocol Independent Performance Evaluator: DAQPIPE](https://reader036.fdocuments.net/reader036/viewer/2022062410/568161d9550346895dd1e1a0/html5/thumbnails/6.jpg)
Network topologies
Bidirectional uniform flow
● Full-Duplex on core – mixed-servers.
● Implementation details: RUs sit on servers.
● Fraction of computing done in receiving nodes, offloading done to FUs.
Daniel Hugo Cámpora Pérez LHCb DAQ workshop, 27-05-2013
6
![Page 7: LHCb DAQ Protocol Independent Performance Evaluator: DAQPIPE](https://reader036.fdocuments.net/reader036/viewer/2022062410/568161d9550346895dd1e1a0/html5/thumbnails/7.jpg)
Event-Building protocols
● PUSH● PUSH with Barrel Shifting● PULL
Daniel Hugo Cámpora Pérez LHCb DAQ workshop, 27-05-2013
7
![Page 8: LHCb DAQ Protocol Independent Performance Evaluator: DAQPIPE](https://reader036.fdocuments.net/reader036/viewer/2022062410/568161d9550346895dd1e1a0/html5/thumbnails/8.jpg)
Push
Daniel Hugo Cámpora Pérez LHCb DAQ workshop, 27-05-2013
8
![Page 9: LHCb DAQ Protocol Independent Performance Evaluator: DAQPIPE](https://reader036.fdocuments.net/reader036/viewer/2022062410/568161d9550346895dd1e1a0/html5/thumbnails/9.jpg)
Barrel shifting
● Equal-length receiving queues, asymmetric sending queues.
● Potential dramatic decrease of congestion in DAQ network.
Daniel Hugo Cámpora Pérez LHCb DAQ workshop, 27-05-2013
9
![Page 10: LHCb DAQ Protocol Independent Performance Evaluator: DAQPIPE](https://reader036.fdocuments.net/reader036/viewer/2022062410/568161d9550346895dd1e1a0/html5/thumbnails/10.jpg)
Pull
Daniel Hugo Cámpora Pérez LHCb DAQ workshop, 27-05-2013
10
![Page 11: LHCb DAQ Protocol Independent Performance Evaluator: DAQPIPE](https://reader036.fdocuments.net/reader036/viewer/2022062410/568161d9550346895dd1e1a0/html5/thumbnails/11.jpg)
Let's talk software
● Test suite should be– Portable– Flexible– C-like entry point (interfaceable)– Scalable– Efficient!
Daniel Hugo Cámpora Pérez LHCb DAQ workshop, 27-05-2013
11
![Page 12: LHCb DAQ Protocol Independent Performance Evaluator: DAQPIPE](https://reader036.fdocuments.net/reader036/viewer/2022062410/568161d9550346895dd1e1a0/html5/thumbnails/12.jpg)
Output
Daniel Hugo Cámpora Pérez LHCb DAQ workshop, 27-05-2013
● Logs & rates– Effective throughput– Maximum trigger rate– Error rate– Incomplete event rate
12
![Page 13: LHCb DAQ Protocol Independent Performance Evaluator: DAQPIPE](https://reader036.fdocuments.net/reader036/viewer/2022062410/568161d9550346895dd1e1a0/html5/thumbnails/13.jpg)
Preliminary tests
Daniel Hugo Cámpora Pérez LHCb DAQ workshop, 27-05-2013
● PTP allows for microsecond-level synchronization.
● RDMA over IB CPU consumption is negligible.
13
![Page 14: LHCb DAQ Protocol Independent Performance Evaluator: DAQPIPE](https://reader036.fdocuments.net/reader036/viewer/2022062410/568161d9550346895dd1e1a0/html5/thumbnails/14.jpg)
Preliminary tests
Daniel Hugo Cámpora Pérez LHCb DAQ workshop, 27-05-2013
● 40GbE
14
Tx Rate CPU Consumption
40G (TCP), eight streams
36 Gb/s 5 – 8%
4 x 10G (TCP)
36 Gb/s
40G (UDP), four streams
32 Gb/s 4%
HP 59002x Mellanox Technologies MT27500 Family ConnectX-3
8x Chelsio Communications Inc T420-CR
![Page 15: LHCb DAQ Protocol Independent Performance Evaluator: DAQPIPE](https://reader036.fdocuments.net/reader036/viewer/2022062410/568161d9550346895dd1e1a0/html5/thumbnails/15.jpg)
Thanks!
Daniel Hugo Cámpora Pérez LHCb DAQ workshop, 27-05-2013
15
![Page 16: LHCb DAQ Protocol Independent Performance Evaluator: DAQPIPE](https://reader036.fdocuments.net/reader036/viewer/2022062410/568161d9550346895dd1e1a0/html5/thumbnails/16.jpg)
Backup
Daniel Hugo Cámpora Pérez LHCb DAQ workshop, 27-05-2013
16
![Page 17: LHCb DAQ Protocol Independent Performance Evaluator: DAQPIPE](https://reader036.fdocuments.net/reader036/viewer/2022062410/568161d9550346895dd1e1a0/html5/thumbnails/17.jpg)
Network topologies
Bidirectional mixed flow
● Full-Duplex on core - TOR connections.
● Port-density requirements for core switches significantly reduced.
● Server-wise identical to unidirectional solution.
Daniel Hugo Cámpora Pérez LHCb DAQ workshop, 27-05-2013
17
![Page 18: LHCb DAQ Protocol Independent Performance Evaluator: DAQPIPE](https://reader036.fdocuments.net/reader036/viewer/2022062410/568161d9550346895dd1e1a0/html5/thumbnails/18.jpg)
Network topologies
Bidirectional uniform flow
● Relatively high load per mixed server (RU/BU/FU) – 4 x rate
– Input – RU
– Input / Output – BU
– Output - FU
Daniel Hugo Cámpora Pérez LHCb DAQ workshop, 27-05-2013
18