EEMeeting 23.10.06 XDAQ & InfiniBand - J.Adamczewski, H.G ... · EEMeeting 23.10.06 XDAQ &...

EEMeeting 23.10.06 XDAQ & InfiniBand - J.Adamczewski, H.G.Essel, S.Linev 1


Outline

• Requirements of DAQ for CBM• The InfiniBand Cluster, uDAPL• XDAQ tests on the IB-Cluster• Summary and outlook


CBM data acquisition

data dispatcher

FEEdeliver time stamped data

CNetcollect data into buffers

Detector Detector Detector

collect

~50000 FEE chips

event dispatcher

1000x1000 switchingBNetsort time stamped data

~1000 links a 1 GB/sec

HNethigh level selection

to high level computingand archiving

~1 GB/sec Output

processing

PNetprocess eventslevel 1&2 selection

subfarm subfarm subfarm ~100 subfarms

~100 nodes per subfarm

~10 dispatchers → subfarm

~1000 collectors

~1000 active buffers

TNettime distribution


CBM DAQ features

• Triggerless data acquisition and transport until filter farm

• Event building on full data rate ~1TB/s• BNet: ~1000 nodes, high-speed interconnections• Linux may run on most DAQ nodes (even FPGAs) • Test cluster with InfiniBand:

small „demonstrator“ set-up within next year


InfiniBand test cluster

• 4 nodes:– Double Opteron 2.2 GHz, 2GB RAM– Mellanox MHES18-XT host adapter– 2x Gigabit Ethernet host adapters– SuSE Linux 9.3, x64bit version

• Mellanox MTS2400 24X InfiniBand switch

PCI Express x8


Testcluster networks

InfiniBand switch: MTS2400

InfiniBand HCA: MHES18-XT

Ethernet cabling

Gigabit Ethernet switch

InfiniBandcabling


uDAPL test application

• C++ wrapper for C-based uDAPL library• Same program runs on all nodes• All-to-all connectivity• One master (deliver commands) and many slaves• Time synchronization between nodes ~2 µs• Several kinds of traffic schedule can be generated

and executed• Statistics like transfer rate, lost packets, schedule

accuracy can be measured


All-to-all communication test

• Each node able to send data to any other node• Message- and RDMA-based transfer are supported• Master generate schedule and distribute it to all slaves• At predefined time schedule execution is started

• Several traffic patterns:– One-to-all schedule– All-to-one schedule– All-to-all round-robin– All-to-all fix target– Chaotic (nonscheduled)

master

node02node01

node03correspond to expected B-net traffic


uDAPL test results

Mellanox firmware upgraded!

• Data rates up to 960 Bytes/µs were achieved• Performance depends on size of receiving queue• Nonscheduled (chaotic) approach shows better performance for

small packets• Improvement after firmware upgrade!


Data rate over time for 1K buffers

Mellanox firmware upgraded!


Data Acquisition framework requirements

• Configuration of multiple nodes (connections and algorithms, database?)

• Controls and monitoring• Data transport (format, protocol, hardware drivers?)

• Message logging• Error handling and failure recovery (ARMOR?)

• Modular architecture, extensability! (sub-detector tests)

• User interface?


The CMS XDAQ framework

Standard DAQ framework for LHC CMS experiment(Orsini, Gutleber) http://xdaqwiki.cern.ch

• C++ libraries on Linux, modular packages

• Each node: context, xdaq process with embedded xdaq applications

• Configuration: XML• Data transport: I2O protocol (Intelligent IO)

• Communication: http, cgi; SOAP messages

http://xdaqwiki.cern.ch/


The CMS XDAQ framework

http://xdaqwiki.cern.chXDAQ features continued... • State machines (sync/async FSM) • Message logger, error handler • Monitoring tool • Hardware access library (HAL)• Front End Driver (FED kit, for CMS!)• Job Control (task handler for node control)• others:

• exceptions • threads• infospace

• data (de)serializers , ...

still kernel 2.4 modules!

http://xdaqwiki.cern.ch/


XDAQ tests on CBM InfiniBand cluster

Tested features on IB cluster: http://wiki.gsi.de/cgi-bin/view/Daq4FAIR/DaqClusterSoftwareXdaqStatus

• hyperdaq webserver (main test controls ui)• xml configuration setup• control variable access and export (infospace) • SOAP messaging (test controller application)• state machines (sync./async. with web/SOAP ui)• monitoring application (very raw ui!)• multithreading (workloop framework)• exceptions, data wrappers, message logging, toolbox,…• i2o data transport (tcp roundtrip example)

http://wiki.gsi.de/cgi-bin/view/Daq4FAIR/DaqClusterSoftwareXdaqStatus


XDAQ: messaging and peer transport

0..*

0..*

0..1

0..1

pt::JxtaMessenger

+send:void

service:std::string

exception

UnknownProtocolOrService DuplicateEndpoint AddressMismatch DuplicateListener InvalidFrame Exception NoListener SendFailure PeerTransportNotFound InvalidAddress ReceiveFailure NoMessenger MaxMessageSizeExceeded

pt::JxtaListener

+processIncomingMessage:void

service:std::string

pt::BinaryMessenger

+~BinaryMessenger+send:void

localAddress:pt::Address::Reference destinationAddress:pt::Address::Reference

pt::Messenger

+~Messenger

service:std::string

pt::HTAccessSecurityPolicy

+checkAccess:bool+checkAuth:bool+isAccessLimited:bool+isAuthLimited:bool

policyName:std::string name:std::string type:std::string

pt::PeerTransport

+createAddress:pt::Address::Reference+createAddress:pt::Address::Reference+isServiceSupported:bool

type:TransportType protocol:std::string supportedServices:std::vector<std::string>

pt::PeerTransportAgent

+addPeerTransport:void+removePeerTransport:void+getPeerTransport:pt::PeerTransport*+createAddress:pt::Address::Reference+createAddress:pt::Address::Reference+getMessenger:pt::Messenger::Reference+removeListener:void+getListener:pt::Listener*+getListeners:std::vector<pt::Listener*>+removeAllListeners:void+addListener:void

peerTransports:std::vector<pt::PeerTransport*>

pt::SecurityPolicyFactory

-instance_:SecurityPolicyFactory*-policies_:std::map<std::string,pt::SecurityPolicy*,std::less<std::string>>

+getInstance:SecurityPolicyFactory*+destroyInstance:void+getSecurityPolicy:pt::SecurityPolicy*+addSecurityPolicyImplementation:void+removeSecurityPolicyImplementation:void+getSecurityPolicyNames:std::vector<std::string>-SecurityPolicyFactory-~SecurityPolicyFactory

pt::SecurityPolicy

+~SecurityPolicy

policyName:std::string

pt::PeerTransportSender

+getMessenger:pt::Messenger::Reference

pt::PeerTransportAgentImpl

-instance_:PeerTransportAgentImpl*-senders_:std::vector<pt::PeerTransport*>-receivers_:std::vector<pt::PeerTransport*>-services_:std::map<std::string,pt::Listener*,std::less<std::string>>

+~PeerTransportAgentImpl+instance:PeerTransportAgent*+addPeerTransport:void+removePeerTransport:void+getPeerTransport:pt::PeerTransport*+createAddress:pt::Address::Reference+createAddress:pt::Address::Reference+getMessenger:pt::Messenger::Reference+addListener:void+removeListener:void+getListener:pt::Listener*+getListeners:std::vector<pt::Listener*>+removeAllListeners:void

peerTransports:std::vector<pt::PeerTransport*>

pt::Address

+~Address+toString:std::string+equals:bool

protocol:std::string service:std::string serviceParameters:std::string

pt::BinaryListener


pt::SOAPListener

+processIncomingMessage:xoap::MessageReference

service:std::string

pt::Listener

service:std::string

pt::SOAPMessenger

+~SOAPMessenger+send:xoap::MessageReference

service:std::string localAddress:pt::Address::Reference destinationAddress:pt::Address::Reference

pt::PeerTransportReceiver

-listeners_:std::map<std::string,pt::Listener*,std::less<std::string>>

+~PeerTransportReceiver+addServiceListener:void+removeServiceListener:void+removeAllServiceListeners:void+isExistingListener:bool+getListener:pt::Listener*+config:void

peer transport

xoap::SOAPBodyElementxoap::SOAPFault

+SOAPFault+getFaultCode:std::string+getFaultString:std::string+setFaultCode:void+setFaultString:void

xoap::StandardObjectPolicy

+dispose:void

exception

HTTPException Exception

xoap::Endpoint

#id_:std::string

+Endpoint+toString:std::string

xoap::SOAPConnection

#httpURLConnection_:HTTPURLConnection

+SOAPConnection+call:xoap::SOAPMessage

xoap::URLEndpoint

+URLEndpoint

URL:std::string

xoap::SOAPAllocator

+allocate:void *+dispose:void+testMemoryLeakage:void

xoap::MimeHeaders

-headers_:std::multimap<std::string,std::string,std::less<std::string>>

+MimeHeaders+getHeader:std::vector<std::string>+setHeader:void+addHeader:void+removeHeader:void+removeAllHeaders:void

allHeaders:std::multimap<std::string,std::string,std::less<std::string>>

xoap::XStr

-fUnicodeForm:XMLCh*

+XStr+XStr+~XStr+unicodeForm:const XMLCh*+operatorconstXMLCh*:

xoap::HTTPURLConnection

#socket_:int

+HTTPURLConnection+~HTTPURLConnection+receiveFrom:std::string+sendTo:void+close:void+connect:void#receive:int#send:void#open:void#extractMIMEBoundary:std::string#writeHttpPostMIMEHeader:void#writeHttpPostHeader:void

xoap::SimpleReferenceCount

-counter:unsigned long *

+SimpleReferenceCount+init:void+dispose:void+increment:void+decrement_and_is_zero:bool

xoap::SOAPName

#prefix_:std::string#name_:std::string#uri_:std::string

+SOAPName+operator==:bool

qualifiedName:std::string localName:std::string prefix:std::string URI:std::string

T:typenameSimpleReferenceCount:typename CounterPolicy =StandardObjectPolicy:typename ObjectPolicy =

CounterPolicyObjectPolicy

xoap::CountingPtr

-object_pointed_to:T*

+CountingPtr+CountingPtr+CountingPtr+~CountingPtr+operator=:CountingPtr<T,CP,OP>&+operator=:CountingPtr<T,CP,OP>&+operator->:T*+operator*:T&+isNull:bool-init:void-attach:void-detach:void

LISTENER:classxoap::Method

+obj_:LISTENER*+func_:xoap :: MessageReference+name_:std::string+namespaceURI_:std::string

+~Method+type:std::string+invoke:xoap::MessageReference+name:std::string+name:void+namespaceURI:void+namespaceURI:std::string

toolbox::lang::Methodxoap::MethodSignature

+invoke:xoap::MessageReference+name:std::string

I2O

0..*

0..1

1..*

0..1

0..1

0..1

0..1

xdaq::WebApplicationxdata::ActionListener

PeerTransportTCP

-pts_:tcp::PeerTransportSender*-ptr_:tcp::PeerTransportReceiver*-autosize_:xdata::Boolean-maxPacketSize_:xdata::UnsignedLong-poolName_:xdata::String

+XDAQ_INSTANTIATOR:+PeerTransportTCP+~PeerTransportTCP+Default:void+apply:void+failurePage:void-actionPerformed:void-reset:void

tcp::Transmitter

+isConnected_:bool+retry_:unsigned long

+Transmitter+~Transmitter+connect:void+disconnect:void+receive:int+send:void+close:void+isConnected:bool

pt::Addresstcp::Address

#url_:toolbox::net::URL#service_:std::string

+Address+~Address+toString:std::string+equals:bool

protocol:std::string service:std::string serviceParameters:std::string host:std::string port:std::string URL:std::string path:std::string socketAddress:sockaddr_in

i2o::Messengertcp::I2OMessenger

-pt_:tcp::PeerTransportSender*-channel_:tcp::Channel*-destination_:pt::Address::Reference-local_:pt::Address::Reference

+I2OMessenger+~I2OMessenger+send:void


tcp::Channel

#sockaddress_:sockaddr_in#sockaddressSize_:socklen_t#socket_:int

+Channel+~Channel+connect:void+disconnect:void+receive:int+send:void+close:void+isConnected:bool

exception

CannotConnect Exception

pt::PeerTransportSenderTask

tcp::PeerTransportSender

-outputQueue_:toolbox::squeue<tcp::PostDescriptor>-activated_:bool-done_:bool

+PeerTransportSender+~PeerTransportSender+createAddress:pt::Address::Reference+createAddress:pt::Address::Reference+isServiceSupported:bool+getMessenger:pt::Messenger::Reference+svc:int+post:void+start:void+stop:void

type:pt::TransportType protocol:std::string supportedServices:std::vector<std::string>

toolbox::lang::Classtcp::ReceiverLoop

-sockets_:std::vector<int>-address_:pt::Address::Reference-fdset_:fd_set-allset_:fd_set-timeout_:timeval-done_:bool-stop_:bool-maxfd_:int-current_:int-listenfd_:int-accepted_:int-nochannels_:int-nready_:int-autoSize_:bool-maxPacketSize_:unsigned long-listener_:i2o::Listener*-pool_:toolbox::mem::Pool*-manager_:toolbox::mem::MemoryPoolFactory*-process_:toolbox::task::ActionSignature*-logger_:Logger

+ReceiverLoop+~ReceiverLoop+receive:int+send:void+disconnect:void+connect:void+close:void+process:bool+addServiceListener:void+removeServiceListener:void+removeAllServiceListeners:void+activate:void+cancel:void-accept:int-isConnected:bool-isActive:bool-onRequest:void-safeReceive:void

_enum ReceiverLoop h 50

address:pt::Address::Reference pool:toolbox::mem::Pool* maxPacketSize:unsigned long autoSize:bool

structtcp::PostDescriptor

+ref:toolbox::mem::Reference*+channel:tcp::Channel*+handler:toolbox::exception::HandlerSignature*+context:void *

pt::PeerTransportReceivertcp::PeerTransportReceiver

#autoSize_:bool#maxPacketSize_:unsigned long#loop_:std::vector<tcp::ReceiverLoop*>#pool_:toolbox::mem::Pool*#logger_:Logger

+PeerTransportReceiver+~PeerTransportReceiver+createAddress:pt::Address::Reference+createAddress:pt::Address::Reference+addServiceListener:void+removeServiceListener:void+removeAllServiceListeners:void+isServiceSupported:bool+config:void+start:void+stop:void

type:pt::TransportType protocol:std::string supportedServices:std::vector<std::string> pool:toolbox::mem::Pool* maxPacketSize:unsigned long autoSize:bool

tcp0..1

1..*

0..1

exception

Exception

i2o::Messengerfifo::I2OMessenger

-pt_:fifo::PeerTransport*

+I2OMessenger+send:void


pt::PeerTransportSenderpt::PeerTransportReceiver

toolbox::lang::Classfifo::PeerTransport

#sync_:BSem*#mutex_:BSem*#fifo_:toolbox::rlist<fifo::PostDescriptor>#pending_:int#i2oListener_:i2o::Listener*#messenger_:pt::Messenger::Reference#localAddress_:pt::Address::Reference#process_:toolbox::task::ActionSignature*#logger_:Logger

+PeerTransport+~PeerTransport+post:void+process:bool+createAddress:pt::Address::Reference+createAddress:pt::Address::Reference+isServiceSupported:bool+getMessenger:pt::Messenger::Reference+addServiceListener:void+removeServiceListener:void+removeAllServiceListeners:void+config:void

_enum PeerTransport h 119


pt::Addressfifo::Address



protocol:std::string service:std::string serviceParameters:std::string host:std::string port:std::string URL:std::string path:std::string

xdaq::ApplicationPeerTransportFifo

-pt_:fifo::PeerTransport*

+XDAQ_INSTANTIATOR:+PeerTransportFifo+~PeerTransportFifo

structfifo::PostDescriptor

+ref:toolbox::mem::Reference*+handler:toolbox::exception::Han+context:void *

fifo0..1

0..1

0..1

0..*

pt::SOAPMessengerhttp::SOAPLoopbackMessenger

-local_:pt::Address::Reference-destination_:pt::Address::Reference

+SOAPLoopbackMessenger+~SOAPLoopbackMessenger+send:xoap::MessageReference


http::Utils

+receiveHeaderFrom:char *+receiveBodyFrom:char *+receiveFrom:char *+sendTo:void+replyTo:void+replyBodyTo:void+replyHeaderTo:void#extractMIMEBoundary:std::string#writeHttpPostMIMEHeader:void#writeHttpPostHeader:void#writeHttpReplyMIMEHeader:void#writeHttpReplyHeader:void

toolbox::lang::Classhttp::ReceiverLoop

-sockets_:std::vector<int>-clientIP_:std::map<int,std::string,std::less<int>>-clientHost_:std::map<int,std::string,std::less<int>>-address_:pt::Address::Reference-fdset_:fd_set-allset_:fd_set-maxfd_:int-current_:int-listenfd_:int-accepted_:int-nochannels_:int-nready_:int-listener_:pt::SOAPListener*-cgiListener_:xgi::Listener*-process_:toolbox::task::ActionSignature*-httpRootDir_:std::string-logger_:Logger-policy_:pt::HTAccessSecurityPolicy*-is_:xdata::InfoSpace*-aliasName_:std::string-aliasPath_:std::string

+ReceiverLoop+~ReceiverLoop+receive:int+send:void+disconnect:void+connect:void+close:void+process:bool+addServiceListener:void+removeServiceListener:void+removeAllServiceListeners:void+activate:void-accept:int-isConnected:bool-isActive:bool-onRequest:void-reply:void-authenticateUser:bool-verifyAccess:bool-isBrowserSupported:bool


address:pt::Address::Reference

exception


http::ClientChannel

+ClientChannel+connect:void+disconnect:void+receive:int+send:void+close:void+isConnected:bool

pt::Addresshttp::Address

#url_:toolbox::net::URL*#service_:std::string



pt::PeerTransportReceiverhttp::PeerTransportReceiver

#loop_:std::vector<http::ReceiverLoop*>#logger_:Logger#is_:xdata::InfoSpace*

+PeerTransportReceiver+~PeerTransportReceiver+createAddress:pt::Address::Reference+createAddress:pt::Address::Reference+addServiceListener:void+removeServiceListener:void+removeAllServiceListeners:void+isServiceSupported:bool+config:void


pt::PeerTransportSenderhttp::PeerTransportSender

#sync_:BSem*#mutex_:BSem*#logger_:Logger

+PeerTransportSender+~PeerTransportSender+createAddress:pt::Address::Reference+createAddress:pt::Address::Reference+isServiceSupported:bool+getMessenger:pt::Messenger::Reference


http::Channel

#sockaddress_:sockaddr_in#sockaddressSize_:socklen_t#socket_:int#mutex_:BSem

+Channel+~Channel+connect:void+disconnect:void+receive:int+send:void+close:void+isConnected:bool+lock:void+unlock:void

xdaq::ApplicationPeerTransportHTTP

-pts_:http::PeerTransportSender*-ptr_:http::PeerTransportReceiver*-aliasName_:xdata::String-aliasPath_:xdata::String

+XDAQ_INSTANTIATOR:+PeerTransportHTTP+~PeerTransportHTTP

pt::SOAPMessengerhttp::SOAPMessenger

-channel_:http::ClientChannel*-local_:pt::Address::Reference-destination_:pt::Address::Reference

+SOAPMessenger+~SOAPMessenger+send:xoap::MessageReference


http

0..1

0..1

i2o::Listener

+getService:std::string+processIncomingMessage:void

service:std::string

LISTENER:classstruct

i2o::Method

+obj_:LISTENER*+func_:void+fid_:unsigned long

+type:std::string+invoke:void+fid:unsigned long+fid:void

toolbox::lang::Methodi2o::MethodSignature

+invoke:void+fid:unsigned long

i2o::Messenger

+getService:std::string

service:std::string

exception

Exception

i2o::utils::Dispatcher

#logger_:Logger#registry_:xdaq::ApplicationRegistry*#context_:xdaq::ApplicationContext*#addressMap_:i2o::utils::AddressMap*

+Dispatcher+~Dispatcher+processIncomingMessage:void

pt::BinaryListener


pt::BinaryMessenger

+~BinaryMessenger+send:void+getLocalAddress:pt::Address::Referenc

localAddress:pt::Address::ReferencedestinationAddress:pt::Address::Referenc

pt::Listener

+getService:std::string

service:std::string

pt::Messenger

+~Messenger+getService:std::string

service:std::string

i2o::utils::AddressMap

-instance_:AddressMap*-tidToApplication_:std::map<unsignedlong,xdaq::ApplicationDescriptor*,std::less<unsignedlong>>-applicationToTid_:std::map<xdaq::ApplicationDescriptor*,unsignedlong,std::less<xdaq::ApplicationDescriptor*>>

+getApplicationDescriptor:xdaq::ApplicationDescriptor*+getTid:unsigned long+setApplicationDescriptor:void+removeTid:void+clear:void+getInstance:AddressMap*+destroyInstance:void

instance:AddressMap*

SOAPmessages

transport layer

0..*

0..1

1..*

0..1

0..1

0..1

0..1

xdaq::WebApplicationxdata::ActionListener

PeerTransportTCP

-pts_:tcp::PeerTransportSender*-ptr_:tcp::PeerTransportReceiver*-autosize_:xdata::Boolean-maxPacketSize_:xdata::UnsignedLong-poolName_:xdata::String

+XDAQ_INSTANTIATOR:+PeerTransportTCP+~PeerTransportTCP+Default:void+apply:void+failurePage:void-actionPerformed:void-reset:void

tcp::Transmitter

+isConnected_:bool+retry_:unsigned long

+Transmitter+~Transmitter+connect:void+disconnect:void+receive:int+send:void+close:void+isConnected:bool

pt::Addresstcp::Address




i2o::Messengertcp::I2OMessenger

-pt_:tcp::PeerTransportSender*-channel_:tcp::Channel*-destination_:pt::Address::Reference-local_:pt::Address::Reference

+I2OMessenger+~I2OMessenger+send:void


tcp::Channel

#sockaddress_:sockaddr_in#sockaddressSize_:socklen_t#socket_:int

+Channel+~Channel+connect:void+disconnect:void+receive:int+send:void+close:void+isConnected:bool

exception


pt::PeerTransportSenderTask

tcp::PeerTransportSender

-outputQueue_:toolbox::squeue<tcp::PostDescriptor>-activated_:bool-done_:bool

+PeerTransportSender+~PeerTransportSender+createAddress:pt::Address::Reference+createAddress:pt::Address::Reference+isServiceSupported:bool+getMessenger:pt::Messenger::Reference+svc:int+post:void+start:void+stop:void


toolbox::lang::Classtcp::ReceiverLoop

-sockets_:std::vector<int>-address_:pt::Address::Reference-fdset_:fd_set-allset_:fd_set-timeout_:timeval-done_:bool-stop_:bool-maxfd_:int-current_:int-listenfd_:int-accepted_:int-nochannels_:int-nready_:int-autoSize_:bool-maxPacketSize_:unsigned long-listener_:i2o::Listener*-pool_:toolbox::mem::Pool*-manager_:toolbox::mem::MemoryPoolFactory*-process_:toolbox::task::ActionSignature*-logger_:Logger

+ReceiverLoop+~ReceiverLoop+receive:int+send:void+disconnect:void+connect:void+close:void+process:bool+addServiceListener:void+removeServiceListener:void+removeAllServiceListeners:void+activate:void+cancel:void-accept:int-isConnected:bool-isActive:bool-onRequest:void-safeReceive:void


address:pt::Address::Reference pool:toolbox::mem::Pool* maxPacketSize:unsigned long autoSize:bool

structtcp::PostDescriptor

+ref:toolbox::mem::Reference*+channel:tcp::Channel*+handler:toolbox::exception::HandlerSignature*+context:void *

pt::PeerTransportReceivertcp::PeerTransportReceiver

#autoSize_:bool#maxPacketSize_:unsigned long#loop_:std::vector<tcp::ReceiverLoop*>#pool_:toolbox::mem::Pool*#logger_:Logger

+PeerTransportReceiver+~PeerTransportReceiver+createAddress:pt::Address::Reference+createAddress:pt::Address::Reference+addServiceListener:void+removeServiceListener:void+removeAllServiceListeners:void+isServiceSupported:bool+config:void+start:void+stop:void

type:pt::TransportType protocol:std::string supportedServices:std::vector<std::string> pool:toolbox::mem::Pool* maxPacketSize:unsigned long autoSize:bool

DAPL

new development


Peer Transport DAPL (ptdapl)

GSI development 03/2006-05/2006• C++ wrapper class for uDAPL C library was used• peer transport tcp as „template“ (starting point)• uDAPL buffers managed within XDAQ memory pool• avoids memcopy and new buffer allocation for each send package:

• lookup if posted memory reference is known as send buffer• user code can write directly into uDAPL send buffer

• multiple threads for sending, releasing, and receiving buffer


Transport bandwidth B and latency τ

P: package sizeτ : transfer time („latency“)

Observation: τ linear with P: τ

B

P

P

P big: network limit

P small: framework latency !


Benchmark summary

0100200

300400500600700

800900

1000

0 20 40 60 80 100 120 140 160Size [Kbyte]

Mby

tes/

sec

XDAQ: ptTCP, GEXDAQ: ptTCP, IBuDAPL: RDMA 4x4 (per node)XDAQ: ptDAPL, RoundTripXDAQ: ptDAPL, 1x1 (sender)

955 MB/s

plain uDAPL

234 MB/s

XDAQ, I2O, TCP over IB

905 MB/s

XDAQ, I2O, ptDAPL


XDAQ: web server as user interface

hyperdaq web interface


XDAQ: SOAP messaging for Controls

xrelay: send SOAP control messages

• example commands:Configure, Enable, Halt, Reset,..• any new commands may be defined• web interface not suited as

real control system UI!• XDAQ applications may exchange SOAP messages with other UI


Control system with XDAQ?

• Designed class hierarchy for all (test) applications:FDaqApplication

<- FDaqDataNode<- FDaqDataSource, FDaqDataDrain, FDaqRoundTrip,...

• FDaqApplication provides common controls features:• state machine• process variable set (“XDAQ infospace”) • interfaces: web server, SOAP, plain http request• ? may offer other mechanisms (EPICS ioc? DIM server? ...)

• Test: external JAVA application as controls GUI• Swing widgets, developed with ECLIPSE

• SOAP to exchange commands and status information


JAVA Control GUI study

• SOAP (de-)serialising well supported by XDAQ and JAVA• Native Swing widgets not sufficient• GUI needs active request to FDaqApplication for status refresh

(command-response mechanism instead publisher-subscriber)


Active Buffer Boards

• PCI express boards with virtex4 FPGA (developed in Mannheim, A.Kugel): Ready 1/2007?

• Purpose: • Receives data from readout / concentrator boards• Pre-processing (sorting?, time frames or event tagging?)• Forward data frames to filter nodes:

DMA to PC memory -> IB sender -> IB receiver -> Event filter appl.


Active Buffer Boards (2)

• Integration to XDAQ (JA):• Linux kernel 2.6 module for generic PCI driver (G.Marcus) • XDAQ Hardware Access Library as C++ user space

interface• Excercise this with GSI pcigtb board (under work...)

• FPGA programming (SL)• required to change test set up (ABB as data generator)• Excercise this with simple FPGA development board

• Obtain ATLAS RobIn board from Mannheim before first ABB


Summary and Outlook

• CBM DAQ requires fast builder network: InfiniBand test cluster • uDAPL: data transport tests with different traffic patterns• performance on bigger InfiniBand clusters? tests still to do!• AB boards: FPGA programming to learn and to do• XDAQ as software framework:

• powerful, many features! basic functionalities tested • data transport: I2O implementation for uDAPL done • control System? Work on JAVA GUI with SOAP? to do!• hardware access? – AB boards with HAL to do!• cluster configuration? process control? scalability?...

EEMeeting 23.10.06 XDAQ & InfiniBand - J.Adamczewski, H.G ... · EEMeeting 23.10.06 XDAQ &...

Documents

Transcript of EEMeeting 23.10.06 XDAQ & InfiniBand - J.Adamczewski, H.G ... · EEMeeting 23.10.06 XDAQ &...