OO Software and Data Handling in AMS
Computing in High Energy and Nuclear Physics
Beijing , September 3-7 , 2001
Vitali Choutko, Alexei Klimentov
MIT, ETHZ
A.Klimentov AMS software and data handling CHEP01
2
Outline AMS – particle physics experiment on
the international space station :
Data flow and AMS ground centers Software development Conditions and Tag Database Data Processing
AMS Detector STS91 precursor flight AMS ISS mission
A.Klimentov AMS software and data handling CHEP01
3
AMS : a particle physics experiment in space
PHYSICS GOALS :
Accurate, high statistics measurements of charged, cosmic ray spectra in space > 0.1GV
Nuclei and e- spectra measurement• The study of dark matter (90% ?)
• Determination of the existence or absence of antimatter in the Universe Look for negative nuclei
• The study of the origin and composition of cosmic rays Measure isotopes D, He, Li,
Be…
+
A.Klimentov AMS software and data handling CHEP01
4
A.Klimentov AMS software and data handling CHEP01
5
Magnet : Nd2Fe14BTOF : trigger, velocity and ZSi Tracker : charge sign, rigidity, Z Aerogel Threshold Cerenkov : velocityAnticounters : reject multi particle events
Results :
Anti-matter search : He / He = 1.1x 10Charged Cosmic Ray spectra Pr, D, e- , He, NGeomagnetic effects on CR under/over geomagnetic cutoff components
10 events recordedTrigger rates 0.1-1kHzDAQ lifetime 90%
8
-6_
Precursor flight :
+
A.Klimentov AMS software and data handling CHEP01
6
A.Klimentov AMS software and data handling CHEP01
7
AMS on ISS , 3 years in space
Separate e- from p,p+_
up to 300 GeV
He, He, B, C…
3 4
e-,up to 1000 GeV
+
A.Klimentov AMS software and data handling CHEP01
8
Tracker
TRD
TOF (S1, S2)
MagnetHe Vessel
RICH
ECAL
TOF (S3, S4)
R. BeckerAugust 15, 2001
Veto Counter
USS
Radiators
Electronics
AMS 02 In Cargo Bay
separate e from p,p up to 300 GeV
± −
8Layers
e,to1000GeV±
He,He,B,C,...3
4
AMSonISSfor3years
A.Klimentov AMS software and data handling CHEP01
9
Monitoring& science data
Stored data
Real-timeDataH&S
Real-time & “Dump” data
Real-time, “Dump”, & White Sand’s LOR playback
AMS
ACOP
High Rate Frame MUX
White Sand, NM facility
MSFC, Al
Payload DataService system
Telescience centers
Exte
rnal C
om
mu
nic
ati
on
s
GSELongTerm
ShortTerm
PayloadOperationsControlCenter
ScienceOperationsCenter
ISS to Remote AMS Centers Data Flow
ISSNASAGroundInfrastructure
RemoteAMS Sites
H&SMonitoringScienceFlight ancillarydata
Real-time & “dump”
NearReal-time
File transferplayback
A.Klimentov CHEP01 10
AMS Ground Centers
Science Operations Center
POCCPOCCPOIC@MSFC AL
AMS Remotecenter
RT data CommandingMonitoringNRT Analysis
NRT Data Processing Primary storage Archiving DistributionScience Analysis
MC productionData mirror archiving
Exte
rnal
Com
mu
nic
ati
on
s
ScienceOperationsCenter
XTermHOSC Web Server and xterm
TReK WS
commandsMonitoring, H&S dataFlight Ancillary dataAMS science data (selected)
TReK WS“voice”loop
Video distribution
Production Farm
AnalysisFacilities
PC Farm
Data Server
AnalysisFacilities
GSE D S
A eT rA v e r
GSEBuffer dataRetransmitTo SOC
AMS Station
AMS Station
AMS Station
GSE
MC production
cmds archive
AMS Data, NASA data,
metadata
A.Klimentov AMS software and data handling CHEP01
11
AMS SW development
Been started mid 1996 : basic decisions : new code C++ only (though we had a large part
of legacy SW written on Fortran) Existing libraries (CERNLIB, Geant, etc)
incorporated via C/Fortran interface (R.Burow) transient and persistent classes are separated
with implementing of copy member functions Decide to use Root and HBOOK for histogramming
and data visualization
A.Klimentov AMS software and data handling CHEP01
12
AMS SW development (cont’d) Use different persistency solutions for
various type of data :Flat files for the raw dataNtuples and Root files for ESDRelational Database (Oracle) tables for file cataloguesRelational Database (Oracle) [Objectivity up to Sep 1998]
o Event Tags
o Calibration data
o Slow control data
o NASA ancillary data
o Various catalogues (processing history, etc)
A.Klimentov AMS software and data handling CHEP01
13
Tag Storage with Oracle RDBMS
Tag is an unsigned 32 bit integer containing 16, 1 to 5 bit long parameters such as charge, momentum sign, ß,…
Model :
Query : retrieve tags with 3 parameters satisfied to the given limits (query taken from the “real analysis chain”)
Data stored on Raid array connected to AS4100 (quad-CPU rated at 600MHz, 2GB RAM)
• Flat files – 2400 files, one file per DAQ run, tags are stored as an array of unsigned int.• RootN - 10 files, each file with ~240 trees, one tree per DAQ run with single branch (tag) per tree• RootS - 10 files, each file with ~240 trees, one tree per DAQ run , having 16 branches, every parameter stored in a dedicated branch• OracleN - table with 10 partitions and 1 column, mapping tag to a column• OracleI - table with 10 partitions and 1 column with 16 bitmap indices, mapping tag to a column• OracleS – table with 10 partitions and 16 columns, every parameter mapping to a column
A.Klimentov AMS software and data handling CHEP01
14
Oracle RDBMS to store AMS tags
Method Size Gbyte
Query Time (sec)
Total write time (sec)
Record write time (sec)
Flat Files 1.4 600 - -
RootN 0.9 700 2168 22
RootS 1.2 112 2200 66
OracleN 3.4 1420 6600 66
OracleS 6.6 600 6600 66
OracleI 3.4 3.9 6600+500 661)
1) 500 sec to build indices for 100M tags
A.Klimentov AMS software and data handling CHEP01
15
Design of the Conditions Database Collection of Time Dependent Values (TDVs)
Primary access keys : name, id, validity interval Secondary key : insert time Major Components : table of names and ids, default TDVs,
TDVs Applications : Loading data into database Fetching conditions during event reconstruction Management utilities (TDV browser)
•Name, id•Validity begin, validity end time•Insert time•Array of unsigned integers (size 100 byte – 8 Mbyte)
A.Klimentov AMS software and data handling CHEP01
16
AMS Conditions Database Initially Objectivity, then flat files, now Oracle Performance test for
TDV Total Records
Flat file size (Mbyte)
Oracle table size (Mbyte)
Record write time (msec)
TOF Temperature 9835 1.9 2.8 17
Tracker pedestals (a) 330 36.3 45.2 75
Tracker pedestals (b) 330 36.3 44.9 103(a) BLOB array is stored inside the table, (b) - outside
TOF temperature (many short records)Tracker pedestals (small amount of large records)
A.Klimentov AMS software and data handling CHEP01
17
Oracle RDBMS to store Tags and TDVs
Currently 8 Gbyte is stored in the Conditions DB (115 different TDV types)
100 million event tags are stored in Tag DB Oracle RDBMS performance and functionality
satisfy AMS requirements. Using of bitmap indices for tags improves query time dramatically.
The current implementation works with distributed CORBA technology. It allowes to reduce the number of database clients and machine loading.
A.Klimentov AMS software and data handling CHEP01
18
producers
producers
Raw data
server
server
Oracle RDBMS
Conditions DB
Tag DB
Active Tables :Hosts, Interfaces,
Producers, Servers
Catalogues
server
server
server
server
server
server
Nominal Tables
Hosts, InterfacesProducers, Servers…
ESD
ESD
ESDESD
ESD
server
Raw data
{I}{II}
{III}
{IV}
{V}
{VI}
{VI}
• {I} submit 1st server•{II} “cold” start•{III} read “active” tables (available hosts, number of servers, producers, jobs/host)•{IV} submit servers•{V} get “run”info (runs to be processed, ESD output path)• {VI} submit producers (LILO, LIRO,RIRO…)•Notify servers
AMS Production
A.Klimentov AMS software and data handling CHEP01
19
AMS Production Highlights Stable running for more than 1 month Average efficiency 95% (98% without Oracle) Processes communication and control via Corba LSF for process submission Run Oracle server on AS4100 Alpha and Oracle clients on
Linux. Oracle RDBMS
Tag DB with 100M entries Conditions DB with 100K entries Bookkeeping
Production status Runs history File catalogues
Top Related