1 RDMS CMS Computing: Russian CMS Tier-2 and Tier-1 for CMS at JINR llya Gorbunov and Sergei Shmatov...

1

RDMS CMS Computing:Russian CMS Tier-2 and Tier-1 for CMS at JINR

llya Gorbunov and Sergei ShmatovJoint Institute for Nuclear Research, Dubna

[email protected]

School on JINR/CERN GRID and Advanced Information Systems

“Joint Institute for Nuclear Research, Dubna

May 14-18, 2012

mailto:[email protected]

2

Compact Muon Solenoid goals and performance participation of Russian and Dubna Member State Institutions data taking, triggering and selection data flows

CMS Computing Model data model data flows data processing Tier-1/Tier-2 requirements and resources

RDMS Computing Model RDMS CMS Tier-2s (readiness, job status, data transfers) computing for remote operations (JINR ROC) CMS Tier-1 at Dubna (plans and prospects)

Ilya Gorbunov and Sergei Shmatov “RDMS CMS Computing”, AIS2012, Dubna, May 18, 2012

Brief Intro:What does physics require

from Computing?

3Ilya Gorbunov and Sergei Shmatov “RDMS CMS Computing”, AIS2012, Dubna, May 18, 2012

Why we need more accelerators and experiments ?

Being the remarkable event on the long way of searching of the true, Standard Model of interactions is powerless

to answer on very important questions:

• Origin of masses of fundamental blocks (quarks, leptons) • A reason for three generations of matter fermions• A reason of domination matter over antimatter and formation of observable Universe structure• A source of dark mater in the Universe • Possibility of high energy unification of three forces

We hope we will be able to answer on these questions at least partially with next generations of accelerators

And after all, what about gravity?


Open Questions:

Mass: why do fundamental particle have mass and why these masses are such different?

Antimatter: why at the present days matter dominates antimatter?

Dark matter: where is concentrated the hidden mass of Universe?

World geometry: are there extra dimensions? bulk volume geometry?

New particles: do they exist? (sparticles, extra gauge bosons, graviton states etc)

QCD matter: what are the properties of QCD at extreme high densities, does QGP exist?

Standard model: validity bounds -> high precision tests of SM !

Open Questions:

Mass: why do fundamental particle have mass and why these masses are such different?

Antimatter: why at the present days matter dominates antimatter?

Dark matter: where is concentrated the hidden mass of Universe?

World geometry: are there extra dimensions? bulk volume geometry?

New particles: do they exist? (sparticles, extra gauge bosons, graviton states etc)

QCD matter: what are the properties of QCD at extreme high densities, does QGP exist?

Standard model: validity bounds -> high precision tests of SM !

LHC Physics Perspectives


Compact Muon Solenoid


7

Compact Muon Solenoid (CMS)+ TOTEM (TOTal Elastic and diffractive cross section Measurement)

A Toroidal LHC ApparatuS (ATLAS)

A Large Ion Collider Experiment (ALICE)

LHC-B

LHC Experiments

LHCf

CMS at LHC

CMS

CERNBuilding 40

14 km15 min

1234

1233

1232

10)(

10)(

10)(

147)(

ccmhighL

ccmlowL

ccmupstartL

TeVppS

112810

/ 62.5)(

ccmL

nucleonTeVPbPbS


Compact Muon Solenoid - CMS

Muon Chambers Tracker Electromagnetic calorimeter, ECAL

Hadron calorimeter HCAL

Forward Calorimeter, HF

Superconducting Coildiameter 6 m, length 13 m Return Yoke

Silicon Tracker

2.7% 0.150.5% E

E EE

PbWO4 ECAL

sampling brass HCAL

1205%E

E E

MUON Chambers

Detector subsystems are designed to measure: the energy and momentum of photons, electrons, muons, jets, missing ET up to a few TeV)%(15~ TeVpp

pT

T

T

GeVp

p

T

T 100@%5.10.1~

Weight 12 500 tDiameter 15.00 mLength 21.6 mMagnetic Field 3.9 Tesla

Weight 12 500 tDiameter 15.00 mLength 21.6 mMagnetic Field 3.9 Tesla 9Ilya Gorbunov and Sergei Shmatov “RDMS CMS Computing”, AIS2012, Dubna, May 18, 2012

Russian and Dubna Member State CMS Collaboration

CERN

France

Italy

UK

Switzerland

USA

Austria

Finland

Greece

Hungary

Belgium

Poland

Portugal

Spain

Pakistan

Georgia

Armenia

Ukraine

Uzbekistan

Cyprus

Croatia

China, PR

Turkey

Belarus

Estonia

India

Germany

Korea

Russia and JINR Member States

Bulgaria

China (Taiwan)

Iran

Serbia

New-Zealand

Brazil

Ireland

Mexico ColombiaLithuania

RDMS CMS:

2310 authors38 countries175 institutions

the RDMS CMS Collaboration was founded

in Dubna in September 1994


JINR Participation in CMS ConstructionJINR participates in the CMS in a framework of the RDMS CMS

Collaboration

RDMS bearsFull Responsibility

JINRParticipates

SE

HE

ME1/1


Cross-sections of physics processes at LHC Energy

The fraction of “important” events is ~ 10-8 of total event flow !

The fraction of events from “new physics” is 10-10 - 10-15 (Higgs, SUSY, BSM)

• A very small part of events is a proper for physics analysis• No opportunity to store and to process the full data flow

Triggering and Selection12Ilya Gorbunov and Sergei Shmatov “RDMS CMS Computing”, AIS2012, Dubna, May 18, 2012

OnlineOnline:: Trigger System Trigger System

• Multilevel trigger• rejection of background (filtration of events)• reduction of data volume• on-line reduction factor ~107

• Trigger menu• selection of required events

The data flow is reduced from 1000 TB/sec (1TB=1012 Bytes) at input of the first-level trigger down to 100-200 MB/sec on output of the third-level trigger (to record on disks or tapes for further analysis)

Level 1 - Special Hardware

Level 2 - Embedded Processors

40 MHz

40 MHz (1000 TB/sec)

(1000 TB/sec)

Level 3 – Farm of commodity CPUs

75 KHz 75 KHz (75 GB/sec)

(75 GB/sec)5 KHz5 KHz (5 GB/sec)

(5 GB/sec)100 Hz 100 Hz (100 MB/sec)

(100 MB/sec)

Data Recording &

Data Recording &

Offline Analysis

Offline AnalysisCMS event – 1-2 MB L1 trigger – 75 GB/с HLT trigger – 0.1 GB/с total flow ~ 3 PByear


Level-1 Event Storage kHz MByte MByte/s ATLAS 100 1 100 CMS 100 1 100 LHCb 400 0.1 20 ALICE 1 25 1500

LHC Data Flows

~ 3 PB/year

~ 3 PB/year


The CMS Computing Requests

Event rate to tape: 100 Hz (109 events/year)

Days of running/year: 200

Data written to tape: ~ 3 PetaBytes/Year

Monte Carlo events: 9∙108

CPU (Tier-0 + Tier-1s + Tier-3s): ~ 26 000 kSI2k

Typical networks: 1 - 10 Gbps

Users: ~ 1000 physicists distributed all over the world

1 billion people surfing the Web

105

104

103

102

Level 1 Rate (Hz)

High Level-1 Trigger(1 MHz)

High No. ChannelsHigh Bandwidth(500 Gbit/s)

High Data Archive(PetaByte)

LHCB

KLOE

HERA-B

CDF II

CDFH1

ZEUS

UA1

LEP

NA49

ALICE

Event Size (bytes)

104 105 106

ATLASCMS

106

107

The full processing and analysis can not be performed by the standard tool kits

It is essential to use Distributed Computer Resources

The full processing and analysis can not be performed by the standard tool kits

It is essential to use Distributed Computer Resources


CMS Computing Model


17

Tier-0 (CERN):• accepts data from the CMS Online Data Acquisition and Trigger System (TriDAS)

• archives RAW data • the first pass of reconstruction and performs Prompt Calibration

• data distribution to Tier-1

Tier-1 (11 centers):• receives a data from the Tier-0 • data processing (re-reconstruction, skimming , calibration etc)• distributes data and MC to the other Tier-1 and Tier-2• secure storage and redistribution for data and MC

Tier-2 (>200 centers):• simulation• user physics analysis

Tier Structure of GRID Distributed Computing:

Tier-0/Tier-1/Tier-2/Tier-3


CMS GRID Distributed Computing

Tier-0 at CERN

Tier-1 (7 centers): CERN , KIT (Germany), PIC (Spain), CCIN2P3 (France), CNAF (Italy), ASGC (Taipei), RAL (UK), FNAL (US) + two sites under construction at Dubna and India

Tier-2 (85 centers): 7 in Russian and DMS


CMS Data ModelEvent Format Contents Purpose Event Size (MB) DAQ-RAW Detector data from front end electronics + L1 trigger result. Primary record of physics event. Input to

online HLT 1-1.5

RAW Detector data after online formatting, the L1 trigger result, the result of the HLT selections (HLT trigger bits

Input to Tier-0 reconstruction. Primary archive of events at CERN.

1.5

RECO Reconstructed objects (tracks, vertices, jets, electrons, muons, etc.) and reconstructed hits/clusters

Output of Tier-0 reconstruction and subsequent rereconstruction passes..

0.25

AOD Subset of RECO. Reconstructed objects (tracks, vertices, jets, electrons, muons, etc

Physics analysis, limited refitting of tracks and clusters

0.05

TAG Run/event number, high-level physics objects, used to index Rapid identification of events 0.01

FEVT Full Event: Term used to refer to RAW+RECO together multiple 1.75

GEN Generated Monte Carlo event - - SIM Energy depositions of MC particles in detector (sim hits). - -

DIGI Sim hits converted into detector response (the same as the RAW) - 1.5


https://twiki.cern.ch/twiki/bin/view/CMSPublic/WorkBookDataFormats?sortcol=0;table=1;up=0




~ Pbyte/sec

10 Gbps

~ 100-1500 MBytes/sec

10 Gbps

0.1 - 10 Gbps

2-10 Gbps

CMS Data Flows (I)


CMS Data Flows (II)T0 to T1:

scheduled, time-critical, continuous during data-taking reliable transfer needed for fast aaccess to new data, and to ensure that data is stored safely

T1 to T1: redistributing data, generally after reprocessing (e.g. processing with improved algorithms)

T1 to T2: Data for analysis at Tier-2s

Detector data flow

Monte-Carlo data flow

T2 to T1: data transfer simulation events

T1 to T1/T2: data transfer re-processed simulation events (e.g. processing with improved algorithms)


Data Processing (II)

Calibration Data Base

RAW Data Reconstruction Event Selection

Physics Analysis

Theory/ Monte Carlo Simulation

CMS-PAS-HIG-11-032CMS-PAS-HIG-11-032

Final Physics Results!


CMS Data Processing (II)


CMS Computing Resources


RDMS CMS Computing Model


RDMS CMS Grid Sector

CERNTier-0

CMS RDMS Tier-1

RDMS Grid Sector

Russia DMS

Tier-2

JINR

Dubna

Tier-2

ITEP

Moscow

Tier-2

SINP MSU

Moscow

Tier-2

INR

Troitsk

Tier-2

FIAN

Moscow

Tier-2

IHEP

Protivino

Tier-2

PNPI

St.Petersburg

Tier-2

KIPT

Kharkov

Tier-2

Minsk

Tier-2

Erevan

Tier-2

Sofia

Tier-2

Tbilisi

To provide data management and data transfer to RDMS Tier-2 and data processing in the framework of RDMS Physics Tasks the special СMS RDMS Tier-1 was created in CERN

Dubna CMS Tier-1

and

other CMS Tier-1 centres


CMS T2 requirements

Basic requirements to CMS VO T2 sites for Physics group hosting:• info on contact persons responsible for site operation• site visibility (BDII) and CMS VO support• availability of CMSSW actual version• daily SAM availability > 80%• daily CMS job robot efficiency > 90% • regular file transfer test “OK”• Certified links with CMS T1: 2 up and 4 down• disk space ~ 150-200 TB for: - central space (~30 TB) - analysis space (~60-100 TB) - MC space (~20 TB) - local space (~30-60 TB)

- local CMS users space (~2 TB per user)• CPU resources ~ 3-4 KSI2K per 1 TB disk space, 2GB RAM memory per job


RDMS CMS Tier-2 Availability


Status of RDMS Site Metrics


Normalized CPU time (in 1K.SI2K.Hours) consumed by the CMS sites September, 2010 – August, 2011)

In total 283,729,684 K.SI2K.Hours


Total Number of CMS Jobs at RDMS CMS sites

1 747 182 jobs

2 253 116 jobs

5.8%

8.7%


Job Summary at RDMS CMS Sites as in the CMS Dashboard

Job Status per Sites

Site Activity Job Efficiency


PHEDEX Transfer Rate & Transfer Volumes for RDMS CMS sites


RDMS CMS T2 association

Now Future interest

Analysis Groups

Exotica: T2_RU_JINR Exotica: T2_RU_INR

HI: T2_RU_SINP QCD: T2_RU_PNPI

Electroweak: T2_UA_KIPT Top: T2_RU_SINP

FWD: T2_RU_IHEP

Object/Performance Groups

Muon: T2_RU_JINR e-gamma-ECAL: T2_RU_INR

JetMET-HCAL: T2_RU_ITEP


JINR Monitoring and Analysis Remote Centre

Monitoring of detector systems Data Monitoring / Express Analysis Shift Operations (except for run

control) Communications of JINR shifter

with personal at CMS Control Room (SX5) and CMS Meyrin centre

Communications between JINR experts and CMS shifters

Coordination of data processing and data management

Training and Information

Monitoring of detector systems Data Monitoring / Express Analysis Shift Operations (except for run

control) Communications of JINR shifter

with personal at CMS Control Room (SX5) and CMS Meyrin centre

Communications between JINR experts and CMS shifters

Coordination of data processing and data management

Training and Information


RDMS CMS Computing Model:Tier-1 at Dubna


WLCG Tier1 center in Russia

• Proposal to create the LCG Tier1 center in Russia (official letter by Minister of Science and Education of Russia A. Fursenko has been sent to CERN DG R. Heuer in March 2011).

• The corresponding point to include in the agenda of next 5x5 meeting Russia-CERN (October 2011)

- for all four experiments ALICE, ATLAS, CMS and LHCb

- ~10% of the summary Tier1 (without CERN) resources

- increase by 30% each year

- draft planning (proposal under discussion) to have prototype in the endof 2012, and full resources in 2014 to meet the start of

next working LHC session.

• Proposal to create the LCG Tier1 center in Russia (official letter by Minister of Science and Education of Russia A. Fursenko has been sent to CERN DG R. Heuer in March 2011).

• The corresponding point to include in the agenda of next 5x5 meeting Russia-CERN (October 2011)

- for all four experiments ALICE, ATLAS, CMS and LHCb

- ~10% of the summary Tier1 (without CERN) resources

- increase by 30% each year

- draft planning (proposal under discussion) to have prototype in the endof 2012, and full resources in 2014 to meet the start of

next working LHC session.


Joint NRC "Kurchatov Institute" – JINR Tier1 Computing Centre

Project: «Creation of the automated system of data processing for experiments at the Large Hadron Collider (LHC) of Tier1 level and maintenance of Grid-services for a distributed analysis of these data»

Terms: 2011-2013

Type of project: R&D

Cost: federal budget - 280 million rubles (~8.5 MCHF), extrabudgetary sources - 50% of the total cost

Leading executor: RRC KI «Kurchatov institute» for ALICE, ATLAS, and LHC-B

Co-executor: LIT JINR (Dubna) for the CMS experiment

Project goal: creation in Russia of a computer-based system for processing experimental data received at the LHC and provision of Grid-services for a subsequent analysis of these data at the distributed centers of the LHC global Grid- system.

Core of the proposal: development and creation of a working prototype of the first-level center for data processing within the LHC experiments with a resource volume not less than 15% of the required one and a full set of grid-services for a subsequent distributed analysis of these data.


Tier-1 Services

Security (GSI)Computing Element (CE)Storage Element (SE)Monitoring and AccountingVirtual Organizations (VOMS)Workload management (WMS) Information service (BDII)File transfer service (FTS + PhEDEx)VOBOXSQUID / Frontier ServersCMS user services (Reconstruction Services, Analysis Services etc)


CMS Tier-1 Requirements

• input flow ~ 7 GB/s

• output flow ~ 3-4 GB/s

• CPU – 2.5 MSI2k

• disks – 1.2 PB

• tapes for archives and secure store – 2.8 PB

• data loss – 10 GB per 1 PB

• access to data ~ 800 MB/s

• I/O computing node bandwidth ~ GB


JINR Resources

JINR-LCG2Tier2 Sept. 2011 April 2012 Dec. 2012 Dec. 2013

CPU(HEP-SPEC06)

16 000 20 000 30 000 45 000

Disk(Tbytes)CMSATLASALICE

635240240275

995360 360275

1800660 660480

300011001100800

Tape (Tbytes) - - - 5000

JINR-CMSTier1

Dec. 2012Prototype

Sep. 2014Start

CPU(HEP-SPEC06)

4 000 45 000

Disk(Tbytes)CMS

500 8 000

Tape (Tbytes) - 12 000


SUMMARY

RDMS Tier-2 sites are fully satisfied to CMS Computing Requirements are involved actively into data processing and analysis (~3.8% of all CMS resources)

New CMS Tier-1 site is under construction in Dubna (in 2012-2013 a prototype need to be designed and commmisined)

CMS Regional Operation Center is founded and operated in JINR for remote monitoring of detector systems and data express-analysis

1 RDMS CMS Computing: Russian CMS Tier-2 and Tier-1 for CMS at JINR llya Gorbunov and Sergei Shmatov...

Documents

Transcript of 1 RDMS CMS Computing: Russian CMS Tier-2 and Tier-1 for CMS at JINR llya Gorbunov and Sergei Shmatov...