Port d’Informació Científica The LHC Computing Grid Project in … · 2010. 6. 16. ·...

29.03.2003 M. Delfino / PIC / The LCG-ES Coordinated Project - Presentation to RECFA 1

Portd’Informació

Científica

The LHC Computing Grid Project in Spain(LCG-ES)

Presentation to RECFA

Prof. Manuel DelfinoCoordinating Principal Investigator, LCG-ES

Physics Department, Universitat Autònoma de BarcelonaDirector, Port d’Informació Científica (PIC)*

* Collaboration of the Generalitat de Catalunya, IFAE and Universitat Autònoma de Barcelona


Portd’Informació

CientíficaLHC is multi-Petabyte, Tera-object data-intensive scientific computing

Long Term Tape Storage EstimatesLong Term Tape Storage Estimates

CurrentCurrentExperimentsExperiments COMPASSCOMPASS

LHCLHC

002'0002'0004'0004'0006'0006'0008'0008'000

10'00010'00012'00012'00014'00014'000

1995

1995

1996

1996

1997

1997

1998

1998

1999

1999

2000

2000

2001

2001

2002

2002

2003

2003

2004

2004

2005

2005

2006

2006

YearYear

TeraBytesTeraBytes

Accumulation: 10 PB/yearAccumulation: 10 PB/yearSignal/Background up to 1:10Signal/Background up to 1:101212


Portd’Informació

Científica Complex LHC data = more CPU/byte

Estimated CPU Capacity required at CERN

0

1,000

2,000

3,000

4,000

5,000

1998

1999

2000

2001

2002

2003

2004

2005

2006

2007

2008

2009

2010

Jan 2000:3.5K SI95

LHCLHC

K SI95

Moore’s law –some measure of the capacity technology advances provide for a constant number of processors or investment


Portd’Informació

CientíficaLHC analysis will be driven by the demands

of ~6000 physicists worldwide

DetectorDetector

Raw dataRaw data

EventReconstruction

EventEventReconstructionReconstruction

EventSimulation

EventEventSimulationSimulation Interactive

Data AnalysisInteractive

Data Analysis

ProcessedData

ProcessedData

1-100 GB/sec1-100 GB/sec

One ExperimentOne ExperimentOne Experiment35K SI9535K SI95

~200 MB/sec~200 MB/sec

250K SI95250K SI95

350K SI9564 GB/sec350K SI9564 GB/sec

500 TB500 TB

1 PB / year1 PB / year

~100 MB/sec~100 MB/sec

200 TB / year200 TB / year

analysis objects

Event Filter(selection &

reconstruction)

Event FilterEvent Filter(selection &(selection &

reconstruction)reconstruction)Event

SummaryData

Event Summary

Data

Batch PhysicsAnalysis

Batch PhysicsBatch PhysicsAnalysisAnalysis

0.1 to 10.1 to 1GB/secGB/sec

Thousands of scientists worldwide


Portd’Informació

CientíficaTechnology Domains for developing

solutions for LHC Data Analysis

DEVELOPER VIEW

GRID

FABRIC

APPLICATION “GRIDIFICATION”

USER VIEW


Portd’Informació

Científica From physical to virtual communities

User point of view:Virtual analysis communities

H→µµ

H→γγ

b-physics

LHC Computing

Grid Infrastructure

Physical setup

Transparent

Analysis Region

Transpare

nt

Analysis

Region

Transparent Analysis Region

man

uel.d

elfino

@ce

rn.ch

Institutional, National, Geographical Setups

Worldwide multi-cultural collaboration


Portd’Informació

Científica

ScientistsCollaborating

Field-specific middleware

General middleware

Grid Infrastructure

Benefits of Grid Infrastructure:

User-centric view

Collaborating Network and Computing

Infrastructures Interoperating Security, Registration, Authentication

and Authorization Frameworks


Portd’Informació

CientíficaThe Internet connects Computers

The Grid enables Virtual Communities

Application

Fabric“Controlling things locally”: Access to, & control of, resources

Connectivity“Talking to things”: communication (Internet protocols) & security

Resource“Sharing single resources”: negotiating access, controlling use

Collective“Managing multiple resources”: ubiquitous infrastructure services

User“Specialized services”: user- or appln-specific distributed services

InternetTransport

Application

Link

Intern

et Proto

col A

rchitectu

re

Grid Services Architecture

Adapted by permission from Foster and Kesselman


Portd’Informació

CientíficaFrom “farms” to “computing fabrics”

55

250250

0.80.888 24 *24 *

960 *960 *

6 *6 *

1.51.5

1212

LAN-WAN RoutersLAN-WAN Routers

0.80.8

0.80.8Data NetworkData Network

Farm NetworkFarm Network

* Data Ratein Gbps

* Data Ratein Gbps

10 Thousand dual-CPU boxes10 Thousand dual-CPU boxes

10 Thousand disk units10 Thousand disk units

Hundreds oftape drives

Hundreds oftape drives Real-time

detector dataReal-time

detector data

Grid InterfaceGrid Interface

Multi-Gigabit Ethernet switchesMulti-Gigabit Ethernet switches

Computing fabricat CERN (2006)[per experiment!]

Computing fabricat CERN (2006)[per experiment!]


Portd’Informació

CientíficaFor a fixed Quality of Service for the Users:- Costs will be dominated by

the level of Automation and the reliability of Certification procedures.

- This in turn is related to the Architecture of the solution.

(Hardware)

Trend: These two are increasing


Portd’Informació

Científica

IFIC (ATLAS)

UB (LHCb)IFAE (ATLAS)

UAM (ATLAS)CIEMAT (CMS)

USC (LHCb)IFCA (CMS)

• LCG-ES is a Coordinated Project of 7 Groups involved in 3 LHC experiments.It is a development and deployment project, not an R&D project. It is managed similar to an EU project.You can think of it as the overlapped “pre-production” ,“pre-assembly” and “development of Engineering Design Report” of the Computing Subdetector in 36 months.


Portd’Informació

Científica Background leading to LCG-ES

• Overall situation of Spanish groups in computing was analyzed and found to be too weak to make “just-in-time LHC ramp-up”– Analysis for LEP mostly done at CERN towards the end– General lack of technicians, system admins and sw professionals– Work on LHC dispersed, not fully recognized by LHC collaborations (in the

shadows of “US ATLAS/CMS”, “France”, etc…)

• On the other hand, we had a small string of high quality activities– Various contributions to online and slow controls.– FALCON quasi-online farm for ALEPH (+ Florida reprocess farm)– CERN RD-47 (HEP Data Processing using Commodity Components)– R&D in Object Databases for HEP Data– Participation in EU DataGrid and EU CrossGrid

• Initial boosts from “ad-hoc” mechanisms, lobbying and FPA– 2 multi-disciplinary farms (~200 nodes) at IFIC and IFCA– “Special Action” for 1 year of 500 K€ (about 50% materials, 50% personnel

& travel) loosely coordinated by Jesus Marco (IFCA).


Portd’Informació

Científica LCG-ES: High-Level Objectives

• Develop and test under realistic conditions a series of prototypes, increasingly more powerful and complex, of data processing systems for the LHC utilizing Grid techniques.

• Research the possibilities of sharing complex data processing infrastructures between several communities (those of ATLAS, CMS and LHCb in our case).

Coordinated within the global LCG project plan and the Data Challenges of the experiments


Portd’Informació

Científica Finer-grain objectives (1 of 3)

1. Establish a solid base of knowledge and experience in Grid distributed computing in order to ensure participation of Spanish groups in LHC data analysis through a shared infrastructure.

2. Establish a solid base of knowledge and experience in management and processing of the large data volumes (Petabyte level) that will be generated by the LHC experiments.

3. Actively participate in Grid computing projects in order to ensure the deployment in Spain of a Grid and DataGrid infrastructure which is integrated internationally.


Portd’Informació


4. Take active part in development of Grid computing within the LHC experiments and contribute to the LHC Computing Grid Technical Design Report.

5. Create Monte Carlo production centers such that groups can contribute to the official productions of the LHC experiments. In particular:

• Generate official MC consisted with Spanish participation in the LHC program and the objectives of groups within their experiment.

• Explore efficient and affordable methods to deliver the data of these official productions to the international collaborators in the experiments.


Portd’Informació


6. Deliver to the groups a local software development environment (for simulation, reconstruction and analysis) appropriate for their experiment.

7. Facilitate the transfer of the experience acquired, fostering the possible re-use of this knowledge in other HEP projects or in other disciplines.


Portd’Informació

Científica Work Methodology

• Major agreements by groups– Commit to give services to each other– Allow jobs from any experiment to run at institute’s facilities (with some

safeguards)

• Opened the possibility to weave deliverables together into a “Spanish Grid Economy” linked to the global LCG Grid

• Forced to use build-to-cost, as cost estimates are difficult with fast-moving technologies and demand seems to far exceed reasonable funding expectations in 2003-2005

• Allow a reasonable amount of personnel “scaling” in order to study farm and data management automation to minimize personnel needs in the production system


Portd’Informació

Científica Risk Analysis

• Major agreements by groups (R1: Management support)– Commit to give services to each other– Allow jobs from any experiment to run at institute’s facilities (with some

safeguards)

• Opened the possibility to weave deliverables together into a “Spanish Grid Economy” linked to the global LCG Grid (R2: Will Grid techniques deliver. Alternatives?)

• Forced to use build-to-cost, as cost estimates are difficult with fast-moving technologies and demand seems to far exceed reasonable funding expectations in 2003-2005

• (R3: Fulfilling official MC. R4: Production scale-up)• Allow a reasonable amount of personnel “scaling” in order to study farm

and data management automation to minimize personnel needs in the production system(R5: “Ratchet” effect, local vs national personnel policy)


Portd’Informació

Científica LCG-ES Budget summary

Notes:

1. Spanish contribution to LCG-CERN (300 K€) handled administratively through IFAE (Additional 100 K€ for LCG-CERN Phase 1 to be funded in 2005)

2. Project budget is 50% human resources. If you include human resources funded by institutes and universities, it is approximately 2/3 people. Consistent with Gartner Group????


Portd’Informació

Científica LCG-ES Personnel details (estimates)

Personnel per Work Package

Personnel per Group and Funding Source


Portd’Informació

Científica

IFIC

UBIFAE

UAMCIEMAT

USCIFCA

Deliverables to fulfill the objectives

EAD = Analysis Farm

EDS = SW Dev Platform

RSG = SW repository

GSW = SW Gridification

MCF = MC Fabric

GVM = Virtual MC farm

ETD = Data Transform

PIC = Gridified Data Store

SEG = Security Architect

CTS = Tech MC Support

CDC = Data Chal. Coord.

EAD, MCF, GSW, CDC

EAD, GSW, CDC

EAD, MCF, EDS, RSG, SEG, CTS

EAD, GVM

EAD, ETD, PIC, EDS, GSW

EAD,EDS, CTS

EAD, MCF, CTS, CDC

• Stay away from glitz.

• Concentrate on deployment, MC & analysis

• Use local Univ. for TT to other disciplines

• 600 KCHF materials contribution to LCG-CERN


Portd’Informació

CientíficaUse Grid technology to generate a

common infrastructure

UIJDL

Logging &Logging &BookBook--keepingkeeping

Job SubmitEvent

ResourceResourceBrokerBroker

Output Sandbox

Input Sandbox

Job SubmissionJob SubmissionServiceService

StorageStorageElementElement

ComputeComputeElementElement

Brokerinfo

Output Sandbox

Input Sandbox

Information Information ServiceService

Job Status

ReplicaReplicaCatalogueCatalogue

Job Status


Portd’Informació

CientíficaFunctionality developed by EDG, Globus, etc.

brought to high performance level by LCG + Data Challenges

http://ccwp7.in2p3.fr/mapcenter/

EU DataGrid Monitor


Portd’Informació

Científica

grid for a physicsstudy group

grid for a regional group

les.ro

bertso

n@ce

rn.ch

Tier2

Lab a

Uni a

Lab c

Uni n

Lab m

Lab b

Uni bUni y

Uni x

Tier3physics

department

α

β

γDesktop

Germany

Tier 1

USAUK

France

Italy

……….

CERN Tier 1

……….

“The LHC Computing Centre”

The LHC Computing Model:1999-2002 and still evolving

CERN Tier 0


Portd’Informació

Científica The “Tier-1 Dilemma”

Spanish “Grid Economy” Zone

Global LHC Community

Spain:

~7% of CERN

~3% of LHC

Classic T1:

Serve 10%of a

global experiment


Portd’Informació

Científica LCG Requirements Ctte (SC2) revisited Tier 1/2/3 issue. New document focuses more on services than on capacity

Spanish “Grid Economy” Zone

Global LHC Community

PIC

Prototype a “Grid Services Exchange” focused on LHC data

PIC = Port d’Informació Científica

• “Legacy-free” center for data-intensive scientific computing

• Exploit commodity hardware and automation to lower costs

• Store and offer produced datasets to global LCG securely

• Cache datasets from global LCG for secure use by ES physicists


Portd’Informació

Científica Objectives of the PIC

• Coordination centre for data-intensive scientific computing for users in Spain and open to other users in the EU and other nations.

• Coordination centre for LHC computing of Spanish HEP, maintaining contacts particularly with CERN.

• Reference centre in data-intensive scientific computation techniques, including transfer of these techniques to scientific disciplines which may benefit from them.

• Establishment of collaborations with other institutes involved in scientific computation, in particular supercomputing and massively parallel computing.

• Development of techniques and general services in the context of the future Global Information Grid.

• Development of a Centre of Excellence to enable participation in European projects for the development of a future International Grid Infrastructure.


Portd’Informació

CientíficaPIC Base Funding Plan

(Note: CIEMAT agreement not signed yet)

Port d'Informació Científica (PIC)

0

0.2

0.4

0.6

0.8

1

1.2

2002 2003 2004 2005 2006 2007

(Meu

ros)

Year

Bas

e Fu

ndin

g Pl

an CIEMAT-ICIEMAT-PGENCAT-IGENCAT-PUAB-IUAB-M&O


Portd’Informació

Científica PIC infrastructure at UAB

• UAB is turning “Edifici D” into a shared infrastructure to house computationally oriented scientific research activities. The largest “user” of the building will be the PIC.

• PIC will have ~150 m2 office space there (mostly “open-plant” type, we already have 90 m2). Some space reserved for visitors.

• The PIC tape robots and data servers are housed in a partition of the computer room of Edifici D.– 150 m2 of high quality machine room with false floor– 200 KVA electricity with battery backup capilarized to

about 70 16-Amp circuits each with smoke detector– 300 KW of triply redundant air conditioning– 500 KVA diesel generator (Rolls-Royce!!)

• The Catalan and Spanish A&R Network are being boosted to the Gbpsrange. GÈANT is already multi-Gbps.


Portd’Informació

CientíficaA few pictures of where we are with

PIC

Overall view of machine room

Space for:

2 STK cylindrical robots

35 19” racks

Future:

Double tape space by using additional room


Portd’Informació

CientíficaA few pictures of where we are with

PIC

Latest Generation L5500 StoragetekTape Robot (6000 slots, >1.4 PBytescapacity)

2 Latest Generation 9940B drives (200 GB cartridge, 30 MB/sec)


Portd’Informació

Científica

General middlewareGrid Infrastructure

Field-specific middleware

Communities of geographically

dispersed, collaborating

scientists from various fields

Distinct, Collaborating Network and Computing

Infrastructures

Field-specific middlewareField-specific middleware

One Grid Infrastructure makes many Grids possible


Portd’Informació

Científica

General middleware

Grid Infrastructure

Grid-enabled Computing

and Storage Resources

Network Infrastructure

Grids and Grid Infrastructure: Provider-centric view

Multiple Scientific Communities

collaborating in subsets


Portd’Informació

Científica Questions of Terminology

Enhanced Science possible by continuous collaboration of mobile scientists supported by digital resources“e-Science”

The ‘net that we know and which will grow influenced by and influencing Grid Infrastructure and Grids

“Research NetworkInfrastructure

and/or Internet”

The set of persistent Services and Processes which permit construction and reconfiguration of Grids

“GridInfrastructure”

The set of digital resources that a Community is using at any given time to support their e-activity“a Grid”

The set of all instances of “a Grid”“Grids”


Portd’Informació

Científica

R&

D o

n G

RID

sR

&D

on

GR

IDs

GRIDs useGÉANT infrastructure

GÉANT profits from technological innovation

GRIDs empowered GÉANT

International dimension

GÉANT networkGÉANT network

Grid Infrastructure Prototypes

Grid Infrastructure Prototypes

Grids in support of various subjectsGrids in support of various subjects

An illustration: GÉANT and GRIDs (as seen by EU IST programme)

Adapted by permission from A. Karlson, EU DG-INFSO

Enhanced Science Enhanced Science or eor e--ScienceScience


Portd’Informació

CientíficaLead-times and momentum in R+D

Internet – Web – Web Services – Grid

DNS, Akamai, SSL, Yahoo, co-location, Web Hosting, etc.


Portd’Informació

Científica Evolution of data-processing environments

Mainframes

Operating systems

Mini-computers, Relational Databases

Microprocessors, PC, Mac

Fiber optics, Ethernet, TCP/IP, Internet

Digital ID infrastructure, Grid Infrastructures

Ubiquitous wireless self-configuring devices

Numerical methods

Time-sharing computing

Process Automation

Personal Productivity

The Network Society

Virtual Communities sharing Digitized Data

Ambient Computing

1 10 100 1000 10000 100000 1000000 10000000

1950

1960

1970

1980

1990

2000

2010

2020D

ecad

e

Complexity and Functionality

ProgrammingComputers


Portd’Informació

Científica Evolution of data-processing environments

Mainframes

Operating systems

Mini-computers, Relational Databases

Microprocessors, PC, Mac

Fiber optics, Ethernet, TCP/IP, Internet

Digital ID infrastructure, Grid Infrastructures

Ubiquitous wireless self-configuring devices

Numerical methods

Time-sharing computing

Process Automation

Personal Productivity

The Network Society

Virtual Communities sharing Digitized Data

Ambient Computing

1 10 100 1000 10000 100000 1000000 10000000

1950

1960

1970

1980

1990

2000

2010

2020D

ecad

e

Complexity and Functionality

ProgrammingComputers

We are here

WWW invented hereInternet “born” here

First e-mail ever sent


Portd’Informació

Científica

IFIC

UBIFAE

UAMCIEMAT

USCIFCA

Deliverables to fulfill the objectives

EAD = Analysis Farm

EDS = SW Dev Platform

RSG = SW repository

GSW = SW Gridification

MCF = MC Fabric

GVM = Virtual MC farm

ETD = Data Transform

PIC = Gridified Data Store

SEG = Security Architect

CTS = Tech MC Support

CDC = Data Chal. Coord.

EAD, MCF, GSW, CDC

EAD, GSW, CDC

EAD, MCF, EDS, RSG, SEG, CTS

EAD, GVM

EAD, ETD, PIC, EDS, GSW

EAD,EDS, CTS

EAD, MCF, CTS, CDC

• Stay away from glitz.

• Concentrate on deployment, MC & analysis

• Use local Univ. for TT to other disciplines

• 600 KCHF materials contribution to LCG-CERN

Port d’Informació Científica The LHC Computing Grid Project in … · 2010. 6. 16. ·...

Documents

Transcript of Port d’Informació Científica The LHC Computing Grid Project in … · 2010. 6. 16. ·...