What Part of NO, NO, No, does the parasitic bureaucracy CLRC not Understand?
PS INFN GRID · Taiwan – Academia SInica (Taipei) UK – CLRC (Oxford) US – FermiLab (Illinois)...
Transcript of PS INFN GRID · Taiwan – Academia SInica (Taipei) UK – CLRC (Oxford) US – FermiLab (Illinois)...
PS INFN GRID
Obiettivi ScientificiStato del progettoStato del progetto
Connessione con attività ATLASPersone e risorsePersone e risorse
5-luglio-2007 Milano-LASA L.Perini 1
Obiettivi PS INFN GRID• Coordinare la partecipazione INFN ai progetti Computing
Grid– Ottenere così il finanziamento delle persone che sviluppano eOttenere così il finanziamento delle persone che sviluppano e
mantengono mw e infrastruttura Grid con fondi extra INFN• EU ma anche italiani extra INFN
• Per INFN interesse specifico di GRID èPer INFN interesse specifico di GRID è– Il suo utilizzo per il calcolo esperimenti LHC– Assunzione di leadership in campo tecnologico promettente
• La principale fonte di fondi è oggi EGEE II• La principale fonte di fondi è oggi EGEE-II• L’utilizzazione di Grid per LHC è coordinata nel progetto
LHC Computing Grid (LCG)
5-luglio-2007 Milano-LASA L.Perini 2
INFNINFN GRIDGRIDINFNINFN--GRIDGRID
INFN-GRID è il Progetto Speciale INFN “contenitore” di tutte le attività GRID comprese quelle degli esperimenti: i ricercatori-tecnologi afferenti agli esperimenti che si impegnano in GRID lo fanno per implementare il calcolodel loro esperimento, quindi la loro % INFN GRID è inclusa in quella dell’esperimentoData di inizio: giugno 2000http://grid.infn.ithttp://grid.infn.it
LCGLCGLHC Computing Grid H L’ bi tti di li i f t tt (h/ / ) il l l di t ib itHa L’obiettivo di realizzare una infrastruttura (h/w e s/w) comune per il calcolo distribuito degli esperimenti LHC ( 4 aree: Grid Tecnology, Grid Deployment, Fabric, Applications)Fase 1 (2002-2005): realizzazione prototipoFase2 (2006-2008): realizzazione e gestione del “servizio di calcolo”( ) g
per LHC http://www.cern.ch/LCG
5-luglio-2007 3
LCG WLCG Collaboration
The Collaboration~100 computing centres12 large centres
(Tier-0, Tier-1)38 federations of smaller
“Tier-2” centres20 countries
M d f U d t diMemorandum of UnderstandingAgreed in October 2005
ResourcesC i d h O b f h i Commitment made each October for the coming year5-year forward look
HEPiX Rome 05apr06 [email protected]
LCG LCG Service HierarchyTier-0 – the accelerator centre
Data acquisition & initial processingLong-term data curationDistribution of data Tier-1 centres
Tier-1 – “online” to the data acquisition process high availabilityManaged Mass Storage –
Canada – Triumf (Vancouver)France – IN2P3 (Lyon)Germany – Forschunszentrum KarlsruheItaly – CNAF (Bologna)N th l d Ti 1 (A t d )
Spain – PIC (Barcelona)Taiwan – Academia SInica (Taipei)UK – CLRC (Oxford)US – FermiLab (Illinois)
Managed Mass Storage grid-enabled data
serviceData-heavy analysisN ti n l i n l s pp tNetherlands Tier-1 (Amsterdam)
Nordic countries – distributed Tier-1 US – FermiLab (Illinois)
– Brookhaven (NY) National, regional support
Tier-2 – ~100 centres in ~40 countriesSi l ti
HEPiX Rome 05apr06 [email protected]
SimulationEnd-user analysis – batch and interactive
LCG
Summary of Computing Resource RequirementsAll experiments - 2008From LCG TDR - June 2005
CERN All Tier-1s All Tier-2s TotalCPU (MSPECint2000s) 25 56 61 142Disk (PetaBytes) 7 31 19 57
CPU Disk Tape
Disk (PetaBytes) 7 31 19 57Tape (PetaBytes) 18 35 53
CERN18%
All Tier-2s43%
CERN12%
All Tier-2s33%
CERN34%
p
All Tier-1s39%
All Tier-1s55%
All Tier-1s66%
HEPiX Rome 05apr06 [email protected]
LCG LCG depends on two major sciencegrid infrastructures grid infrastructures ….
EGEE - Enabling Grids for E-SciencegOSG - US Open Science Grid
HEPiX Rome 05apr06 [email protected]
LCGProblem Response Time and Availability targets
Tier-1 CentresLCG
Service
Maximum delay in responding to operational problems (hours)
AvailabilityDegradation of theService Availability Service
interruption
gservice
> 50% > 20%
Acceptance of datafrom the Tier-0 Centreduring accelerator operation
12 12 24 99%operation
Other essential services– prime service hours 2 2 4 98%p
Other essential services– outside prime
service hours24 48 48 97%
HEPiX Rome 05apr06 [email protected]
LCGLCG
Problem Response Time and Availability targetsTier-2 Centres
Maximum delay in responding
Service
y p gto operational problems
availabilityPrime time Other periods
E d l i 2 h 72 h 95%End-user analysis facility
2 hours 72 hours 95%
Other services 12 hours 72 hours 95%
HEPiX Rome 05apr06 [email protected]
Enabling Grids for E-sciencE
Project GoalsFrom EGEE Final EU Review May 2006
Within a four year programme:
• Build, deploy and operate a consistent, robust and secure grid that attracts new computing resourcessecure grid that attracts new computing resources
• Improve and maintain the middleware in order to pdeliver a reliable service to users
• Attract new users from science and industry and ensure training and support for them
INFSO-RI-508833 10
Enabling Grids for E-sciencE
Activities Definition
• Network Activities– NA1: Project Management– NA2: Dissemination and Outreach– NA3: User Training and Induction– NA4: Application Identification and Support– NA5: Policy and International Cooperation
• Service Activities– SA1: Grid Support, Operation and Management– SA2: Network Resource Provision– SA2: Network Resource Provision
• Joint Research Activities– JRA1: Middleware Reengineering + Integrationg g g– JRA2: Quality Assurance– JRA3: Security– JRA4: Network Services Development
Emphasis in EGEE is on operating a production
id d ti th
INFSO-RI-508833 Milano-LASA L.Perini 11
grid and supporting the end-users
Enabling Grids for E-sciencE
What has EGEE delivered?I f t t ti• Infrastructure operation– Currently includes >180 sites across 40 countries build
on GEANT network– Continuous monitoring of grid services & automated site
fi ti / tconfiguration/management
• Middleware– Production quality middleware distributed under q y
business friendly open source licence
• User Support - Managed process from first contact through to production usagethrough to production usage– Training– Documentation – Expertise in grid-enabling applications
Online helpdesk– Online helpdesk– Networking events (User Forum, Conferences etc.)
• Interoperability
INFSO-RI-508833 Milano-LASA L.Perini 12
– Expanding interoperability with related infrastructures– Major contributor to grid standardisation
Enabling Grids for E-sciencE
Related Projects
EGEE as incubator & supporter of related projects
INFSO-RI-508833 Milano-LASA L.Perini 13
DiligentA DIgital Library Infrastructureon Grid ENabled Technology
Etc.
Enabling Grids for E-sciencE
Manpower levels
• Further refined PPT tool to monitor project effort consumption (timesheets)– Over 1000 people registered across Europe and Russiap p g p– Total equivalent of ~ 460 FTEs
Legend:
Total (F+UF) Effort in PM
12000
PM
Legend:• F: Funded• U: Unfunded• PM: Person Months8000
10000
• PM: Person Months• FTE: Full time equiv.
6000
8000
Consumed F+UF EffortExpected F+UF Effort
Consumed Effort
Funded Unfunded Total
112% 122% 114%2000
4000
INFSO-RI-508833 Milano-LASA L.Perini 14
0Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8
Enabling Grids for E-sciencE
The EGEE-II projectEGEE• EGEE– 1 April 2004 – 31 March 2006– 71 partners in 27 countries, federated in regional Grids
• EGEE-II– 1 April 2006 – 31 March 2008– 91 partners in 32 countries91 partners in 32 countries – 13 Federations
• ObjectivesL l d ti lit– Large-scale, production-quality infrastructure for e-Science
– Attracting new resources and users from industry as well asusers from industry as well asscience
– Maintain and further improve“gLite” Grid middleware
INFSO-RI-508833 Milano-LASA L.Perini 15
gLite Grid middleware
Enabling Grids for E-sciencE
Sustainability: Beyond EGEE-II
• Need to prepare for permanent Grid infrastructure– Maintain leading position in global science Grids
Ensure a reliable and adaptive support for all sciences– Ensure a reliable and adaptive support for all sciences– Independent of short project funding cycles– Modelled on success of GÉANT
Infrastructure managed in collaboration with national grid initiatives
INFSO-RI-508833 Milano-LASA L.Perini 16
Enabling Grids for E-sciencE
• PoW of technical activities is coordinated by the Technical Coordination Group (TCG)
• Makes decision process clearer and ensures application driven progress
• Started in November 2005
• Main accomplishments– Definition of gLite 3.0
Clarification and prioritization of main– Clarification and prioritization of main application requirements
– Draft short term workplans for mw development and integration
INFSO-RI-508833 Milano-LASA L.Perini 17
p g
Enabling Grids for E-sciencE
Middleware structure• Applications have access
both to Higher-level Grid Services and to Foundation Grid MiddlewareGrid Middleware
• Higher-Level Grid Services are supposed to help the users building theirusers building their computing infrastructure but should not be mandatory
• Foundation Grid Middleware will be deployed on the EGEE infrastructure– Must be complete and robust
Should allow interoperation– Should allow interoperation with other major grid infrastructures
– Should not assume the use of
INFSO-RI-508833 Milano-LASA L.Perini 18
Higher-Level Grid Services
Enabling Grids for E-sciencE
Highlights of EGEE-II
EGEE Production Grid InfrastructureSteady growth over the lifetime of the projectImproved reliability
Data TransferM
B/s
M
04/2006 08/2006
EGEE-II INFSO-RI-031688 Milano-LASA L.Perini 19
Enabling Grids for E-sciencE
CPU, countries, sites
CERN; 5943 SWE; 1593
Russia; 583 A‐P; 1632
CPU / ROC
CERN; 4France; 1
De/CH; 2
Italy; 1
A‐P; 8
Countries / ROC
France; 2700
SEE; 2568
Italy; 1
UK/I; 2
CE; 7
SWE; 2
Russia; 2
De/CH; 3364CE; 1875
NE; 3031
NE; 8
SEE; 8
Sites / ROC
Italy; 3628
UK/I; 7720
CERN; 12France; 10
De/CH; 14
SWE; 15
Russia; 15
A‐P; 20
Sites / ROC
UK/I; 7720
Italy; 37
SEE; 3835000 CPU45 countries (31 partner countries)
EGEE-II INFSO-RI-031688 Milano-LASA L.Perini 20SA1 - Ian Bird - EGEE-II 1st EU Review - 15-16 May 2007 20
UK/I; 25
CE; 24NE; 27
( p )237 sites (131 partner sites)
Enabling Grids for E-sciencE
Workload
2000000
2500000
3000000No. jobs / month ‐ all 98000 jobs/day
500000
1000000
1500000 OPS
Non‐LHC
LHC
0
250000
300000
350000
400000
450000No. jobs / month ‐ exc. LHC + Ops Other VOs
planck
magic
4
13000 jobs/day
0
50000
100000
150000
200000geant4
fusion
esr
egrid
egeode
EGEE-II INFSO-RI-031688 Milano-LASA L.Perini 21SA1 - Ian Bird - EGEE-II 1st EU Review - 15-16 May 2007 21
egeode
compchem
Enabling Grids for E-sciencE
CPU time delivered
6000000
7000000
8000000
9000000
10000000Normalised CPU hours ‐ all
14000 CPU-month/month
2000000
3000000
4000000
5000000
6000000OPS
Non‐LHC
LHC
0
1000000
apr‐06 mag‐06 giu‐06 lug‐06 ago‐06 set‐06 ott‐06 nov‐06 dic‐06 gen‐07 feb‐07 mar‐07
3000000
2000000
2500000
3000000Normalized CPU hours ‐ exc. LHC + Ops OPS
Other VOs
planck
magic
3600 CPU-month~ 1/3 of total
500000
1000000
1500000 geant4
fusion
esr
egrid
d
EGEE-II INFSO-RI-031688 Milano-LASA L.Perini 22SA1 - Ian Bird - EGEE-II 1st EU Review - 15-16 May 2007 22
0
apr‐06 mag‐06 giu‐06 lug‐06 ago‐06 set‐06 ott‐06 nov‐06 dic‐06 gen‐07 feb‐07 mar‐07
egeode
compchem
Enabling Grids for E-sciencE
Regional distributionRegional share of jobs
AsiaPacific, 3.5% CentralEurope, 6.5%
CERN; 7,0%UKI; 22,2%
Regional share of jobs
France; 12,6%SouthWest 6.4%
Asia Pacific, 2.8% CentralEurope; 4,9%
Regional share of CPU deliveredGermany/Switzerland, 8.6%
Italy; 15,8%NorthernEurope;
6,7%
Russia; 3,9%
South East, 6.9%
CERN; 6,9%
France; 11,2%
UKI; 30,3%
GermanySwitzerland; 7,8%
SouthWesternEurope; 7 9%
EGEE-II INFSO-RI-031688 Milano-LASA L.Perini 23SA1 - Ian Bird - EGEE-II 1st EU Review - 15-16 May 2007 23
Italy; 11,1%
NorthernEurope; 5,4%Russia; 2,1%
SouthEasternEurope; 9,7%
ope; 7,9%
Ruolo e persone Milano GRID • Milano ha ruolo molto rilevante, essendo:
– Tier2 – sj ( dettagli in presentazione ATLAS-GRID)• Interfaccia locale GRID Luca Vaccarossa, finanziato 100% da SA1 EGEE
– Sede di uno dei cluster di sviluppo del m/w in EGEE• F.Prelz (50%) (membro del advisory board internazionale) e personale al
100% finanziato da EGEE: art.23 (M. Mezzadri, D.Rebatto) ( )– Sede “experimental services”
• WMS gLite 3.1 è stato portato a production level in collaborazione da ATLAS e SA3 EGEE ( E. Molinari art.23) e i servers WMS sono ora 2 al CNAF e 1 a Milano
• A Milano è presente anche OMII (Open Mw Initiative) – Progetto Eu collegato ad EGEE per costruire e mantenere un set di mw e
servizi a disposizione di tutte le comunità Grid. G. Fiorentino art.23 . In i l BLAH i i d l i i CE ilparticolare per BLAH componente critica del servizio CE, sviluppata a
Milano in EGEE.• Persone degli esperimenti ( 0% formalmente)
5-luglio-2007 Milano-LASA L.Perini 24
Da EGEE2 a EGEE3• EGEE2 ha appena superato con la valutazione migliore fra
quelle possibili la review dopo il suo primo anno– Ormai la grid è l’infrastruttura per il calcolo distribuito di LHC eOrmai la grid è l infrastruttura per il calcolo distribuito di LHC e
non solo…• EGEE3 e’ in fase di negoziazione, è certo che sarà
finanziato con livello simile a quello di EGEE2 circa 35 Mfinanziato con livello simile a quello di EGEE2, circa 35 M Euro– Potrebbero essere però 30 o 38 e fa una bella differenza
• Altri progetti GRID in cui INFN (Milano) è:• Altri progetti GRID in cui INFN (Milano) è:– EuChina, EuMed, EuIndia, EELA, LIBI, GRIDCC, CoreGrid,
ETICS, OMII, EGI/IGI
5-luglio-2007 Milano-LASA L.Perini 25
Persone di Milano e sinergie conPersone di Milano e sinergie con ATLAS-Grid
• Funded da progetti Grid EU– S. Resconi NA4-ATLAS – D. Rebatto JRA1 EGEE2 (blah) ATLAS-Lexor (TF)– L. Vaccarossa SA1 EGEE2 ( Tier2, Atlas-TF)– E. Molinari SA3 EGEE2 (ATLAS-TF)– M. Mezzadri JRA1 EGEE2 (blah e info.system)
I l 50% l l• Internal agreement 50% serv.calcolo– G. Fiorentino OMII (blah)
• Unfunded 50% F P l JRA1 INFN– 50% F. Prelz JRA1-INFN
– 50% L. Perini TCG (COLG INFN GRID) in uscita…– Probabile che ci venga richiesto in EGEE3 aumento unfunded
(tipicamente da serv. Calcolo)
5-luglio-2007 Milano-LASA L.Perini 26
(tipicamente da serv. Calcolo)
Necessità PS GRID 2007 e oltre• Per i fondi si tratta principalmente di missioni e consumo, riporto
richieste 2008 ancora tentative..– MI 17 kEuro
• Inclusi 3 kEuro Perini (deputy responsabile nazionale)• Inclusi 3 kEuro Perini (deputy responsabile nazionale)– ME 43 kEuro
• Inclusi 5 k Euro per L.Perini EGI/IGI– Consumo 5 kEuro– Inv. 2.5 k Euro
• EGEE2 termina a fine marzo 2008. – Negoziazione avanzata e positiva per EGEE3 e OMII2, contiamo di potere
avere 6 persone funded a Milano anche per i 2 anni successivi• Non ci sarà EGEE4, si stabilirà un consorzio fra Grid nazionali su
modello GARR per retiEGI/IGI è il progetto EU approvato per studiare come strutturare e far– EGI/IGI è il progetto EU approvato per studiare come strutturare e far partire in tempo questo consorzio
– INFN in EGI/IGI: M. Mazzucato (con supporto D.Cresti) e L.Perini
5-luglio-2007 Milano-LASA L.Perini 27