Fabrizio GagliardiEU DataGrid Project LeaderEGEE Project Coordinatorf.gagliardi@cern.ch

EGEE is proposed as a project funded by the European Union under contract IST-2003-508833

From Grid Prototypes to Grid Infrastructures

NORDUnet 2003 - Reykjavík, Iceland – 24/27 August 2003

NORDUnet 2003 - Reykjavík, Iceland – 24/27 August 2003- 2

Introduction

• International computing networks are major enablers of new computing models

• GRID computing is one of them, probably the most promising

• Effective implementation of a truly distributed computing model across different non uniform computing administrative domains

• Many prototype projects based on RN Geant and similar international initiatives

NORDUnet 2003 - Reykjavík, Iceland – 24/27 August 2003- 3

Through collaborative Grid projects, there is thepotential for a truely global scientific applications grid

Some DataGrid Projects

NORDUnet 2003 - Reykjavík, Iceland – 24/27 August 2003- 4

• Connections of the different nodes of the EDG testbed are made possible by the EU-funded GEANT project

connecting more than 30 countries across Europespeeds of up to 10 Gbit/shigh data throughput Quality of Service

• EDG and GEANT: the first major production quality tests of the network

EDG & GEANT

speedreliabilitymonitoring capabilities

NORDUnet 2003 - Reykjavík, Iceland – 24/27 August 2003- 5

• Genomic Exploration• Earth Observation• High Energy Physics

Applications

more and more scientists begin to use the Grid computing model and existing Grid testbeds, relying on Grid technology to solve huge data challenges

NORDUnet 2003 - Reykjavík, Iceland – 24/27 August 2003- 6

LHC – Resources Requirements

CMSATLAS

LHCb

Storage – Raw recording rate 0.1 – 1 GByte/sec

Accumulating data at 5-8 PetaBytes/year

10 PetaBytes of disk

Processing – 100,000 of today’s fastest PCs

NORDUnet 2003 - Reykjavík, Iceland – 24/27 August 2003- 7

DataGrid prototypes: DataGrid (I)

• 9.8 M Euros EU funding over 3 years (twice as much from partners)

• 90% for middleware and applications (High Energy Physics, Earth Observation, Genomic Exploration)

• Total of 21 partners, over 150 programmers from research and academic institutes as well as industrial companies

• Three year phased developments & demos (2001-2003)

• Several improved versions of middleware software (final release end 2003)

• Several components of software integrated in the large Particle Physics Production LHC Computing Project (LCG)

• Software used by partner projects: DataTAG, CROSSGRID, GRACE

NORDUnet 2003 - Reykjavík, Iceland – 24/27 August 2003- 8

DataGrid prototypes : DataGrid (II)

• DataGrid testbed: more than 1000 CPUs at more than 15 sites (up to 40)

• Connections made possible by the EU-funded GEANT project • connecting more than 30 countries across Europe

• speeds of up to 10 Gbit/s

• high data throughput

• quality of Service

Site Country CPUs Storage

CC-IN2P3* FR 620 192 GB

CERN* CH 138 1321 GB

CNAF* IT 48 1300 GB

Ecole Poly. FR 6 220 GB

Imperial Coll. UK 92 450 GB

Liverpool UK 2 10 GB

Manchester UK 9 15 GB

NIKHEF* NL 142 433 GB

Oxford UK 1 30 GB

Padova IT 11 666 GB

RAL* UK 6 332 GB

SARA NL 0 10000+ GB

TOTAL 5 1075 14969 GB

*also Dev. TB; +200 TB including tape

NORDUnet 2003 - Reykjavík, Iceland – 24/27 August 2003- 9

The next step: go production

• Similarly to research networks few years ago, after many prototype projects we need to go production

• Major issues: Security, (im)maturity of M/Ware toolkits, difficult user interface for average use, cost and complexity of operations, etc…

NORDUnet 2003 - Reykjavík, Iceland – 24/27 August 2003- 10

EGEE vision:Enabling Grids for E-science in Europe

• Goal•Create a wide European Grid production quality infrastructure on top of present and future EU RN infrastructure

• Build on•EU and EU member states major investments in Grid Technology

•International connections (US and AP)•Several pioneering prototype results •Larg Grid development team (>60 people in EDG)

•Requires major EU funding effort

• Approach•Leverage current and planned national and regional Grid programmes

•Work closely with relevant industrial Grid developers, NRENs and US-AP projects

 

EGEE

Applications

Geant network

NORDUnet 2003 - Reykjavík, Iceland – 24/27 August 2003- 11

Why EGEE? Impact on Society

• Access to a production quality grid will change the way science and business is done in Europe

An international network of scientist will be able to model a new flood of the Danube in real time, using meteorological and geological data from several centers across Europe

A team of engineering students will be able to run the latest 3D rendering programs from their laptops using the Grid

A geneticist at a conference, inspired by a talk she hears, will be able to launch a complex biomolecular simulation from her mobile phone

NORDUnet 2003 - Reykjavík, Iceland – 24/27 August 2003- 12

Why EGEE? Political context

• Current Grid R&D projects run out within few months

• The EGEE partners have already made major progress in aligning national and regional Grid R&D efforts, in preparation for EGEE

• EGEE will preserve the current strong momentum of the European Grid community, and the enthusiasm of the hundreds of young European researchers already involved in EU Grid projects (>150 in EDG only)

NORDUnet 2003 - Reykjavík, Iceland – 24/27 August 2003- 13

Why EGEE? Historical analogy

• Prior to the EU Geant programme, there was in Europe a multitude of exploratory projects in networking technology. Geant was truly production oriented, and brought European telecom operators actively into the picture

• In a similar way, EGEE can ensure preservation of current investments in European Grid R&D, extending the present infrastructure and focussing all activities towards establishing a production quality Grid

NORDUnet 2003 - Reykjavík, Iceland – 24/27 August 2003- 14

EGEE Partner Federations

• Integrate regional Grid efforts

• Represent leading grid activities in Europe

NORDUnet 2003 - Reykjavík, Iceland – 24/27 August 2003- 15

EGEE Activity Areas

• Services• Deliver “production level” grid services (manageable, robust, resilient to failure)• Ensure security and scalability

• Middleware• Professional Grid middleware re-engineering activity in support of the production

services• Networking

• Proactively market Grid services to new research communities in academia and industry

• Provide necessary education

NORDUnet 2003 - Reykjavík, Iceland – 24/27 August 2003- 16

ResourceCenter

(Processors, disks)

Grid server Nodes

ResourceCenter

ResourceCenter

ResourceCenter

OperationsCenter

Regional SupportCenter

(Support for Applications

Local Resources)

Regional Support

Regional Support

Regional Support

EGEE Operations Structure

NORDUnet 2003 - Reykjavík, Iceland – 24/27 August 2003- 17

EGEE Service Activity

• Create, operate, support and manage a production quality infrastructure

• Structure: • EGEE Operations Management at CERN • EGEE Core Infrastructure Centres in the UK, France,

Italy and CERN (leveraging HEP LCG at the start), responsible for managing the overall Grid infrastructure

• Regional Operations Centres, responsible for coordinating regional resources, regional deployment and support of services in all other countries

• Offered services:• Middleware deployment and installation• Software and documentation repository• Grid monitoring and problem tracking• Bug reporting and knowledge database• VO services• Grid management services

NORDUnet 2003 - Reykjavík, Iceland – 24/27 August 2003- 18

EGEE Networking Activity

• Dissemination and outreach

• User training and induction

• Application identification and support

• Two pilot application centers (for high energy physics and biomedical grids)

• One more generic component dealing with longer term recruitment and support of other communities

• Policy and International cooperation

rely on a supporting network in the partner regions

NORDUnet 2003 - Reykjavík, Iceland – 24/27 August 2003- 19

EGEE Middleware Activity

• Hardening and re-engineering of existing middleware functionality, leveraging the experience of partners

• Activity concentrated in few major centers

• Key services: Resource Access• Data Management• Information Collection and Accounting• Resource Brokering (Italy)• Quality Assurance• Grid Security• Middleware Integration• Middleware Testing

NORDUnet 2003 - Reykjavík, Iceland – 24/27 August 2003- 20

EGEE & Industry

• Industrial participation encouraged both as potential end-users and IT technology and service suppliers

• Normally through national and regional Grid EGEE federations

• EGEE will maintain an Industry Forum to keep selected Industrial and Commercial interested parties in close contact

• Services developed in first EGEE 2 years phase (2004-5) might be tendered to Industry in second phase (2006-7)

NORDUnet 2003 - Reykjavík, Iceland – 24/27 August 2003- 21

EGEE Timeline

• May 2003: proposal submitted

• July 2003: proposal accepted

• September 2003: start negotiation

• April 2004: start project

NORDUnet 2003 - Reykjavík, Iceland – 24/27 August 2003- 22

Conclusions

• The EU DataGrid project has successfully fulfilled its role of EU Grid flagship project in collaboration with several other EU and international projects

• Essential to keep the momentum and the current lead in production Grids in Europe

• Important to build an international cooperation between European and US/AP Grid infrastructure projects

• The scientific user communities are already international (HEP is an excellent example) and so the computing resources and most of the experimental instruments

• EGEE proposes the right framework and plans to accomplish the above objectives leveraging present and future international research networks

NORDUnet 2003 - Reykjavík, Iceland – 24/27 August 2003- 23

More information

• More information on the EU DataGrid project on:www.edg.org

• More information on EGEE on:www.cern.ch/egee

Or mail me at: fabrizio.gagliardi@cern.ch