Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688EGEE and gLite are registered trademarks INFN...

62
Enabling Grids for E-sciencE www.eu-egee.org www.glite.org EGEE-II INFSO-RI- 031688 EGEE and gLite are registered trademarks INFN Grid and EU Mirco Mazzucato (INFN-Padova)
  • date post

    19-Dec-2015
  • Category

    Documents

  • view

    215
  • download

    0

Transcript of Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688EGEE and gLite are registered trademarks INFN...

Enabling Grids for E-sciencE

www.eu-egee.orgwww.glite.org

EGEE-II INFSO-RI-031688 EGEE and gLite are registered trademarks

INFN Grid and EU

Mirco Mazzucato(INFN-Padova)

2

Enabling Grids for E-sciencE

EGEE-II INFSO-RI-031688

Outline

• A little of history• The INFN GRID project• The EGEE project• The EGEE middleware • The INFN Production Grid: Grid.it• Long-term sustainability: EGI and IGI• Summary

3

Enabling Grids for E-sciencE

EGEE-II INFSO-RI-031688

The INFN GRID project

• The 1° National Project (Feb. 2000) aiming to develop the grid technology and the new e-infrastructure to solve LHC (and e-Science) computing requirements

• e-Infrastructure = Internet + new WEB and Grid Services on top of a physical layer composed by Network, Computing, Supercomputing and Storage Resources, made properly available in a shared fashion by the new Grid services

• Since then many Italian and EU projects made this a reality• Many scientific sectors in italy, EU and the entire World base now their

research activities on the Grid• INFN Grid continues to be the national container used by INFN to reach

its goals coordinating all the activities:– In the national, european and international Grid projects – In the standardization processes of the Open Grid Forum (OGF)– In the definition of EU policies in the ICT sector of Research

Infrastructures – Through its managerial structure: Executive Board, Technical Board…

4

Enabling Grids for E-sciencE

EGEE-II INFSO-RI-031688

The INFN GRID portal

http://grid.infn.it

5

Enabling Grids for E-sciencE

EGEE-II INFSO-RI-031688

The strategy

• Clear and stable objectives: development of the technology and of the infrastructure needed for the LHC computing but of general value

• Variable instruments: use of projects and external funds ( from EU, MIUR...) to reach the goal

• Coordination among all the projects (Executive Board) – Grid middleware & infrastructure Grid needed by INFN and LHC within

a number of core European and International projects, often coordinated by CERN

DataGrid, DataTAG, EGEE, EGEE II, WLCG– Often fostered by INFN itself

• International collaboration with US Globus and Condor for the middleware and Grid projects like Open Science Grid e Open Grid Forum in order to reach global interoperability among developed services and the adoption of international standards

• National pioneer developments of the MW and the national infrastructure in the areas not covered by EU projects via national projects like Grid.it , LIBI, EGG …

• Strong contribution to political committees: e-Infrastructure Reflection Group (eIRG ->ESFRI), EU Concertation meetings and with involved Units of Commission (F2 e F3) to establish activities programs (Calls)

6

Enabling Grids for E-sciencE

EGEE-II INFSO-RI-031688

Some history … LHC EGEE Grid

• 1999 – Monarc Project– Early discussions on how to organise distributed

computing for LHC• 2000 – growing interest in grid technology

– HEP community was the driver in launching the DataGrid project

• 2001-2004 - EU DataGrid project / EU DataTAG project– middleware & testbed for an operational grid

• 2002-2005 – LHC Computing Grid – LCG– deploying the results of DataGrid to provide aproduction facility for LHC experiments

• 2004-2006 – EU EGEE project phase 1– starts from the LCG grid– shared production infrastructure– expanding to other communities and sciences

• 2006-2008 – EU EGEE-II – Building on phase 1– Expanding applications and communities …

• … and in the future – Worldwide grid infrastructure??– Interoperating and co-operating infrastructures?

CERN

7

Enabling Grids for E-sciencE

EGEE-II INFSO-RI-031688

The EGEE project

• EGEE– 1 April 2004 – 31 March 2006– 71 partners in 27 countries, – federated in regional Grids

• EGEE-II– 1 April 2006 – 31 March 2008– 91 partners in 32 countries

• Objectives– Large-scale, production-quality

grid infrastructure for e-Science – Attracting new resources and

users from industry as well asscience

– Maintain and further improve“gLite” Grid middleware

8

Enabling Grids for E-sciencE

EGEE-II INFSO-RI-031688

EGEE activities

• Service Activities– SA1 – Grid Operations, Support

and Management (CERN)– SA2 – Networking Support (CNRS)– SA3 – Integration, Testing and

Certification (CERN)• Joint Research Activities

– JRA1 – Middleware Re-engineering (INFN)– JRA2 – Quality Assurance (CS-SI)

• Networking Activities– NA1 – Management (CERN)– NA2 – Dissemination, Outreach and

Communication (CERN)– NA3 – Training and Induction (UEdin)– NA4 – Application Identification and Support (CNRS)– NA5 – Policy and International Cooperation (GRNET)

9

Enabling Grids for E-sciencE

EGEE-II INFSO-RI-031688

EGEE Applications

• Multitude of applications from a growingnumber of domains– Astrophysics– Computational Chemistry– Earth Sciences– Financial Simulation– Fusion– Geophysics– High Energy Physics– Life Sciences– Multimedia– Material Sciences– …..

Book of abstracts: http://doc.cern.ch//archive/electronic/egee/tr/egee-tr-2006-005.pdf

App. Deployment Plan: https://edms.cern.ch/document/722131/2

10

Enabling Grids for E-sciencE

EGEE-II INFSO-RI-031688

High Energy Physics

Large Hadron Collider (LHC):• One of the most powerful instruments ever

built to investigate matter• 40 Million Particle collisions per second• 4 Experiments: ALICE, ATLAS, CMS, LHCb • ~15 PetaBytes/year from the 4 experiments• First beams in 2007

Mont Blanc(4810 m)

Downtown Geneva

11

Enabling Grids for E-sciencE

EGEE-II INFSO-RI-031688

In silico drug discovery

• Diseases such as HIV/AIDS, SRAS, Bird Flu etc. are a threat to public health due to world wide exchanges and circulation of persons

• Grids open new perspectives to in silico drug discovery– Reduced cost and adding an accelerating factor in the search for new drugs

•Avian influenza:

•bird casualties

International collaboration is required for: • Early detection

• Epidemiological watch

• Prevention

• Search for new drugs

• Search for vaccines

12

Enabling Grids for E-sciencE

EGEE-II INFSO-RI-031688

WISDOM

http://wisdom.healthgrid.org/

13

Enabling Grids for E-sciencE

EGEE-II INFSO-RI-031688

Medical image processing: analysing tumours

• Pharmacokinetics: contrast agent diffusion study– co-registration of a time series of volumetric medical images to analyse the

evolution of the diffusion of contrast agents

Sequential

If you have enough resources

20x12=240 computers, EGEE has >30,000

HPC

Grid

• Computational Costs– 20 Patients: 2623 hours (Co-registration + Parametric Image)

– Using a 20-processor Computing Farm: 146 hours

– Using the Grid: <20 hours

14

Enabling Grids for E-sciencE

EGEE-II INFSO-RI-031688

EGEE Operations Structure

• Operations Coord. Centre (OCC)

• Regional Operations Centres (ROC)

– Front-line support for user and operations issues

– Provide local knowledge and adaptations

– One in each region – many distributed

– Manage daily grid operations – oversight, troubleshooting

“Operator on Duty”– Run infrastructure

services

15

Enabling Grids for E-sciencE

EGEE-II INFSO-RI-031688

Grid monitoring tools

• Tools used by the Grid Operator on Duty team to detect problems

• Distributed responsibility• CIC portal

– single entry point– Integrated view of monitoring tools

• Site Functional Tests (SFT) -> Service Availability Monitoring (SAM)

• Grid Operations Centre Core Database (GOCDB)

• GIIS monitor (Gstat)• GOC certificate lifetime• GOC job monitor• Others

16

Enabling Grids for E-sciencE

EGEE-II INFSO-RI-031688

Site Functional Tests

• Site Functional Tests (SFT)

– Framework to test (sample) services at all sites

– Shows results matrix– Detailed test log available for

troubleshooting and debugging– History of individual tests is kept – Can include VO-specific tests

(e.g. sw environment)– Normally >80% of sites pass

SFTs

• Very important in stabilising sites:

• Apps use only good sites• Bad sites are automatically excluded• Sites work hard to fix problems

17

Enabling Grids for E-sciencE

EGEE-II INFSO-RI-031688

•RC A

•RC B

•RC C

•RC A

•RC B

•RC C

The EGEE support infrastructure

•ROC C•ROC B•ROC N

•RC A

•RC B

•RC C

TPM

VO TPM CVO TPM B

VO TPM A

GGUS

Central

System

Middleware

supportMiddleware

supportMiddleware

support

Deployment

supportMiddleware

supportDeployment

support

VO Support

CVO Support

BVO Support

A

Middleware

supportMiddleware

supportMiddleware

support

•ROC C•ROC B•ROC N

Network Support

Network Support Other GridsOther GridsOther Grids

Other GridsOther GridsOther Grids

CODCIC

Portal

18

Enabling Grids for E-sciencE

EGEE-II INFSO-RI-031688

0

5000

10000

15000

20000

25000

30000

35000

40000

Apr-0

4

Jun-

04

Aug-0

4

Oct-

04

Dec-0

4

Feb-0

5

Apr-0

5

Jun-

05

Aug-0

5

Oct-

05

Dec-0

5

Feb-0

6

Apr-0

6

Jun-

06

Aug-0

6

Oct-

06

Dec-0

6

No

. CP

U

0

50

100

150

200

250

Apr-0

4

Jun-

04

Aug-0

4

Oct-

04

Dec-0

4

Feb-0

5

Apr-0

5

Jun-

05

Aug-0

5

Oct-

05

Dec-0

5

Feb-0

6

Apr-0

6

Jun-

06

Aug-0

6

Oct-

06

Dec-0

6

No

. Sit

es

Status

~17.5 million jobs run (6450 cpu-years) in 2006;

Workloads of the “not HEP VOs” start to be significant – approaching 8-10K jobs per day; and 1000 cpu-months/month

• one year ago this was the overall scale of work for all VOs

~17.5 million jobs run (6450 cpu-years) in 2006;

Workloads of the “not HEP VOs” start to be significant – approaching 8-10K jobs per day; and 1000 cpu-months/month

• one year ago this was the overall scale of work for all VOs

19

Enabling Grids for E-sciencE

EGEE-II INFSO-RI-031688

EGEE Resources

Region #countries#sites #cpu

#cpu DoW

disk (TB)

CERN 0 1 7800 1800 1000*

UK/I 2 23 7309 2010 2410

Italy 1 27 3336 2280 1350

France 1 10 3715 1252 1300*

De/CH 2 13 3237 1852 280*

Northern Europe 6 16 2655 1860 200

SW Europe 2 13 1125 898 200*

SE Europe 8 26 1286 1189 33

Central Europe 7 21 1759 1163 90

Russia 1 15 515 445 52

Asia-Pacific 8 19 1239 751 120

North America 2 8 2973 - 300

Totals 40 192 36949 20265 7335

PPS 14 24 2517 - -

* Estimates taken from reporting as IS publishes total MSS space

20

Enabling Grids for E-sciencE

EGEE-II INFSO-RI-031688

Grid Virtual Organizations

• Routine and large-scale use of EGEE infrastructure.• Virtual Organizations:

–200+ visible on the grid–100+ registered with EGEE

http://www3.egee.cesga.es/gridsite/accounting/CESGA/tree_vo.php

21

Enabling Grids for E-sciencE

EGEE-II INFSO-RI-031688

Usage History

Virtual Organizations

Dec

. ’0

5N

ov.

’06

http://www3.egee.cesga.es/gridsite/accounting/CESGA/tree_vo.php

22

Enabling Grids for E-sciencE

EGEE-II INFSO-RI-031688

EGEE Related projects & other grids

Potential for linking ~80 countries

23

Enabling Grids for E-sciencE

EGEE-II INFSO-RI-031688

Other FP6 activities of INFN Grid in Europe/1

• To guarantee Open Source Grid Middleware evolutions towards international standards– OMII Europe

• …and its availability through an effective repository– ETICS

• To contribute to R&D informatics activities– Core Grid

• To Coordinate EGEE extension in the world– EUMedGrid– Eu-IndiaGrid – EUChinaGrid – EELA

24

Enabling Grids for E-sciencE

EGEE-II INFSO-RI-031688

Other FP6 activities of INFN Grid in Europe/2

• To promote EGEE for new scientific communities– GRIDCC (real time applications and instruments control)– BioInfoGrid (Bioinformatics: Coordinated by CNR)– LIBI (MIUR, Bionfomatics in Italy)– Cyclops (Civil Protection)

• To contribute to e-IRG, the e-Infrastructure Reflection Group born in Rome the December 2003 – Initiative of Italian Presidency on “eInfrastructures (Internet and

Grids) – The new foundation for knowledge-based Societies”Event organised by MIUR, INFN and EU Commission

– Representatives in EIRG appointed by EU Science Ministres– Policies and Roadmap for e-Infrastrutture development in EU

• To coordinate participation to Open Grid Forum (ex GGF)

25

Enabling Grids for E-sciencE

EGEE-II INFSO-RI-031688

INFN Grid/FP6 active projects

26

Enabling Grids for E-sciencE

EGEE-II INFSO-RI-031688

EGEE Middleware Distribution

• gLite– Exploit experience and existing

components from VDT (Condor, Globus), EDG/LCG, and others

– Develop a lightweight stack of generic middleware useful to EGEE applications (HEP and Life Sciences are pilot applications) Pluggable components – cater for different implementations Follow SOA approach, WS-I compliant where possible

– Focus is on re-engineering and hardening– Business friendly open source license

Moving to Apache-2

27

Enabling Grids for E-sciencE

EGEE-II INFSO-RI-031688

Parts of the Grid “ecosystem”

. . .

LCG

2004

2001

EGEE

Used in

USA EU

NextGrid DEISAGridCC

Future grids

EDG

Globus MyProxyCondor ...

VDT

DataTAG

CrossGrid ...

OSG, …

SRM

interactive

28

Enabling Grids for E-sciencE

EGEE-II INFSO-RI-031688

gLite approach

Foundation Grid MiddlewareSecurity model and infrastructure

Computing (CE) & Storage Elements (SE)

Accounting

Information providers and monitoring

Applications

Higher-Level Grid ServicesWorkload Management

Replica Management

Visualization

Workflows

Grid economies

etc.

• Provide specific solutions for supported applications

• Host services from other projects• More rapid changes than

Foundation Grid Middleware • Deployed as application software

using procedure provided by grid operations

• Application independent• Evaluate/adhere to new stds• Emphasis on robustness/stability

over new functionality• Deployed as a software

distribution by grid operations

Service-oriented approach:•Lightweight services•Allow for multiple interoperable implementations•Easily and quickly deployable•Use existing services where possible

by R.Jo

nes, 24/1

0/05

29

Enabling Grids for E-sciencE

EGEE-II INFSO-RI-031688

gLite Service Decomposition

API Access

Job Mgmt. Services

ComputingElement

WorkloadManagement

MetadataCatalog

Data Management

StorageElement

DataMovement

File & ReplicaCatalog

Authorization

Security Services

Authentication

Information &Monitoring

Information & Monitoring Services

Service

Discovering

Accounting

Auditing

JobProvenance

PackageManager

CLI

Network

Monitoring

Overview paper http://doc.cern.ch//archive/electronic/egee/tr/egee-tr-2006-001.pdf

30

Enabling Grids for E-sciencE

EGEE-II INFSO-RI-031688

Workload Management

API

Access

Job Mgmt Services

ComputingElement

WorkloadManagement

MetadataCatalog

Data Management

StorageElement

DataMovement

File & ReplicaCatalog

Authorization

Security Services

Authentication

Information &Monitoring

Information & Monitoring Services

Accounting

Auditing

JobProvenance

PackageManager

CLI

Network

Monitoring

Service

Discovering

31

Enabling Grids for E-sciencE

EGEE-II INFSO-RI-031688

Computing Element

• Three flavours available now: LCG-CE (GT2 GRAM)

In production now but will be phased-out soon

gLite-CE (GSI-enabled Condor-C) Already deployed but still needs

thorough testing and tuning. Being done now

CREAM (WS-I based interface) Deployed on the JRA1 preview test-bed.

After a first testing phase will be certified and deployed together with the gLite-CE

Our contribution to the OGF-BES group for a standard WS-I based CE interface

CREAM and WMProxy demo at SC06!• BLAH is the interface to the local

resource manager (via plug-ins)– CREAM and gLite-CE– Information pass-through: pass

parameters to the LRMS to help job scheduling

WMS,Clients

LRMSWN

bdIIR-GMACEMon

ComputingElement

glexec +LCAS/

LCMAPSBLAH

Gri

dS

ite

InformationSystem

32

Enabling Grids for E-sciencE

EGEE-II INFSO-RI-031688

gLite CE is evolving towards ICE-CREAM

• CREAM: web service Computing Element– Cream WSDL allows defining custom

user interface– C++ CLI interface allows direct

submission• Lightweight• Fast notification of job status changes

– via CEMon• Improved security

– no “fork-scheduler”• Will support for bulk jobs on the CE

– optimization of staging of input sandboxes for jobs with shared files

• ICE: Interface to Cream Environment– being integrated in WMS for

submissions to CREAM

33

Enabling Grids for E-sciencE

EGEE-II INFSO-RI-031688

Workload Management System (a.k.a. WMS, a.k.a. RB)

Job managementJob managementrequests (submission, requests (submission, cancellation) expressedcancellation) expressed

via a Job Descriptionvia a Job DescriptionLanguage (JDL)Language (JDL)

Finds an appropriateFinds an appropriateCE for each submission CE for each submission

request, taking into account request, taking into account job requests and preferences, job requests and preferences, Grid status, utilization policies Grid status, utilization policies

on resources on resources

Keeps submission Keeps submission requestsrequests

Requests are keptRequests are kept for a whilefor a while

if no matchingif no matchingresources availableresources available

Repository of resourceRepository of resource informationinformation

available to matchmakeravailable to matchmaker

Updated via notifications Updated via notifications and/or active and/or active

polling on resourcespolling on resources

Performs the actual Performs the actual job submission job submission and monitoring and monitoring

34

Enabling Grids for E-sciencE

EGEE-II INFSO-RI-031688

• WMS helps the user accessing computing resources – Resource brokering, management of job input/output, ...

• LCG-RB: GT2 + Condor-G– To be replaced when the gLite WMS proves reliability

• gLite WMS: Web service (WMProxy) + Condor-G– Management of complex workflows (DAGs) and compound jobs

bulk submission and shared input sandboxes support for input files on different servers (scattered

sandboxes)– Support for shallow resubmission of jobs– Job File Perusal: file peeking during job execution– Supports collection of information from CEMon, BDII, R-GMA and

from DLI and StorageIndex data management interfaces– Support for parallel jobs (MPI) when the home dir is not shared

Workload Management System (a.k.a. WMS, a.k.a. RB)

35

Enabling Grids for E-sciencE

EGEE-II INFSO-RI-031688

Workflows

• Direct Acyclic Graph (DAG) is a set of jobs where the input, output, or execution of one or more jobs depends on one or more other jobs

• A Collection is a group of jobs with no dependencies

– basically a collection of JDL’s

• A Parametric job is a job having one or more attributes in the JDL that vary their values according to parameters

• Using compound jobs it is possible to have one shot submission of a (possibly very large, up to thousands) group of jobs

– Submission time reduction Single call to WMProxy server Single Authentication and Authorization process Sharing of files between jobs

– Availability of both a single Job Id to manage the group as a whole and an Id for each single job in the group

nodeEnodeC

nodeA

nodeD

nodeB

36

Enabling Grids for E-sciencE

EGEE-II INFSO-RI-031688

Accounting - DGAS

• DGAS: accumulates Grid accounting information– User, JobId, user VO, VOMS FQAN(role,capabilities), SI2K, SF2K,

system usage (cpuTime, wallTime…),…

– allows billing and scheduling policies– levels of granularity: from single jobs to VO or grid aggregations– Privacy: only the user or VO manager can access information

site managers can keep accounting information available just for site internal analysis

– Sites can substitute DGAS metering system with their own

Usage accounting

Account balancing, resource pricing, (billing)

usage records

accounting data

Usage Analysis

Usage Metering

Interface

37

Enabling Grids for E-sciencE

EGEE-II INFSO-RI-031688

The Italian e-Infrastructure: Grid.IT

• In 2006 ended Grid.IT, the 3+1 years National Project funded by MIUR with 12 M€ (2002-05)

• Grid.IT has been crucial to develop and complete the R&D phase for operating a Italian Grid Infrastructure Grid (IGI):– developed tools and services required for a national Grid Operation

Centre (GOC) integrated with EU Regional Operation Centres (ROC)– The italian GOC is run by INFN -CNAF in Bologna, but the effort is

distributed among other italian centres

• Grid.IT allowed to extend the e-Infrastructure support from INFN to other sciences, following the model successfully implemented with the Research Network (GARR)− from INFNet to GARR

38

Enabling Grids for E-sciencE

EGEE-II INFSO-RI-031688

The Italian Production Grid

• 39 RCs(31 in EGEE)

• >3400 CPUs >360 TB

• 33 VOs

• gLite 3.0 +DGAS

http://grid-it.cnaf.infn.it

39

Enabling Grids for E-sciencE

EGEE-II INFSO-RI-031688

Resouces

40

Enabling Grids for E-sciencE

EGEE-II INFSO-RI-031688

Supported VOs

41

Enabling Grids for E-sciencE

EGEE-II INFSO-RI-031688

INFN-GRID Customisations

Release INFN-GRID 3.0 is a customisation of gLite 3.0:– Support for regional VOs– DGAS deployment (sensor + HLR server)– Network Monitor Element (NME), interfaced with GridIce for

data presentation– Customized tools to install and use the grid:

installation by a customized version of glite-yaim (ig-yaim) support to interface ig-yaim with a Quattor installation; UIPnP: a PlugAndPlay User Interface to access the grid as user of

every Linux system without RPMs

– AFS (read-only) on WN

42

Enabling Grids for E-sciencE

EGEE-II INFSO-RI-031688

Operation Structure and Organisation

The National Grid Central Management Team (CMT): – Activities:

‘integration’ and testing of the InfnGrid middleware release (based on gLite m/w release)

deployment procedures and configuration tools Monitoring and control of the status of the grid services and

resources

– Responsibilities: site registration procedure middleware deployment certification procedure for all InfnGrid sites Operation of the GRID services

43

Enabling Grids for E-sciencE

EGEE-II INFSO-RI-031688

Operations Support

• The Italian ROC provides local front line support to Virtual Organization, Users and Resources Centres

• The Italian Roc team is organized in daily shifts:– 2 people per shift, 2 shifts per day, from Monday to Friday.

• Activities planned during the shift– Log trouble tickets created, updated and closed, problems on

grid services and sites, monitor successful site certification– check the actions of the previous shift and the downtime page– check the status of production grid services and the GRIS

status of production CE and SE.– check the status of the production sites using the Site Functional

Tests report• Periodic (every 15 days) phone conferences

– ROC teams and site managers • Provide and write the ROC report for the weekly EGEE operation

meeting

44

Enabling Grids for E-sciencE

EGEE-II INFSO-RI-031688

Grid Monitoring

• The status of the Italian grid infrastructure is monitored using GridIce, – It is one of the monitoring tools used by EGEE– It is used to control

the status of the submitting queues Process/daemons status in the services (RB, BDII) VO view: list of CE and SE available for a the VOs and their status

and capacity Job monitoring

45

Enabling Grids for E-sciencE

EGEE-II INFSO-RI-031688

46

Enabling Grids for E-sciencE

EGEE-II INFSO-RI-031688

Accounting

47

Enabling Grids for E-sciencE

EGEE-II INFSO-RI-031688

Load

48

Enabling Grids for E-sciencE

EGEE-II INFSO-RI-031688

User, Operations and VO support

• The user support system provides tickets exchange between: – ROC on Duty and site managers– Site managers and Central management team and viceversa– Site manager and certification team during installation/upgrade– GGUS to ROC ROC to GGUS

• Italian ROC ticketing system is built upon a suite of web based tools written in PHP: Xhelp

• The support system components are accessible form the main interface of the deployment portal (grid-it.cnaf.infn.it) providing a SSO point of registration/identification certificate-based.

• The end-user can open a request, view and follow his own tickets and related replies;

• A supporter can view tickets assigned to his own groups, add responses and solutions, and change status/priority

• While operating tickets, a side content is always available for all classes of users (related to their access level) – Site Functional Tests, – site downtimes calendaring system– file archive– net query tools– IRC applet, contextual questions and answers– reports from daily shifts

49

Enabling Grids for E-sciencE

EGEE-II INFSO-RI-031688

Sustainability: Beyond EGEE-II

• Need to prepare for permanent Grid infrastructure– Ensure a reliable and adaptive support for all sciences– Independent of short project funding cycles– Infrastructure managed in collaboration

with national grid initiatives

50

Enabling Grids for E-sciencE

EGEE-II INFSO-RI-031688

FP7:guarantee sustainability

• The future of Grids in FP7 after 2008– EGEE proposed to European Parlament to set up an European Grid

Initiative (EGI) in order to: Guarantee long-term support & development to European e-

Infrastructure based on EGEE, DEISA and the Grid national projects being fundend by the National Grid Initiatives (NGI)

Provide a coordination framework at EU level as done for the research networks by Geant, DANTE and the National Networks like GARR

• The Commission asked that a plan for long-term sustainability Grid infrastructure (EGI + EGEE-III, …) to be included among the golas of EGEE-II (other than DANTE+ Geant 1-2)

• The building of EGI at EU level and of a National Grid Initiave at national level is among the main goals of FP7

51

Enabling Grids for E-sciencE

EGEE-II INFSO-RI-031688

Objectives in 2007: to define EGI

Objectives: – Operating the EU Productions Grid for all Sciences– Developing, integrating, testing, validating and distributing the Grid

middleware – Providing help, training and support to new users’ communities

EGI

52

Enabling Grids for E-sciencE

EGEE-II INFSO-RI-031688

The next step: EGI design study

• General agreement on the urgent need to prepare for a new “permanent” general EU Organization providing the evolution and operation of the Grid infrastructure at EU level– In collaboration with National Grid Initiatives (e.g.IGI)

• New FP7 proposal for a design study of a new European Grid Initiative (EGI) with the general objectives to:– Provide long term foundation for the operation of production Grid

infrastructures for all sciences in Europe in coordination with national initiatives

– Provide coordination at EU level for middleware development and standardization

– Integrate, test, validate and package Grid middleware for distribution

– Provide advice, training and support to new user communities

– Support industrial up taking

53

Enabling Grids for E-sciencE

EGEE-II INFSO-RI-031688

Italian objectives in 2007 :IGI

• In 2006 ended Grid.IT, the 3+1 years National Project funded by MIUR with 12 M€ (2002-05)

• The future: the Italian Grid Infrastructure (IGI) Association – EU (eIRG, ESFRI) requires the fusion of different pieces of National Grids into a

single National Organisation (NGI) to be unique interface to EU --> IGI for Italy– Substantial consensus for the cration of IGI for a common governance of the italian

e-Infrastructure from all involved public bodies:INFN Grid, S-PACI, ENEA Grid, CNR, INAF, Centri Nazionali di supercalcolo : CINECA, CILEA, CASPUR, and new consortia “nuovi PON”

– Under evaluation with MIUR the evolution of GARR towards a more general body to manage all the components of the infrastructure: Network, Grid, Digital Libraries…

• Crucial for INFN in 2007 will be to manage the transition from INFN Grid to IGI, in such a way to preserve and if possible enhance the organisation levels which allowed Italy to reach world leadership and become a leading partner of EGI

54

Enabling Grids for E-sciencE

EGEE-II INFSO-RI-031688 From the source...From the source...

Middleware Sustainability

55

Enabling Grids for E-sciencE

EGEE-II INFSO-RI-031688

Sustainability of MW in EU: OMII-EU

• OMII Europe (2006-08)– Project coordinated by UK, INFN and CERN representing EGEE – Convergence of INFN/Italian initiative and English Open

Middleware Institute Initiative (OMII) for a long-term development and support of Open Source Middleware

– Complement e-infrastructure national initiatives

• Objectives– Provisioning, support and quality certification of MW for the e-

Infrastructures– Evolution of MW towards OGF international standards– Inter-operation and inter-operability among Grid services provided

by the main MW developers: EGEE, Unicore, Globus and Condor

56

Enabling Grids for E-sciencE

EGEE-II INFSO-RI-031688

La diffusione della grid nell’industria e nella societa’

57

Enabling Grids for E-sciencE

EGEE-II INFSO-RI-031688

GILDA:Il laboratorio INFN per far provare la Grid

58

Enabling Grids for E-sciencE

EGEE-II INFSO-RI-031688

Il “Consorzio” Italiano per l’Open Middleware Enabling Grid Application (C-OMEGA)

Attivita’ di diffusione della tecnologia Grid all’industria. Obiettivi:• Ceare un punto di riferimento, anche per le attivita’ in ambito Europeo e

Internazionale, per la creazione, lo sviluppo, il supporto, la diffusione e lo sfruttamento della tecnologia Grid prodotta dai progetti Grid EU e nazionali all’interno di una piattaforma tecnologica Grid Open Source

• Rendere sinergici in questo campo la missione e gli obiettivi del mondo della ricerca e accademico, quelli del mondo industriale,in particolare ICT e PMI e dei grandi servizi pubblici nazionali (Ospedali, Scuole, Amministrazioni pubbliche)

• Creare supportare e mettere a disposizione dell’Industria e della societa’ sistemi grid allo stato dell’arte sfruttando le competenze accademiche di eccellenza in questo campo e dell’INFN in particolare

• Estendere con attivita’ d’ informazione, formazione e progetti mirati, lo sfruttamento delle tecnologie Grid in modo da far nascere nuove opportunita’ di crescita e di occupazione aumentando nello stesso tempo la competitivita’ globale del paese e dell’Europa

• 1 Idea progettuale approvata nell’ambito del PNR per la competitivita’– EGG

• Dovrebbe fornire una prima esperienza per il decollo di c-OMEGA

59

Enabling Grids for E-sciencE

EGEE-II INFSO-RI-031688

Partners hanno espresso interesse all’idea del consorzio/associazione c-OMEGA

• Le maggiori Istituzioni pubbliche di Ricerca: INFN, CNR, INAF, ICTP, varie Universita’…

• Consorzi: S-Paci, CRMPA….• Grandi Societa’ Nazionali: Unisis FIAT, RAI • Grandi industrie multinazionali IT• Grandi Industrie IT nazionali: Datamat,

Engineering• PMI: Nice, Eurix, Create-Net, Exadron,

Synapsis, Avanade, K-Solutions, Flextel • Consorzi pubblici: ECT(Tn), Pisa Ricerche, ITC

IRST (Tn) CEFRIEL (Lombardia) • Servizi: Societa’ per la telemedicina @TIM

60

Enabling Grids for E-sciencE

EGEE-II INFSO-RI-031688

EGG:La Grid per la pubblica amministrazione

• Idea progettuale PNR rivolta alla Pubblica Amministrazione • Regione Lombardia come riferimento+ INFN, Insiel, Soft. Houses• La pubblica amministrazione puo’ sfruttare i servizi Grid di base

gia’ sviluppati per il mondo della ricerca– Si possono in questo modo riutilizzare molte componenti Open

Source quali: Il sistema di sicurezza Grid Software Infrastructure basato su certificati

X509 Le procedure per l’emissione dei certificati da parte delle Certification

Authorities(CA) e le stesse CA riconsociute a livello mondiale Molti dei servizi di base per l’accesso a risorse di calcolo e ad archivi

distribuite • Servizi : condivisioni di dati e archivi a livello di comuni usando

servizi Grid generali• Servizi: Sanita’

– Accesso alle cartelle cliniche per studi epidemiologici, per quelli sulle malattie rare, per diminuire i costi di screening sistematici etc,

• Servizi: Emergenze nazionali – Sistemi per permettere un rapido accesso da parte delle protezione

civili a risorse di calcolo per lo studio dell’evoluzione di calamita’ naturali quali innondazioni, eruzioni, frane etc

• In corso di finalizzazione

61

Enabling Grids for E-sciencE

EGEE-II INFSO-RI-031688

Conclusions

• Grids and e-Infrastructures are the enabling platform to establish the European Research Area, and to promote in Europe the scientific and industrial advanced research, able to build the critical mass for innovating all the sectors

• the e-Infrastructures like EGEE based on grid may enhance the possibility to access knowledge, creating a new market of IT resources and applications

• Future sustainability of e-Infrastructures for e-Science has to be guaranteed– -> EGI at EU level (Design study proposal)– -> IGI at national level

• The European project for the Open Source Middleware OMII-EU can provide an important opportunity for long-term exploitation of the grid technology

62

Enabling Grids for E-sciencE

EGEE-II INFSO-RI-031688

Further information

• EGEE’07 Conference: 1-5 October 2007, Budapesthttp://www.eu-egee.org/egee07

• EGEE User Forum, 7-9 May 2007, Manchester http://www.eu-egee.org/uf2

• INFN-GRID grid.infn.it• EGEE www.eu-egee.org• gLite http://www.glite.org/• OMII-EUROPE www.omii-europe.org

• Open Grid Forum http://www.gridforum.org/• Globus Alliance http://www.globus.org/• VDT http://www.cs.wisc.edu/vdt/

NEW!!!