Overview of EGEE and gLite-1.ppt - Indico · 2018. 11. 16. · Enabling Grids for E-sciencE NA3...

Enabling Grids for E sciencEEnabling Grids for E-sciencE

Overview of the EGEE project and the gLite middlewareand the gLite middleware

www eu egee org

EGEE-III INFSO-RI-222667

www.eu-egee.org

Enabling Grids for E-sciencE

Outline

• What is EGEE?What is EGEE?– The project– The infrastructure

• gLite middlewareg

• EGEE applicationspp

• Sources of further informationSources of further information

EGEE-III INFSO-RI-222667 2


Defining the Grid

• A Grid is the combination• A Grid is the combination of networked resources and the corresponding middleware, which provides services for the userprovides services for the user.

EGEE-III INFSO-RI-222667 3Providing a Production Grid Infrastructure for Collaborative Science 3


The EGEE Project• Aim of EGEE:

“to establish a seamless European Grid infrastructure for the support of the European Research Area (ERA)”

• EGEE– 1 April 2004 – 31 March 2006– 71 partners in 27 countries, federated in regional Grids71 partners in 27 countries, federated in regional Grids

• EGEE-II– 1 April 2006 – 30 April 2008

E d d ti– Expanded consortium

• EGEE-III– 1 May 2008 – 30 April 2010y p– Transition to sustainable model



Defining the Grid




EGEE working with related infrastructure projects

GINEGEE-III INFSO-RI-222667 6Providing a Production Grid Infrastructure for Collaborative Science 6

GIN


What is happening now?

Real Time Monitor – Java tool– Displays jobs

running (submitted through RBs)g )

– Shows jobs moving around

ld iworld map in real time, along with changes in status


http://gridportal.hep.ph.ic.ac.uk/rtm/ (snapshot 16 January 2007)


EGEE Infrastructures

• Production service– Scaling up the infrastructure with resource centres around the globe– Stable, well-supported infrastructure, running only well-tested and

reliable middleware

• Pre-production service– Run in parallel with the production service (restricted nr of sites)– First deployment of new versions of the gLite middleware – Test-bed for applications and other external functionality

• T-Infrastructure (Training&Education)– Complete suite of Grid elements 20 sites on 3 continentsp

and application (Testbed, CA, VO, monitoring, support, …)

– Everyone can register and use


Everyone can register and use GILDA for training and testing


NA3 activity: User training and induction

• Expand portfolio of training materials & coursesT i id i f EGEE

NA12%

NA25%

NA38%

NA419%

SA22%

SA39%

JRA15%

• Train a wide variety of EGEE users (internal/external)

• Develop effective mechanisms SA149%

for training end-users of the EGEE infrastructure

• Collaborate in cross-activity

NA51%29 Active

partners ~ 29 FTEsyinitiatives

– ICEAGE Project Digital Library– http://library.iceage-eu.org/

~ 29 FTEs89 Individuals6 Federations

y g g– Videos, MP3 talks on grid computing

• http://www.egee.nesc.ac.uk/– Training eventsg– Training material repository

• http://egee.lib.ed.ac.uk/– EGEE Digital Library


EGEE Digital Library– Repository of training materials


NA4 Activity: Application identification and support

• Application Identification and Support (NA4)– 25 countries, 40 partners, 280+ participants, 1000s of users

S t th l d di EGEE it• Support the large and diverse EGEE user community:– Promote dialog: Users’ Forums & EGEE Conferences– Technical Aid: Porting support, procedural issues– Liaison: Software and operational requirements

• Main activities:– 5 application clusers: HEP Life sciences Astronomy & astrophysics5 application clusers: HEP, Life sciences, Astronomy & astrophysics, Earth science, Computational chemistry, Fusion, Grid observatory– Support:

Application porting supportApplication porting supportwww.lpds.sztaki.hu/gasuc

VO supportDirect user supportDirect user support

www.ggus.orgRegional support

• http://egeena4 lal in2p3 fr


http://egeena4.lal.in2p3.fr


EGEE Infrastructure

Country participating

in EGEE

600007000080000

No. Cores

0100002000030000400005000060000

300No. Sites

0

Apr

-04

Jul-0

4

Oct

-04

Jan-

05A

pr-0

5

Jul-0

5

Oct

-05

Jan-

06

Apr

-06

Jul-0

6

Oct

-06

Jan-

07

Apr

-07

Jul-0

7

Oct

-07

Jan-

08

Apr

-08

> 200 sites in 40 countries38 000 CPU

50100150200250300 ~ 38 000 CPUs

~ 5 PB storage98k jobs/day> 200 Virtual Organizations


050

Apr

-04

Jul-0

4

Oct

-04

Jan-

05

Apr

-05

Jul-0

5

Oct

-05

Jan-

06

Apr

-06

Jul-0

6

Oct

-06

Jan-

07

Apr

-07

Jul-0

7

Oct

-07

Jan-

08

Apr

-08

> 200 Virtual Organizations⇨The world’s largest multi-disciplinary Grid


Resource management: structure

• Operations Coordination Centre (OCC)

– management oversight of– management, oversight of all operational and support activities• Regional Operations

Centres (ROC)Centres (ROC)– providing the core of the

support infrastructure, each supporting a number of resource centres within its region

– Grid Operator on Duty • Resource centres– providing resourcesproviding resources

(computing, storage, network, etc.);• Grid User Support

(GGUS)(GGUS)– At FZK, coordination and

management of user support, single point of contact for users



VO concept

• gLite middleware runs on each shared resource to provideresource to provide– Data services– Computation

services

INTERNET

services – Security service

• Resources and users form Virtual organisations: basis for collaboration

• Distributed services (both people and middleware) enable


)the grid


Defining the Grid




Grid middleware

• The Grid relies on advanced software, called middleware, which interfaces between resources and the applications

Th G id iddl• The Grid middleware:– Basic services

Secure and effective access Secu e a d e ect e accessto resources

– High level servicesOptimal use of resourcesOptimal use of resourcesAuthentication to the different sites that are usedJob execution & monitoringJob execution & monitoring of progressProblem recoveryT f f lt b k t th


Transfer of results back to the user


Grid Middleware

• When using a PC or workstation you

• When using a Grid you– Login with digitalworkstation you

– Login with a username and password (“Authentication”)

Login with digital credentials – single sign-on (“Authentication”)

( Authentication )– Use rights given to you

(“Authorisation”)

– Use rights given you (“Authorisation”)

– Run jobs– Run jobs– Manage files: create

them, read/write, list

– Run jobs– Manage files: create

them, read/write, list them, read/write, list directories

• Components are li k d b b

directories• Services are linked by

th I t tlinked by a bus• Operating system • One admin domain

the Internet• Middleware

M d i d iEGEE-III INFSO-RI-222667 16

• One admin. domain • Many admin. domains


EGEE Middleware: gLite

• gLite 3.0, gLite 3.1⇨ Merger of LCG 2 7 and GLite 1 5⇨ Merger of LCG 2.7 and GLite 1.5

Exploit experience and existing components from– Exploit experience and existing components from VDT (Condor, Globus), EDG/LCG, and others

– Develop a lightweight stack of generic middlewareDevelop a lightweight stack of generic middlewareuseful to EGEE applications (HEP and Biomedics are pilot applications). p pp )

Should eventually deploy dynamically (e.g. as a globus job)Pluggable components – cater for different implementations

– Focus is on providing a stable and usable infrastructure



Basic gLite use case:Job submissionInformation System

Submit job(executable + small inputs) Resource Broker

User Interface

( p )

query

Resource Broker

create

Retrieve status & (small) output files

Submit job publishquery

createproxy

(small) output files

Submit jobRetrieve

output

pstate

File and Replica CatalogJob

status Logging

Computing Element Storage Element

Site X

J bInput file(s)

Register file

VO Management Service

process

Job status

p ( )

Output file(s)


(DB of VO users)Logging and bookkeeping

Ou pu e(s)


Main components

User Interface (UI)User Interface (UI): The place where users logon to the Grid

Resource Broker (RB) (Workload Management System (WMS)Resource Broker (RB) (Workload Management System (WMS):Matches the user requirements with the available resources on the Grid

Information SystemInformation System: Characteristics and status of CE and SE

File and replica catalogFile and replica catalog: Location of grid files and grid file replicas

L i d B kk i (LB)L i d B kk i (LB) L i f ti f j bLogging and Bookkeeping (LB)Logging and Bookkeeping (LB): Log information of jobs

Computing Element (CE)Computing Element (CE): A batch queue on a site’s computers wherethe user’s job is executed


Storage Element (SE)Storage Element (SE): provides (large-scale) storage for files


Main components

User Interface (UI)User Interface (UI): The place where users logon to the Grid

Resource Broker (RB) (Workload Management System (WMS)Resource Broker (RB) (Workload Management System (WMS):Matches the user requirements with the available resources on the Grid

Information SystemInformation System: Characteristics and status of CE and SEAll built upon authorisationFile and replica catalogFile and replica catalog: Location of grid files and grid file replicas

L i d B kk i (LB)L i d B kk i (LB) L i f ti f j b

authorisation, authentication,

Logging and Bookkeeping (LB)Logging and Bookkeeping (LB): Log information of jobssecurityComputing Element (CE)Computing Element (CE): A batch queue on a site’s computers where

the user’s job is executed


Storage Element (SE)Storage Element (SE): provides (large-scale) storage for files


Who provides the resources?!

ServiceService Provider NoteUser interfaceUser interface User / institute / VO Computer with client SWp

Resource Broker Resource Broker (WMS)(WMS)

VOs - EGEE does not fund RBs

Information SystemInformation System Grid operations -EGEE funded effort

File and replica catalogFile and replica catalog VOs - EGEE does p gp gnot fund catalogs

Logging and Logging and BookkeepingBookkeeping

VOs - EGEE does not fund LB serversBookkeepingBookkeeping not fund LB servers

Computing Element Computing Element (CE)(CE)

VOs - EGEE does not fund CEs

VOs provide resources to match average need

Storage Element Storage Element (SE)(SE)

VOs - EGEE does not fund SEs

VOs provide resources to match average need

External services User / institute / VO To extend the capabilities of


External services User / institute / VO To extend the capabilities of the core infrastructure


Empowering VOs Where computer science meets the application communities!

– Recommended External Software Packages for Egee CommuniTies Application g C

– Current RESPECT tools:GridWayP-GRADE PortalGANGA

Applicationtoolkits

Application

GRelCI2glogin

– http://egeena4.lal.in2p3.fr/ “Grid software” menuHigher level gLite services

toolkitsCommand line & APIs

Basic gLite services:

Higher-level gLite services (WMS,…) Production infrastructure contains

these servicesBasic gLite services:CE, SE, info, security – Basic services: Must be complete and robust; Should not assume the

use of Higher-Level Grid Services– High level services: help the users

building their computing infrastructure but should not be


mandatory


Defining the Grid




EGEE Applications

• >270 VOs from several scientific domains

Astronomy & Astrophysics– Astronomy & Astrophysics– Civil Protection– Computational Chemistry– Comp. Fluid Dynamics– Computer Science/Tools– Condensed Matter Physicsy– Earth Sciences– Fusion

High Energy Physics– High Energy Physics– Life Sciences

• Further applications d l tiunder evaluation

Applications have moved from


Applications have moved from testing to routine and daily usage

~80-95% efficiency


Application families

• Simulation– Large number of similar, independent jobs – parameter study

• Bulk Processing– Widely-distributed input data, Sophisticated data management

Workflow• Workflow– Complex dependencies between individual tasks

• Legacy Applicationsg y pp– Licenses: control access to software on the grid– No recompilation ⇒ no direct use of grid APIs

P ll l J b• Parallel Jobs– Many CPUs needed simultaneously, Use of MPI libraries– Limited support in gLite: MPI configuration is not uniformLimited support in gLite: MPI configuration is not uniform

• Responsive Apps.– Short response time


– No real support in gLite Interactive Grid FP6 project


Further information, references

• EGEE– http://www.eu-egee.org/p g g

• gLite middleware– http://www.glite.org

• gLite manuals, documentation– http://glite.web.cern.ch/glite/documentation/

(gLite user guide)• Recommended External Software Packages for Egee

CommuniTies (RESPECT)CommuniTies (RESPECT)– http://egeena4.lal.in2p3.fr/

• Description of work of EGEE-IIIhttps://edms cern ch/document/886385/4


– https://edms.cern.ch/document/886385/4


Summary

• EGEE is running the largest multi-VO grid in the world!– Creating the “grid layer” in e-Infrastructure for research, public

service and industry• Key concepts for EGEE• Key concepts for EGEE

– Sustainability – planning for the long-term– Production qualityProduction quality – User support

• EGEE’s middleware: gLite. Current version 3.1g– Basic middleware services– High level middleware services

• External software to foster uptake of technology


Overview of EGEE and gLite-1.ppt - Indico · 2018. 11. 16. · Enabling Grids for E-sciencE NA3...

Documents

Transcript of Overview of EGEE and gLite-1.ppt - Indico · 2018. 11. 16. · Enabling Grids for E-sciencE NA3...