INFSO-RI-508833 Enabling Grids for E-sciencE Grid Infrastructure & Related Projects Eddie Aronovich...

INFSO-RI-508833

Enabling Grids for E-sciencE

www.eu-egee.org

Grid Infrastructure & Related ProjectsEddie Aronovich

Tel-Aviv University, School of CS

[email protected]

Eddie Aronovich – CS_IAP, Apr 2007 2


INFSO-RI-508833

Acknowledgements

• Presentation is based on slides from:– Roberto Barbera, University of Catania and INFN (EGEE Tutorial

Roma, 02.11.2005)– Mike Mineter, Concepts of grid computing– Fabrizio Gagliardi, EGEE Project Director, CERN, Geneva,

Switzerland (Naregi Symposium 2005 – Tokyo)– Fabrizio Gagliardi, EGEE Project Director, CERN, Geneva,

Switzerland (APAC, 27 September 2005)– Guy Warner, NeSC Training Team (An Induction to EGEE for

GOSC and the NGS NeSC, 8th December 2004 )– http://www.beingrid.eu



INFSO-RI-508833

• What is a grid ?

• EGEE project

• Connectivity with other projects

• How a job is submitted

• A few more words



INFSO-RI-508833

When the Grid can do ?

• Coordinates Distributed Resources

• Open for connectivity (using standard, open, general-purpose protocols)

• Deliver non-trivial qualities of service



INFSO-RI-508833

A grid

• Grid middleware runs on each shared resource– Data storage– (Usually) batch

jobs on pools of processors

• Users join VO’s • Virtual organisation

negotiates with sites to agree access to resources

• Distributed services (both people and middleware) enable the grid

INTERNET



INFSO-RI-508833


• EGEE project





INFSO-RI-508833

EGEE project in 1K words

https://goc.grid-support.ac.uk/gridsite/monitoring/



INFSO-RI-508833

30+ Grid Projects @ EGEE’06



INFSO-RI-508833

20 + Businesses @ EGEE06

Capitalising on e-Science to make e-Business



INFSO-RI-508833

Related EU projects

EUGRIDGRID

Di l i gentA DIgital Library Infrastructureon Grid ENabled Technology

ISSeG



INFSO-RI-508833

EGEE as partner

• Ongoing collaborations– with non EU partners in EGEE: US, Israel, Russia, Korea, Taiwan… – with other European infrastructure projects:

GÉANT DEISA SEE-GRID

– with other European grid projects DILIGENT GRIDCC

– with non-European projects: OSG: OpenScienceGrid (USA) NAREGI (Japan)

• EGEE as incubator– >10 related projects have been created



INFSO-RI-508833

Collaborating e-Infrastructures

Potential for linking ~80 countries

TWGRID



INFSO-RI-508833

e-Infrastructure

• implementation blocks

From a talk by Mario Campolargo, Brussels, 30 May 2005



INFSO-RI-508833

Support for digital repositories

GÉANT .

INFRASTRUCTURE

GRID .

INFRASTRUCTURE

KNOWLEDGE .

INFRASTRUCTURE

3 layered model to support access to heterogeneousinformation and connect resources throughcommon shared services

Grids for digital repositories:• Sharing of resources

• Secure Access Control

• Data management

• Execution of computationally demanding applications (e.g. multi-media content)



INFSO-RI-508833


• EGEE project






INFSO-RI-508833

Jobs life (major components)

ReplicaReplicaCatalogueCatalogue

Logging &Logging &Book-keepingBook-keeping

ResourceResourceBrokerBroker

StorageStorageElementElement

ComputingComputingElementElement

Information Information ServiceService

Job Status

DataSets info

Author.&Authen.

Job S

ub

mit

Even

t

Job

Qu

ery

Job

Stat

us

Input “sandbox”

Input “sandbox” + Broker Info

Output “sandbox”

Output “sandbox”

Pu

blis

h

SE & CE info

““User User interface”interface”



INFSO-RI-508833

UI

NetworkServer

Job Contr.

WorkloadManager

ReplicaLocationServer

Inform.Service

ComputingElement

StorageElement

RB/WMS node

Characts.& status



INFSO-RI-508833

UI

NetworkServer

Job Contr.-

CondorG

WorkloadManager


Inform.Service

ComputingElement

StorageElement

RB node

CE characts& status

SE characts& status

submitted

Job Status

UI: allows users to access the functionalitiesof the WMS(via command line, GUI, C++ and Java APIs)



INFSO-RI-508833

UI

NetworkServer

Job Contr.-

CondorG

WorkloadManager


Inform.Service

ComputingElement

StorageElement

RB node

CE characts& status

SE characts& status

edg-job-submit myjob.jdlMyjob.jdl

JobType = “Normal”;Executable = "$(CMS)/exe/sum.exe";InputSandbox = {"/home/user/WP1testC","/home/file*”, "/home/user/DATA/*"};OutputSandbox = {“sim.err”, “test.out”, “sim.log"};Requirements = other. GlueHostOperatingSystemName == “linux" && other. GlueHostOperatingSystemRelease == "Red Hat 7.3“ && other.GlueCEPolicyMaxCPUTime > 10000;Rank = other.GlueCEStateFreeCPUs;

submitted

Job Status

Job Description Language(JDL) to specify job characteristics and requirements



INFSO-RI-508833

UI

NetworkServer

Job Contr.-

CondorG

WorkloadManager


Inform.Service

ComputingElement

StorageElement

RB node

CE characts& status

SE characts& status

RBstorage

Input Sandboxfiles

Jobwaiting

submitted

Job StatusNS: network daemon

responsible for acceptingincoming requests



INFSO-RI-508833

UI

NetworkServer

Job Contr.-

CondorG


Inform.Service

ComputingElement

StorageElement

RB node

CE characts& status

SE characts& status

RBstorage

waiting

submitted

Job Status

WM: acts to satisfy the request

Job

Workload manager



INFSO-RI-508833

UI

NetworkServer

Job Contr.-

CondorG

WorkloadManager


Inform.Service

ComputingElement

StorageElement

RB node

CE characts& status

SE characts& status

RBstorage

waiting

submitted

Job Status

Match-Maker/Broker

Where must thisjob be executed ?



INFSO-RI-508833

UI

NetworkServer

Job Contr.-

CondorG

WorkloadManager


Inform.Service

ComputingElement

StorageElement

RB node

CE characts& status

SE characts& status

RBstorage

waiting

submitted

Job Status

Match-Maker/ Broker

Matchmaker: responsible to find the “best” CE for a job



INFSO-RI-508833

UI

NetworkServer

Job Contr.-

CondorG

WorkloadManager


Inform.Service

ComputingElement

StorageElement

RB node

CE characts& status

SE characts& status

RBstorage

waiting

submitted

Job Status

Match-Maker/ Broker

Where are (which SEs) the needed data ?

What is thestatus of the

Grid ?



INFSO-RI-508833

UI

NetworkServer

Job Contr.-

CondorG

WorkloadManager


Inform.Service

ComputingElement

StorageElement

RB node

CE characts& status

SE characts& status

RBstorage

waiting

submitted

Job Status

Match-Maker/Broker

CE choice



INFSO-RI-508833

UI

NetworkServer

Job Contr.-

CondorG

WorkloadManager


Inform.Service

ComputingElement

StorageElement

RB node

CE characts& status

SE characts& status

RBstorage

waiting

submitted

Job Status

JobAdapter

Job Adapter: responsible for the final “touches” to the job before performing submission(e.g. creation of wrapper script, PFN, etc.)



INFSO-RI-508833

UI

NetworkServer

Job Contr.

WorkloadManager


Inform.Service

ComputingElement

StorageElement

RB node

CE characts& status

SE characts& status

RBstorage

Job Status

Job Controller: responsible for theactual job managementoperations (done via CondorG)

Job

submitted

waiting

ready



INFSO-RI-508833

UI

NetworkServer

Job Contr.-

CondorG

WorkloadManager


Inform.Service

ComputingElement

StorageElement

RB node

CE characts& status

SE characts& status

RBstorage

Job Status

Job

submitted

waiting

ready

scheduled



INFSO-RI-508833

UI

NetworkServer

Job Contr.-

CondorG

WorkloadManager


Inform.Service

ComputingElement

StorageElement

RB node

RBstorage

Job Status

submitted

waiting

ready

scheduled

running

“Grid enabled”data transfers/

accesses

Job

InputSandboxfiles



INFSO-RI-508833

UI

NetworkServer

Job Contr.-

CondorG

WorkloadManager


Inform.Service

ComputingElement

StorageElement

RB node

RBstorage

Job Status

OutputSandboxfiles

submitted

waiting

ready

scheduled

running

done



INFSO-RI-508833

UI

NetworkServer

Job Contr.-

CondorG

WorkloadManager


Inform.Service

ComputingElement

StorageElement

RB node

RBstorage

Job Status

submitted

waiting

ready

scheduled

running

done

edg-job-get-output <dg-job-id>



INFSO-RI-508833

UI

NetworkServer

Job Contr.-

CondorG

WorkloadManager


Inform.Service

ComputingElement

StorageElement

RB node

RBstorage

Job Status

OutputSandboxfiles

submitted

waiting

ready

scheduled

running

done

cleared



INFSO-RI-508833

UI

Log Monitor

Logging &Bookkeeping

NetworkServer

Job Contr.-

CondorG

WorkloadManager

ComputingElement

RB node

LM: parses CondorG logfile (where CondorG logsinfo about jobs) and notifies LB

LB: receives and stores job events; processes corresponding job status

Log ofjob events

edg-job-status <dg-job-id>edg-job-get-logging-info <dg-job-id>

Job status



INFSO-RI-508833

What about security ?

Certification

Domain A

Server X Server Y

PolicyAuthority

PolicyAuthority

TaskDomain B

Sub-Domain A1

GSI

CertificationAuthority

Sub-Domain B1

Authority

FederationService

VirtualOrganization

Domain

No Cross-

Domain Trust



INFSO-RI-508833


• EGEE project






INFSO-RI-508833

Used CPU time this year



INFSO-RI-508833

Over the Chasm



INFSO-RI-508833

Applications in EGEE

• Astrophysics - MAGIC, Planck • Computational Chemistry • Earth Sciences - Earth Observation, Solid Earth Physics, Hydrology, Climate • Financial Simulation - E-GRID • Fusion • Geophysics - EGEODE • High Energy Physics - LHC (ALICE, ATLAS, CMS, LHCb) BaBar, CDF, DØ, ZEUS

• Life Sciences - Bioinformatics (Drug Discovery, GPS@, Xmipp_MLrefine, etc.) • Medical imaging (GATE, CDSS, gPTM3D, SiMRI 3D, etc.) • Multimedia

press for more details



INFSO-RI-508833

Steps to become grid production

• Contact IAGneeded contribution:– 30CPUs or cores– 1TB of storage attached to the processosrs– 4 computers for operational activities

• Plan your site

• Get a certificates

• Install the site

• Certify the site….

INFSO-RI-508833


www.eu-egee.org

Questions ?



INFSO-RI-508833

Contacts

• Israeli Academic Grid (IAG)http://iag.iucc.ac.il/

• EGEE Websitehttp://www.eu-egee.org

• Grid Demoshttps://glite-demo.ct.infn.it/

INFSO-RI-508833 Enabling Grids for E-sciencE Grid Infrastructure & Related Projects Eddie Aronovich...

Documents

Transcript of INFSO-RI-508833 Enabling Grids for E-sciencE Grid Infrastructure & Related Projects Eddie Aronovich...