Overview of middleware

FP6−2004−Infrastructures−6-SSA-026409

E-infrastructure shared between Europe and Latin America

Alexandre DuarteCERN IT-GD-OPSUFCG LSD1st EELA Grid SchoolItacuruçá, 04-15/12/2006

Overview of middleware

1st EELA Grid School, Itacuruçá, 04-15.12.2006FP6−2004−Infrastructures−6-SSA-026409


Job Workflow in gLite

UIJDL

Logging &Logging &Book-keepingBook-keeping

ResourceResourceBrokerBroker

Job SubmissionJob SubmissionServiceService

StorageStorageElementElement

ComputingComputingElementElement

Information Information ServiceService

Job Status

LFCLFCCatalogCatalog

DataSets info

Author.&Authen.

Job S

ub

mit

Even

t

Job

Qu

ery

Job

Stat

us

Input “sandbox”

Input “sandbox” + Broker InfoGlobus RSL

Output “sandbox”

Output “sandbox”

Job Status

Pu

blis

h

vom

s-pr

oxy-

init

Exp

and

ed J

DL

SE & CE info



Middleware structure

• Applications have access both to Higher-level Grid Services and to Foundation Grid Middleware

• Higher-Level Grid Services are supposed to help the users building their computing infrastructure but should not be mandatory

• Foundation Grid Middleware will be deployed on the EGEE infrastructure– Must be complete and robust– Should allow interoperation

with other major grid infrastructures

– Should not assume the use of Higher-Level Grid Services



Grid Foundation: Security

• Authentication based on X.509 PKI infrastructure– Certificate Authorities (CA) issue (long lived) certificates

identifying individuals (much like a passport) Commonly used in web browsers to authenticate to sites

– Trust between CAs and sites is established (offline)– In order to reduce vulnerability, on the Grid user identification is

done by using (short lived) proxies of their certificates

• Proxies can– Be delegated to a service such that it can act on the user’s

behalf– Include additional attributes (like VO information via the VO

Membership Service VOMS)– Be stored in an external proxy store (MyProxy) – Be renewed (in case they are about to expire)



Virtual Organization Membership Service:– Extends the proxy with info on VO

membership, group, roles– Fully compatible with GSI– Each VO has a database containing

group membership, roles and capabilities informations for each user

– User contacts VOMS server requesting his authorization info

– Server sends authorization info to the client, which includes it in a proxy certificate

[sam@egris195 sam]$ voms-proxy-init -voms gilda

Your identity: /C=IT/O=GILDA/OU=Personal Certificate/L=ITACURUCA/CN=ITACURUCA13/[email protected]

Enter GRID pass phrase:

Creating temporary proxy ............................ Done

Contacting egris197.eela.ufrj.br:15001 [/C=BR/O=ICPEDU/O=UFF BrGrid CA/O=UFRJ/OU=IF/CN=host/egris197.eela.ufrj.br] "gilda" Done

Creating proxy .............................................. Done

Your proxy is valid until Tue Dec 5 10:55:10 2006

Grid Foundation: VOMSQuery

Authentication

Request

AuthDB

C=IT/O=INFN /L=CNAF/CN=Pinco Palla/CN=proxy

VOMSAC

VOMSAC



Grid foundation: Information Systems

• BDII: Information system in gLite 3.0 (by LCG)– LDAP database that is updated by a process – More than one DBs is used separate read and write– A port forwarder is used internally to select the correct DB

• R-GMA: provides a uniform method to access and publish distributed information and monitoring data– Used for job and infrastructure monitoring in gLite 3.0



Grid foundation: Computing Element

• gLite-CE: based on GSI enabled Condor-C– Coming from LCG-CE (based on GT2 GRAM)– Supported by Condor. More efficient.– Deployed for the first time in gLite 3.0

• CREAM: new lightweight CE– Not yet in gLite 3 release. Will need exposure to users on

dedicated system.– WSDL interface– Will support bulk submission of jobs from WMS and optimization

of input/output file transfer.– Plans are to have a CE with both Condor-C and CREAM

interfaces



Grid foundation: Storage Element

• Storage Element– Common interface: SRMv1,migrating to SRMv2– Various implementation from LCG and other external projects

disk-based: DPM, dCache / tape-based: Castor, dCache

– Support for ACLs in DPM (in future in Castor and dCache)– Common rfio library for Castor and DPM being added

• Posix-like file access:– Grid File Access Layer (GFAL) by LCG

Support for ACL in the SRM layer (currently in DPM only) Support for SRMv2 being added



High Level Services: File transfer

• FTS: Reliable, scalable and customizable file transfer– Manages transfers through channels

mono-directional network pipes between two sites

– Web service interface



High Level Services: Workload mgmt.

• WMS helps the user accessing computing resources – Resource brokering, management of job input/output, ...

• gLite-RB: GT2 + Condor-G– To be replaced when the gLite WMS proves to be reliable

• gLite WMS: Web service (WMProxy) + Condor-G– Management of complex workflows (DAGs) and compound jobs

bulk submission and shared input sandboxes support for input files on different servers (scattered sandboxes)

– Job File Perusal: file peeking during job execution– Supports collection of information from CEMon, BDII, R-GMA

and from DLI and StorageIndex data management interfaces– Support for parallel jobs (MPI) when the home dir is not shared– Deployed for the first time in gLite 3.0



High Level Services: Workflows

• Direct Acyclic Graph (DAG) is a set of jobs where the input, output, or execution of one or more jobs depends on one or more other jobs

• A Collection is a group of jobs with no dependencies– basically a collection of JDL’s

• A Parametric job is a job having one or more attributes in the JDL that vary their values according to parameters

• Using compound jobs it is possible to have one shot submission of a (possibly very large, up to thousands) group of jobs – Submission time reduction

Single call to WMProxy server Single Authentication and Authorization process Sharing of files between jobs

– Availability of both a single Job ID to manage the group as a whole and an ID for each single job in the group

nodeEnodeC

nodeA

nodeD

nodeB



www.glite.org

Questions ?

Overview of middleware

Documents

Transcript of Overview of middleware