Grid Tutorial Cameron Kiddle Grid Research Centre University of Calgary.

Grid TutorialGrid Tutorial

Cameron KiddleCameron Kiddle

Grid Research CentreGrid Research CentreUniversity of CalgaryUniversity of Calgary

22Western Grid Summit - 200Western Grid Summit - 20055


AcknowledgmentsAcknowledgments

Rob SimmondsRob Simmonds

Mark FoxMark Fox

Phil RizkPhil Rizk

Abhishek GauravAbhishek Gaurav

Andrey MirtchovskiAndrey Mirtchovski

Nayden MarkatchevNayden Markatchev



OutlineOutline

Grid Computing OverviewGrid Computing Overview

Basic Grid ServicesBasic Grid Services security, data management, resource security, data management, resource

management, information servicesmanagement, information services

Advanced Grid ServicesAdvanced Grid Services meta schedulers, workflow managers, replica meta schedulers, workflow managers, replica

location servicelocation service

Future of Grid Computing Future of Grid Computing



Grid ComputingGrid Computing

Aimed at providing users with a single Aimed at providing users with a single system view of a set of distributed system view of a set of distributed resources that span administrative resources that span administrative domainsdomains

Aids in solving large scale problemsAids in solving large scale problems

Aids in collaborative research effortsAids in collaborative research efforts



Current Grid ProjectsCurrent Grid Projects

CERN LCG (LHC Computing Grid)CERN LCG (LHC Computing Grid)

GriPhyN (Grid Physics Network)GriPhyN (Grid Physics Network)

NEES (Network for Earthquake NEES (Network for Earthquake Engineering Simulation)Engineering Simulation)

BioSimGridBioSimGrid

IVOA (International Virtual Observatory IVOA (International Virtual Observatory Alliance)Alliance)



WestGrid and GridWestGrid and Grid

Is WestGrid a computational grid?Is WestGrid a computational grid?

Provides grid enabled resourcesProvides grid enabled resources Security servicesSecurity services Data transfer toolsData transfer tools Job submission servicesJob submission services

WestGrid resources can be part of WestGrid resources can be part of computational gridscomputational grids



Global Grid Forum (GGF)Global Grid Forum (GGF)

Leads the global standardization effort for Leads the global standardization effort for grid computinggrid computing

Open Grid Services Architecture (OGSA)Open Grid Services Architecture (OGSA)



Open Grid Services ArchitectureOpen Grid Services Architecture

Infrastructure ServicesInfrastructure ServicesResource Management ServicesResource Management ServicesData ServicesData ServicesContext ServicesContext ServicesInformation ServicesInformation ServicesSelf-Management ServicesSelf-Management ServicesSecurity ServicesSecurity ServicesExecution Management ServicesExecution Management Services



Grid MiddlewareGrid Middleware

Globus ToolkitGlobus Toolkit Globus AllianceGlobus Alliance Open sourceOpen source

LegionLegion University of VirginiaUniversity of Virginia Now commercial product developed by AvakiNow commercial product developed by Avaki Focuses on data grid / enterprise integrationFocuses on data grid / enterprise integration



Globus ToolkitGlobus Toolkit

Collection of grid services to aid in Collection of grid services to aid in development of grid applications and development of grid applications and environmentsenvironments

GT-2 – pre standards (defacto standard)GT-2 – pre standards (defacto standard)

GT-3 – Open Grid Services Infrastructure GT-3 – Open Grid Services Infrastructure (OGSI)(OGSI)

GT-4 – Web Services Resource GT-4 – Web Services Resource Framework (WSRF) – beta versionFramework (WSRF) – beta version



Basic Grid ServicesBasic Grid Services

Security ServicesSecurity Services GSI (Grid Security Infrastructure)GSI (Grid Security Infrastructure)

Data ManagementData Management GridFTPGridFTP

Resource ManagementResource Management GRAM (Grid Resource Allocation and GRAM (Grid Resource Allocation and

Management)Management)

Information ServicesInformation Services MDS (Monitoring and Discovery Service)MDS (Monitoring and Discovery Service)



Grid Security Infrastructure (GSI)Grid Security Infrastructure (GSI)

Based onBased on Public key cryptographyPublic key cryptography X.509 certificatesX.509 certificates Secure Socket Layer (SSL)Secure Socket Layer (SSL)

ProvidesProvides Mutual authenticationMutual authentication Single sign-onSingle sign-on



CertificatesCertificates

Used to identify and authenticate users Used to identify and authenticate users and servicesand servicesIncludes four primary pieces of informationIncludes four primary pieces of information Subject nameSubject name Public keyPublic key Identity of Certificate Authority (CA)Identity of Certificate Authority (CA) Digital signature of the Certificate AuthorityDigital signature of the Certificate Authority

Provides global name spaceProvides global name spaceEncoded in X.509 certificate format (IETF)Encoded in X.509 certificate format (IETF)



Mutual AuthenticationMutual Authentication

Process of two parties proving to each Process of two parties proving to each other who they say they areother who they say they are

Both parties must trust CAs that signed Both parties must trust CAs that signed each other’s certificateseach other’s certificates

Each party sends certificate to otherEach party sends certificate to other Make sure certificate is validMake sure certificate is valid Make sure other party really is the party Make sure other party really is the party

identified in the certificateidentified in the certificate



Single Sign-OnSingle Sign-On

Sign-on only once to computational gridSign-on only once to computational grid

Use a proxy to avoid re-entering passwordUse a proxy to avoid re-entering password Consists of new certificate and private keyConsists of new certificate and private key Certificate signed by owner, not CACertificate signed by owner, not CA Less secure – limited lifetimeLess secure – limited lifetime Both proxy certificate and owner’s certificate Both proxy certificate and owner’s certificate

sent for mutual authenticationsent for mutual authentication

grid-proxy-initgrid-proxy-init



GSI DemoGSI Demo



MyProxyMyProxy

Developed by NCSA (National Center for Developed by NCSA (National Center for Supercomputing Applications)Supercomputing Applications)

Credential repositoryCredential repository

Allows proxy credential to be received Allows proxy credential to be received from any machinefrom any machine

Can allow trusted servers to renew proxy Can allow trusted servers to renew proxy credentialscredentials

myproxy-init, myproxy-get-delegationmyproxy-init, myproxy-get-delegation



MyProxy DemoMyProxy Demo



File TransfersFile Transfers

gsiscpgsiscp scp version based on GSIscp version based on GSI Easy to use, but provides poor performance for large Easy to use, but provides poor performance for large

file transfersfile transfers

GridFTPGridFTP Based on FTP (File Transfer Protocol)Based on FTP (File Transfer Protocol) GSI security on control and data channelsGSI security on control and data channels Modification of TCP buffer sizesModification of TCP buffer sizes Parallel transfersParallel transfers Third party transfersThird party transfers



File Transfer PerformanceFile Transfer Performance



File Transfer DemoFile Transfer Demo



Grid Resource Allocation and Grid Resource Allocation and Management (GRAM)Management (GRAM)

Provides a single standard interface for Provides a single standard interface for remote job submission and resource remote job submission and resource managementmanagement

Requires users and application developers Requires users and application developers to learn only one method to gain access to to learn only one method to gain access to a large variety of local management a large variety of local management systemssystems

globus-job-run, globus-job-submitglobus-job-run, globus-job-submit



Job Submission DemoJob Submission Demo



Monitoring and Discovery Service Monitoring and Discovery Service (MDS)(MDS)

Provides information about availability and Provides information about availability and status of resourcesstatus of resourcesUses LDAP (Lightweight Directory Access Uses LDAP (Lightweight Directory Access Protocol) to access this informationProtocol) to access this informationGrid Resource Information Service (GRIS)Grid Resource Information Service (GRIS) Provides resource informationProvides resource information

Grid Index Information Service (GIIS)Grid Index Information Service (GIIS) Provides caching services and search Provides caching services and search

capabilitiescapabilities



Advanced Grid ServicesAdvanced Grid Services

Meta-schedulersMeta-schedulers

WorkflowsWorkflows

Replica Location Service (RLS)Replica Location Service (RLS)

Abstract WorkflowsAbstract Workflows



Meta-SchedulersMeta-Schedulers

Condor-G (Condor)Condor-G (Condor) University of Wisconsin-MadisonUniversity of Wisconsin-Madison

Moab Grid Scheduler (Silver)Moab Grid Scheduler (Silver) Cluster Resources Inc.Cluster Resources Inc.



Condor-GCondor-G

Can submit and manage jobs at multiple Can submit and manage jobs at multiple sitessites

Uses Globus mechanisms to communicate Uses Globus mechanisms to communicate with resources at different siteswith resources at different sites

Can advertise sites that jobs can be Can advertise sites that jobs can be submitted to through Class Adssubmitted to through Class Ads



Class AdClass Ad

Used to represent characteristics and Used to represent characteristics and constraints of resources and jobsconstraints of resources and jobs

Consists of a list of attributes (name/value Consists of a list of attributes (name/value pairs)pairs)

Used in matchmaking of jobs to resourcesUsed in matchmaking of jobs to resources



Condor-G DemoCondor-G Demo



WorkflowsWorkflows

A B

C1

C2 D

C3

E2

E1



DAGManDAGMan

Directed Acyclic Graph ManagerDirected Acyclic Graph Manager

Node = programNode = program

Edge = dependencyEdge = dependency

Monitors Condor log files to enforce Monitors Condor log files to enforce ordering of dependenciesordering of dependencies



DAGMan Input FileDAGMan Input File

Programs in the DAGPrograms in the DAG

Pre/post processing scriptsPre/post processing scripts

DependenciesDependencies

Number of times to retry Number of times to retry

Macro definitionsMacro definitions



Sample DAGMan Input FileSample DAGMan Input File

JOB A A.condor

JOB B B.condor

JOB C C.condor

JOB D D.condor

PARENT A CHILD B C

PARENT B C CHILD D

A

B C

D



Rescue DAGsRescue DAGs

Aids in resubmission of uncompleted Aids in resubmission of uncompleted portions of workflowsportions of workflows

Produced if DAGMan can not make any Produced if DAGMan can not make any further progress in a DAG due to node further progress in a DAG due to node failuresfailures

Same as original DAG but nodes that Same as original DAG but nodes that successfully completed before are marked successfully completed before are marked as DONEas DONE



Life3D WorkflowLife3D Workflow

Life3DSimulation

Rendering

Animation



Life3D Workflow - IILife3D Workflow - II

WestGrid Grid Research Center

blackhole (SFU)

lattice (UofC)

grc15

octarine

1.

2. Life3D Simulation

3.

4. Rendering5.

6. Animation

7.

Data Storage



Life3D Workflow - IIILife3D Workflow - III

Life3dSimulation

Rendering

AnimationStageData

StageData

StageData

StageData

blackhole blackholelattice

grc15

octarine



DAGMan DemoDAGMan Demo



Life3D AnimationLife3D Animation



Replica Location Service (RLS)Replica Location Service (RLS)

Co-developed by Globus and DataGrid projectsCo-developed by Globus and DataGrid projects

Provides a mechanism for registering and Provides a mechanism for registering and locating replicaslocating replicas

Maintains mappings between logical names and Maintains mappings between logical names and target namestarget names Logical names – unique identifiers for data content Logical names – unique identifiers for data content

that may have one or more physical replicasthat may have one or more physical replicas Target name – physical location of a replica or other Target name – physical location of a replica or other

logical names representing the datalogical names representing the data



Chimera Virtual Data SystemChimera Virtual Data System

Part of the GriPhyN projectPart of the GriPhyN project

Distributed with the Virtual Data ToolkitDistributed with the Virtual Data Toolkit

Keeps track of how data products are derivedKeeps track of how data products are derived

Chimera Pegasus

PartialWorkflow

(VDL)Virtual Data Language

AbstractWorkflow

ConcreteWorkflow

(DAX) (DAG)DAG XML

RLS TC MDS

(Transformation Catalog)



ChimeraChimera

Input - Virtual Data Language (VDL)Input - Virtual Data Language (VDL) Transformation – an executable programTransformation – an executable program Derivation – an execution of the Derivation – an execution of the

transformation along with logical input and transformation along with logical input and output file namesoutput file names

Output – Abstract Workflow (DAX)Output – Abstract Workflow (DAX) Workflow activities independent of specific Workflow activities independent of specific

grid resourcesgrid resources



PegasusPegasus

Input – Abstract WorkflowInput – Abstract Workflow

Output – Concrete Workflow (DAG)Output – Concrete Workflow (DAG) Workflow activities assigned to specific Gird Workflow activities assigned to specific Gird

resourcesresources

Uses RLS to determine data file locationsUses RLS to determine data file locations

Uses Transformation Catalog to map logical Uses Transformation Catalog to map logical transformations to physical locationstransformations to physical locations

Uses MDS to determine available resourcesUses MDS to determine available resources



Chimera ExampleChimera Example

add10 add20infile(blackhole)

(lattice) (nexus)

outfile(blackhole)

transitfile



Chimera DemoChimera Demo



SummarySummary

Security Services (GSI)Security Services (GSI) grid-proxy-initgrid-proxy-init myproxy-initmyproxy-init myproxy-get-delegationmyproxy-get-delegation

Data Management Data Management (GridFTP)(GridFTP)

globus-url-copyglobus-url-copy

Resource Management Resource Management (GRAM)(GRAM)

globus-job-runglobus-job-run globus-job-submitglobus-job-submit

Information ServicesInformation Services MDSMDS

Meta-schedulerMeta-scheduler Condor-GCondor-G

Concrete WorkflowsConcrete Workflows DAGManDAGMan

Replica Location Service Replica Location Service (RLS)(RLS)

Abstract WorkflowsAbstract Workflows Chimera/PegasusChimera/Pegasus

Basic Grid Services Advanced Grid Services



Future of Grid ComputingFuture of Grid Computing

Integration of Web Services and Grid ServicesIntegration of Web Services and Grid Services

Web Services Resource Framework (GT-4)Web Services Resource Framework (GT-4)

Standards are more establishedStandards are more established Involvement of more developersInvolvement of more developers Better toolsBetter tools

Grid-enabled applicationsGrid-enabled applications Common library interfaceCommon library interface ““run_job_on_grid”run_job_on_grid”

Wider spread useWider spread use



Contact InformationContact Information

Cameron Kiddle

[email protected]

http://grid.ucalgary.ca/

http://pages.cpsc.ucalgary.ca/~kiddlec/

Grid Tutorial Cameron Kiddle Grid Research Centre University of Calgary.

Documents

Transcript of Grid Tutorial Cameron Kiddle Grid Research Centre University of Calgary.