Globus GRAM for Developers

Stuart Martin, Peter Lane

Argonne National Lab

2

Session OverviewQ: What is this session about?

A:This presentation will cover the features, interface, architecture, performance, and future plans of the Globus Toolkit v4 Web Services Grid Resource Allocation and Management (GRAM4) component.

Four-part discussion (~ 20 mins/each) Overview of GRAM Model How to use client software How to administer servers Future plans

3

GRAM: Part 1

Overview of GRAM Model…

4

What is GRAM? GRAM is a Globus Toolkit component

For Grid job management

GRAM is a unifying remote interface to Resource Managers Yet preserves local site security/control

GRAM is for stateful job control Reliable operation Asynchronous monitoring and control Remote credential management File staging via RFT and GridFTP

5

Grid Job Management Goals

Provide a service to securely: Create an environment for a job Stage files to/from environment Cause execution of job process(es)

Via various local resource managers Monitor execution Signal important state changes to client

Enable client access to output files Streaming access during execution

6

Job Submission Model

Create and manage one job on a resource Submit and wait Not with an interactive TTY

File based stdin/out/err Supported by all batch schedulers

More complex than RPC Optional steps before and after submission message

Job has complex lifecycle Staging, execution, and cleanup states But not as general as Condor DAG, etc.

Asynchronous monitoring

7

Job Submission Options Optional file staging

Transfer files “in” before job execution Transfer files “out” after job execution

Optional file streaming Monitor files during job execution

Optional credential delegation Create, refresh, and terminate delegations For use by job process For use by GRAM to do optional file staging

8

Job Submission Monitoring

Monitor job lifecycle GRAM and scheduler states for job

StageIn, Pending, Active, Suspended, StageOut, Cleanup, Done, Failed

Job execution status Return codes

Multiple monitoring methods Simple query for current state Asynchronous notifications to client

9

Secure Submission Model

Secure submit protocol PKI authentication Authorization and mapping

Based on Grid ID

Further authorization by scheduler Based on local user ID

Secure control/cancel Also PKI authenticated Owner has rights to his jobs and not others’

10

Secure Execution Model

After authorization… Execute job securely

User account “sandboxing” of processes According to mapping policy and request details

Initialization of sandbox credentials Client-delegated credentials Adapter scripts can be customized for site needs

AFS, Kerberos, etc

Multiple levels of audit possible Container Sudo Local scheduler

11

Secure Staging Model

Before and after sandboxed execution… Perform secure file transfers

Create RFT request To local or remote RFT service PKI authentication and delegation In turn, RFT controls GridFTP

Using delegated client credentials

GridFTP PKI authentication Authorization and mapping by local policy files further authorization by FTP/unix perms

12

GRAM WSDLs+

Job Description Schema

(executable, args,env, …)

Users/Applications: Job Brokers, Portals, Command line tools, etc.

Resource Managers: PBS, Condor, LSF, SGE,

Loadleveler, Fork

WS standard interfaces for subscription, notification,destruction

GRAM4

13

GRAM4 Approach

GridFTPRFT

Delegation

GridFTP

GRAMservices

local sched.

user job

compute element

compute element and service host(s)

remote storage element(s)

FTP data

FTP control

clie

nt

job submit

delegate

xfer

req

uest

local job control

delegateGRAMadaptersu

do

14

Other Approach Highlights

Scalability improvements(discussed next)

sudo/auth_and_exec to limit damage risk from software failures

to improve audit capabilities Extensibility

Retain: scheduler adapter structure To extend for new platforms

Improved: authorization callouts To better integrate with site practices

15

Usage Scenarios: the Ideal

“GRAM should add little to no overhead compared to an underlying batch system” Submit as many jobs to GRAM as is possible to the underlying scheduler

Goal - 10,000 jobs to a batch scheduler Goal – efficiently fill the process table for fork scheduler

Submit/process jobs as fast to GRAM as is possible to the underlying scheduler

Goal - 1 per second

16

Usage Scenarios: the Attempt

Efforts and features towards the goal Allow job brokers the freedom to optimize

E.g. Condor-G is smarter than globusrun-ws Protocol steps made optional and shareable

Reduced cost for GRAM service on host Single WSRF host environment Better job status monitoring mechanisms

More scalable/reliable file handling GridFTP and RFT instead of globus-url-copy Removal of non-scalable GASS caching

17

Production Quality Service performance

Throughput Number of jobs (/bin/date) GRAM can process per

minute 100

Max concurrency Total jobs a GRAM service can manage at one time

without failure 32,000

Job burst Many simultaneous job submissions Are the error conditions acceptable? Job should be rejected, before overloading the

service container or service host

18

Production Quality

Service Stability & Recovery Service uptime

Under a moderate load, how long can the GRAM service process jobs without failure / reboot?

Job recovery After reboot, processing/monitoring resumes

for submitted jobs Clients resume control of jobs

19

Reasonable Applications Today

High throughput job sets: two approaches

1. Use GRAM for every application task Jobs durations > 1 minute

2. Use GRAM for starting user/VO services Course-grain jobs handle task/transaction flow As in Condor glide-ins

MPICH-G4 (MPIG) Large-scale multi-site/grid MPI jobs Co-allocation but no co-reservation yet Estimated release - Q4 2006

20

GRAM: Part 2

How to use client software…

21

How to use Client Software

Command line programs WSDL interface

22

Command Line Programs globusrun-ws

Submit and monitor gram jobs grid-proxy-init

Creates client-side user proxy wsrf-query

Query a services resource properties globus-url-copy

Transfer files to remote hosts globus-credential-delegate globus-credential-refresh

Credential management to remote hosts

23

globusrun-ws

Written in C (C WS Core) Faster startup and execution

Supports GRAM multi-jobs or single jobs Submission, monitoring, cancellation

Credential management Automatic or user-supplied delegation

Streaming of job stdout/err during execution Advanced use of GridFTP client library

24

Simple Job: Step 1

Create a user proxy Your temporary grid credential

Command Example:% grid-proxy-initYour identity:/DC=org/DC=doegrids/OU=People/CN=Stuart Martin 564728Enter GRID pass phrase for this identity:Creating proxy......................... DoneYour proxy is valid until: Fri Jan 7 21:35:31 2005

25

Simple Job: Step 2 Submit job to a GRAM service

default factory EPR generate job RSL to default localhost

Command example:% globusrun-ws -submit -c /bin/touch touched_itSubmitting job...Done.Job ID: uuid:002a6ab8-6036-11d9-bae6-0002a5ad41e5Termination time: 01/07/2005 22:55 GMTCurrent job state: ActiveCurrent job state: CleanUpCurrent job state: DoneDestroying job...Done.

26

Complete Factory Contact

Override default EPR Select a different host/service Use “contact” shorthand for convenience

Relies on proprietary knowledge of EPR format!

Command example:

% globusrun-ws -submit –F \https://140.221.65.193:4444/wsrf/services\/ManagedJobFactoryService \-c /bin/touch touched_it

27

Read RSL from File

Command:

% globusrun-ws -submit -f touch.xml

Contents of touch.xml file:

<job> <executable>/bin/touch</executable> <argument>touched_it</argument></job>

28

Batch Job Submissions

% globusrun-ws -submit -batch -o job_epr -c /bin/sleep 50Submitting job...Done.Job ID: uuid:f9544174-60c5-11d9-97e3-0002a5ad41e5Termination time: 01/08/2005 16:05 GMT

% globusrun-ws -monitor -j job_eprjob state: ActiveCurrent job state: CleanUpCurrent job state: DoneRequesting original job description...Done.Destroying job...Done.

29

Batch Job Submissions

% globusrun-ws -submit -batch -o job_epr -c /bin/sleep 50Submitting job...Done.Job ID: uuid:f9544174-60c5-11d9-97e3-0002a5ad41e5Termination time: 01/08/2005 16:05 GMT

% globusrun-ws -status -j job_eprCurrent job state: Active

% globusrun-ws -status -j job_eprCurrent job state: Done

% globusrun-ws -kill -j job_eprRequesting original job description...Done.Destroying job...Done.

30

Common/useful options

globusrun-ws -J Perform delegation as necessary for job

globusrun-ws -S Perform delegation as necessary for job’s file staging

globusrun-ws -s Stream stdout/err during job execution to the terminal

globusrun-ws -self Useful for testing, when you have started the service using your credentials instead of host credentials

31

Staging job<job><executable>/bin/echo</executable><directory>/tmp</directory><argument>Hello</argument><stdout>job.out</stdout><stderr>job.err</stderr><fileStageOut> <transfer> <sourceUrl>file:///tmp/job.out</sourceUrl> <destinationUrl> gsiftp://host.domain:2811/tmp/stage.out </destinationUrl> </transfer></fileStageOut>

</job>

32

RFT Options<fileStageOut>

<transfer> <sourceUrl>file:///tmp/job.out</sourceUrl> <destinationUrl> gsiftp://host.domain:2811/tmp/stage.out </destinationUrl>

<rftOptions>

<subjectName> /DC=org/DC=doegrids/OU=People/CN=Stuart Martin 564728

</subjectName>

<parallelStreams>4</parallelStreams>

</rftOptions>

</transfer>

</fileStageOut>

33

RSL Variable

Enables late binding of values Values resolved by GRAM service

System-specific variables ${GLOBUS_USER_HOME} ${GLOBUS_LOCATION} ${GLOBUS_SCRATCH_DIR}

Alternative directory that is shared with compute node

Typically providing more space than user’s HOME dir

34

RSL Variable Example<job><executable>/bin/echo</executable><argument>HOME is ${GLOBUS_USER_HOME}</argument><argument>SCRATCH = ${GLOBUS_SCRATCH_DIR}</argument><argument>GL is ${GLOBUS_LOCATION}</argument><stdout>${GLOBUS_USER_HOME}/echo.stdout</stdout><stderr>${GLOBUS_USER_HOME}/echo.stderr</stderr>

</job>

35

RSL Extensions Support

4.0.3 does not support extension by default Update packages are available to add extension support

http://www.globus.org/toolkit/downloads/development/

globus_gram_job_manager-7.14 plus dependencies

All 4.1.x releases support extensions by default

36

RSL Extensions Example<job><executable>/bin/echo</executable><extensions>

<email_address>joeshmo@gmail.com</email_address><extensions>

</job> Simple string extension elements are converted

into single-element arrays Code example in pbs.pm:

if($description->email_address() ne ''){ print JOB '#PBS -M ', \ $description->email_address(), "\n";}

37

How to use Client Software

Command line programs WSDL interface

38

ManagedJobFactory portType

createManagedJob operation Creates either an MMJR or MEJR Input:

Initial Termination Time Job ID

UUID of the job resource, for job reliability/recoverability Subscribe Request

Client can include a request to subscribe for job state notifications with the job submission to avoid an extra operation call

Job Description / RSL Either a single or multi-job description

Output: newTerminationTime - new termination time of the job

resource managedJobEndpoint - EPR of the newly created job resource subscriptionEndpoint - EPR of the notification subscription

39

ManagedJob portType

Base port type for the MEJS and MMJS release operation

Release a holdState set in the job description Only one hold state can be set/released

Input: None Output: None

State change notifications State - job state (Active, Pending, Done, Cleanup…) Fault - fault causing a Failed state (if applicable) Exit Code - exit code of the job process Holding - boolean indicating if the job is in a hold state

40

ManagedJob portType

On destroy, or soft state termination…

The MJS will cleanup everything

1. Stop any outstanding tasks Cancel/terminate the execution Destroy RFT stage in, out requests

2. Process CleanUp state Submit request to RFT to remove

files/directories RSL attribute fileCleanUp

Remove job user proxy file

3. Destroy job resource

41

ManagedExecutableJobService

Executes the requested job process(es) specified in the RSL

Resource Properties (ManagedExecutableJobPortType)

serviceLevelAgreement - the RSL / Job Description state - the current job state faults - the fault causing a Failed state localUserId - the username of the resource owner userSubject - the GSI subject of the resource

owner holding - boolean indiciating the job is holding stdoutURL - the GridFTP URL to the stdout file stderrURL - the GridFTP URL to the stderr file credentialPath - the local path to the user proxy file exitCode - the exit code of the job proces (if

applicable)

42

ManagedMultiJobService

Processes a multi-job RSL submits the sub-jobs to the specified ManagedJobFactoryService.

Sub-jobs cannot be multi-jobs themselves. Resource Properties (ManagedMultiJobPortType)

serviceLevelAgreement - the multi-job RSL / Job Description state - the current overall state faults - the fault causing a Failed state localUserId - the username of the resource owner userSubject - the GSI subject of the resource owner holding - boolean indiciating all jobs are holding subJobEndpoint - list of endpoints to the sub-jobs

43

Our Goals Highly functional interface

grid service WSDLs C API Java API

Expressive job description language Basic command line clients

Should be useable from shell scripts Collaborate with others to create more

capable and complete clients E.g. Condor-G, TG’s Science Gateways, Portals

44

GRAM: Part 3

How to administer servers…

45

4.0 Quickstart Guide

Consult this guide first for basic GT setup Setting up first machine Setting up second machine Setting up a compute cluster - PBS www.globus.org/toolkit/docs/4.0/admin/docbook/quickstart.html

Then consult GRAM admin guide for additional details www.globus.org/toolkit/docs/4.0/admin/docbook/ch11.html

46

Typical GRAM service setup Host credentials

For client/service authentication For client authorization of the service Existing GT2/GT3 host certs can be used

Gridmap file Entries for each user allowed to execute job’s

Maps the grid ID to a local user account Same syntax as GT2, GT3 gridmap files

Installed sudo Method for GRAM to runs commands in the user’s account

47

sudo configuration sudo policies

Done by hand by rootRunas_Alias GRAMUSERS = ! root, ! wheel, …

globus ALL=(GRAMUSERS) NOPASSWD: /sandbox/globus/install/libexec/globus-gridmap-and-execute /sandbox/globus/install/libexec/globus-job-manager-script.pl *

globus ALL=(GRAMUSERS) NOPASSWD: /sandbox/globus/install/libexec/globus-gridmap-and-execute /sandbox/globus/install/libexec/globus-gram-local-proxy-tool *

globus-gridmap-and-execute Redundant if sudo is locked down tightly Enforce that GRAM only targets accounts in gridmap

So sudo policy need not enumerate all GRAM users at large/dynamic sites In fact, you can audit this tool and change GRAMUSERS to ALL if you like…

Replace this with your own authz tool (callout)

48

Local Resource Manager Adapters

GT provides/supports 4 RM adapters PBS, LSF, Condor, Fork

3rd party RM adapters exist SGE, LoadLeveler, GridWay Tell us about yours and we’ll add to GT web pages!

All 4 RM adapters are included in all binary and source installers

Only Fork is configured automatically Configuring an RM adapter

Add configure arguments ./configure --enable-wsgram-pbs …

49

File staging functionality

GridFTP Server Could be run on a separate host from GRAM service container to improve performance / scalability

cpu intensive globus_gram_fs_map_config.xml

Config the GridFTP server(s) to use for local file staging

RFT Requires PostgreSQL DB setup Usability: 4.1.x Defaults to embedded DB (Derby)

50

GRAM / GridFTP file system mapping

Associates compute resources and GridFTP servers Maps shared filesystems of the gram and gridftp hosts,

e.g. Gram host mounts homes at /pvfs/home gridftp host mounts same at /pvfs/users/home

GRAM resolves file:/// staging paths to local GridFTP URLs File:///pvfs/home/smartin/file1... resolves to: gsiftp://host.domain:2811/pvfs/users/home/smartin/file1

$GL/etc/gram-service/globus_gram_fs_map_config.xml Client will need to know mappings to stage files

separately from WS GRAM

51

Non-default Setup ./setup-gram-service-common

To change GRAM configuration Run in $GLOBUS_LOCATION/setup

GridFTP Server config Default is for localhost, port 2811 --gridftp-server=gsiftp://gridftp.host.org:1234

RFT Service config Default is localhost, port 8443 --stage-protocol=https --staging-host=host.domain.org --staging-port=4321

52

Setup: Container Credentials

Default: host credentials /etc/grid-security/containercert.pem /etc/grid-security/containerkey.pem

To configure for a user proxy Update container global security descriptor

Comment out <credential> element $GL/etc/globus_wsrf_core/global_security_descriptor.xml

Tell GRAM the subject to expect for authorization of the RFT service

./setup-gram-service-common --staging-subject= "/DC=org/DC=doegrids/OU=People/CN=Stuart Martin 564720”

Use “-self” argument with globusrun-ws Default GT auth in 4.1.1 will be “host” *or* “self”

53

GRAM: Part 4

Future Plans

54

4.2 Series WS GRAM 4.1.x is dev series for eventual stable

4.2.x stable series 4.1.0 released July 06

RSL extension support globus-job-*-ws scripts included by default Improved service throttling controls Persistence data stored in DB resource manager adapter API Removed unnecessary dependencies to Pre-WS GRAM

4.1.1 (no target date yet) Initial support for JSDL jobs Service auditing to DB

55

WS GRAM Standards Compliance

JSDL Target is 4.1.1 (definitely 4.2.0) Will preserve current interface, so 4.0.x job descriptions will work just fine

Adding new createManagedJobFromJSDLDocument operation

Globusrun-ws will choose appropriate create operation based on job description contents

OGSA-BES Target is 4.4 (spec is not finished, so 4.2 is unlikely)

Will preserve 4.0.x interface as well

56

Service Auditing

Follow along on bugzilla “roadmap” item http://bugzilla.globus.org/bugzilla/show_bug.cgi?

id=4409 Add yourself to cc list

Prototype written and deployed on TeraGrid In evaluation phase provides the capability for a TG grid user to get TG

usage info using a grid job id (from GRAM) Audit DB entries provide join between grid job id and

local TG accounting DB Will be included in 4.1.x series to be included in

4.2 Probably disable by default in GT releases

57

Advanced Reservation

Investigation is underway No firm plans yet, but high on our priority

list Follow along on bugzilla “roadmap” item

http://bugzilla.globus.org/bugzilla/show_bug.cgi?id=4045

58

Performance testing with OSG

Test scenario submit large (3500) job run through condor-g to WS GRAM

to LRM condor Job is create unique job dir; 2MB stageIn, 2MB

stageOut, cleanup job dir Solved reliability issue with default condor-g jobs

Included in 4.0.3 Found/fixed bugs in RFT which effected performance

by appox 250% for staging jobs From 5.2 jpm to 13 jpm Patches to 4.0.3 will be made available soon

We plan on writing up results and provide config recommendation for GT container and condor-g

59

WS GRAM Usage Statistics July 6 thru Aug 6th 2006

651517 jobs submitted 25 unique domains (e.g. .edu, .org, .gov) 356 unique IPs (Container installations with WS GRAM)

60

Documentation 4.0.x GRAM documentation

Guides: admin, user, developer, overview, public interface

http://www.globus.org/toolkit/docs/4.0/execution/wsgram/

4.1.x GRAM documentation http://www.globus.org/toolkit/docs/4.1/execution/wsgram/

Main 4.0.x documentation http://www.globus.org/toolkit/docs/4.0/ Download, release notes, links to all GT projects/ components

61

Writing New RM Adapters

http://www.globus.org/toolkit/docs/4.0/execution/wsgram/developer/scheduler-tutorial.html Scheduler perl modules (e.g. pbs.pm)

Submitting jobs, canceling jobs, setup and packaging

Scheduler Event Generator (SEG) Monitoring events from the scheduler for all job for all users; it runs under a privileged account

62

Bugzilla

If you’ve found a bug (not a question!) http://bugzilla.globus.org/ GRAM product, wsrf* components

63

Globus Development

GlobDev - Open development Globus governance model based on Apache

Developers (committers) control direction of software components (projects)

http://dev.globus.org GRAM project

http://dev.globus.org/wiki/GRAM Email lists: gram-user, gram-dev, gram-announce, gram-commit

GT project gt-user, gt-dev

64

Thanks to the GRAM developers!

Peter Lane - ANL Joe Bester - ANL Ravi Madduri - ANL Martin Feller - UofC Plus the entire GT dev team

65

Meet the Developers Session at Globus Alliance Booth (152A-

P7) September 12

8:00am - 9:00am "Java WS Core and Security (C, Java)"  -- Olle Mulmo, Jarek Gawor, Rachana Anantakrishnan

11:30am -12:30pm "RLS" -- Rob Schuler, Ann Chervenak12:30pm -1:30pm "MDS" -- Mike D'arcy, Laura Pearlman3:00pm - 4:00pm ”Resource Management (GRAM, Virtual Workspaces and Dynamic Accounts)" – Stu Martin, Peter Lane, Tim Freeman, Kate Keahey6:00pm - 7:00pm "C WS Core" -- Joe Bester7:00pm - 8:00pm "Python WS Core" -- Joshua Boverhof

September 13

8:00am - 9:00am "GridShib" -- Von Welch, Ton Scavo, Tim Freeman

11:30am - 12:30pm "GT Installation and Administration" -- Charles Bacon12:30pm - 1:30pm "MyProxy" -- Jim Basney3:00pm - 4:00pm "GridFTP, XIO, RFT" -- John Bresnahan, Ravi Madduri

66

COME CELEBRATE WITH US!

In appreciation of your support of all things Globus over the past decade, you are cordially invited to the Globus 10th Birthday Party.

When: Monday, September 11, 2006 - 7:00pm, immediately following Ian Foster’s Globus State of the Union Keynote.

Where: The convention center concourse, in the center of the GlobusWORLD / GridWorld conference activity.

What: Food, drinks, music, friends and lots of fun!