Globus GRAM for Developers Stuart Martin, Peter Lane Argonne National Lab.
-
Upload
cody-mcintosh -
Category
Documents
-
view
217 -
download
0
Transcript of Globus GRAM for Developers Stuart Martin, Peter Lane Argonne National Lab.
Globus GRAM for Developers
Stuart Martin, Peter Lane
Argonne National Lab
2
Session OverviewQ: What is this session about?
A:This presentation will cover the features, interface, architecture, performance, and future plans of the Globus Toolkit v4 Web Services Grid Resource Allocation and Management (GRAM4) component.
Four-part discussion (~ 20 mins/each) Overview of GRAM Model How to use client software How to administer servers Future plans
3
GRAM: Part 1
Overview of GRAM Model…
4
What is GRAM? GRAM is a Globus Toolkit component
For Grid job management
GRAM is a unifying remote interface to Resource Managers Yet preserves local site security/control
GRAM is for stateful job control Reliable operation Asynchronous monitoring and control Remote credential management File staging via RFT and GridFTP
5
Grid Job Management Goals
Provide a service to securely: Create an environment for a job Stage files to/from environment Cause execution of job process(es)
Via various local resource managers Monitor execution Signal important state changes to client
Enable client access to output files Streaming access during execution
6
Job Submission Model
Create and manage one job on a resource Submit and wait Not with an interactive TTY
File based stdin/out/err Supported by all batch schedulers
More complex than RPC Optional steps before and after submission message
Job has complex lifecycle Staging, execution, and cleanup states But not as general as Condor DAG, etc.
Asynchronous monitoring
7
Job Submission Options Optional file staging
Transfer files “in” before job execution Transfer files “out” after job execution
Optional file streaming Monitor files during job execution
Optional credential delegation Create, refresh, and terminate delegations For use by job process For use by GRAM to do optional file staging
8
Job Submission Monitoring
Monitor job lifecycle GRAM and scheduler states for job
StageIn, Pending, Active, Suspended, StageOut, Cleanup, Done, Failed
Job execution status Return codes
Multiple monitoring methods Simple query for current state Asynchronous notifications to client
9
Secure Submission Model
Secure submit protocol PKI authentication Authorization and mapping
Based on Grid ID
Further authorization by scheduler Based on local user ID
Secure control/cancel Also PKI authenticated Owner has rights to his jobs and not others’
10
Secure Execution Model
After authorization… Execute job securely
User account “sandboxing” of processes According to mapping policy and request details
Initialization of sandbox credentials Client-delegated credentials Adapter scripts can be customized for site needs
AFS, Kerberos, etc
Multiple levels of audit possible Container Sudo Local scheduler
11
Secure Staging Model
Before and after sandboxed execution… Perform secure file transfers
Create RFT request To local or remote RFT service PKI authentication and delegation In turn, RFT controls GridFTP
Using delegated client credentials
GridFTP PKI authentication Authorization and mapping by local policy files further authorization by FTP/unix perms
12
GRAM WSDLs+
Job Description Schema
(executable, args,env, …)
Users/Applications: Job Brokers, Portals, Command line tools, etc.
Resource Managers: PBS, Condor, LSF, SGE,
Loadleveler, Fork
WS standard interfaces for subscription, notification,destruction
GRAM4
13
GRAM4 Approach
GridFTPRFT
Delegation
GridFTP
GRAMservices
local sched.
user job
compute element
compute element and service host(s)
remote storage element(s)
FTP data
FTP control
clie
nt
job submit
delegate
xfer
req
uest
local job control
delegateGRAMadaptersu
do
14
Other Approach Highlights
Scalability improvements(discussed next)
sudo/auth_and_exec to limit damage risk from software failures
to improve audit capabilities Extensibility
Retain: scheduler adapter structure To extend for new platforms
Improved: authorization callouts To better integrate with site practices
15
Usage Scenarios: the Ideal
“GRAM should add little to no overhead compared to an underlying batch system” Submit as many jobs to GRAM as is possible to the underlying scheduler
Goal - 10,000 jobs to a batch scheduler Goal – efficiently fill the process table for fork scheduler
Submit/process jobs as fast to GRAM as is possible to the underlying scheduler
Goal - 1 per second
16
Usage Scenarios: the Attempt
Efforts and features towards the goal Allow job brokers the freedom to optimize
E.g. Condor-G is smarter than globusrun-ws Protocol steps made optional and shareable
Reduced cost for GRAM service on host Single WSRF host environment Better job status monitoring mechanisms
More scalable/reliable file handling GridFTP and RFT instead of globus-url-copy Removal of non-scalable GASS caching
17
Production Quality Service performance
Throughput Number of jobs (/bin/date) GRAM can process per
minute 100
Max concurrency Total jobs a GRAM service can manage at one time
without failure 32,000
Job burst Many simultaneous job submissions Are the error conditions acceptable? Job should be rejected, before overloading the
service container or service host
18
Production Quality
Service Stability & Recovery Service uptime
Under a moderate load, how long can the GRAM service process jobs without failure / reboot?
Job recovery After reboot, processing/monitoring resumes
for submitted jobs Clients resume control of jobs
19
Reasonable Applications Today
High throughput job sets: two approaches
1. Use GRAM for every application task Jobs durations > 1 minute
2. Use GRAM for starting user/VO services Course-grain jobs handle task/transaction flow As in Condor glide-ins
MPICH-G4 (MPIG) Large-scale multi-site/grid MPI jobs Co-allocation but no co-reservation yet Estimated release - Q4 2006
20
GRAM: Part 2
How to use client software…
21
How to use Client Software
Command line programs WSDL interface
22
Command Line Programs globusrun-ws
Submit and monitor gram jobs grid-proxy-init
Creates client-side user proxy wsrf-query
Query a services resource properties globus-url-copy
Transfer files to remote hosts globus-credential-delegate globus-credential-refresh
Credential management to remote hosts
23
globusrun-ws
Written in C (C WS Core) Faster startup and execution
Supports GRAM multi-jobs or single jobs Submission, monitoring, cancellation
Credential management Automatic or user-supplied delegation
Streaming of job stdout/err during execution Advanced use of GridFTP client library
24
Simple Job: Step 1
Create a user proxy Your temporary grid credential
Command Example:% grid-proxy-initYour identity:/DC=org/DC=doegrids/OU=People/CN=Stuart Martin 564728Enter GRID pass phrase for this identity:Creating proxy......................... DoneYour proxy is valid until: Fri Jan 7 21:35:31 2005
25
Simple Job: Step 2 Submit job to a GRAM service
default factory EPR generate job RSL to default localhost
Command example:% globusrun-ws -submit -c /bin/touch touched_itSubmitting job...Done.Job ID: uuid:002a6ab8-6036-11d9-bae6-0002a5ad41e5Termination time: 01/07/2005 22:55 GMTCurrent job state: ActiveCurrent job state: CleanUpCurrent job state: DoneDestroying job...Done.
26
Complete Factory Contact
Override default EPR Select a different host/service Use “contact” shorthand for convenience
Relies on proprietary knowledge of EPR format!
Command example:
% globusrun-ws -submit –F \https://140.221.65.193:4444/wsrf/services\/ManagedJobFactoryService \-c /bin/touch touched_it
27
Read RSL from File
Command:
% globusrun-ws -submit -f touch.xml
Contents of touch.xml file:
<job> <executable>/bin/touch</executable> <argument>touched_it</argument></job>
28
Batch Job Submissions
% globusrun-ws -submit -batch -o job_epr -c /bin/sleep 50Submitting job...Done.Job ID: uuid:f9544174-60c5-11d9-97e3-0002a5ad41e5Termination time: 01/08/2005 16:05 GMT
% globusrun-ws -monitor -j job_eprjob state: ActiveCurrent job state: CleanUpCurrent job state: DoneRequesting original job description...Done.Destroying job...Done.
29
Batch Job Submissions
% globusrun-ws -submit -batch -o job_epr -c /bin/sleep 50Submitting job...Done.Job ID: uuid:f9544174-60c5-11d9-97e3-0002a5ad41e5Termination time: 01/08/2005 16:05 GMT
% globusrun-ws -status -j job_eprCurrent job state: Active
% globusrun-ws -status -j job_eprCurrent job state: Done
% globusrun-ws -kill -j job_eprRequesting original job description...Done.Destroying job...Done.
30
Common/useful options
globusrun-ws -J Perform delegation as necessary for job
globusrun-ws -S Perform delegation as necessary for job’s file staging
globusrun-ws -s Stream stdout/err during job execution to the terminal
globusrun-ws -self Useful for testing, when you have started the service using your credentials instead of host credentials
31
Staging job<job><executable>/bin/echo</executable><directory>/tmp</directory><argument>Hello</argument><stdout>job.out</stdout><stderr>job.err</stderr><fileStageOut> <transfer> <sourceUrl>file:///tmp/job.out</sourceUrl> <destinationUrl> gsiftp://host.domain:2811/tmp/stage.out </destinationUrl> </transfer></fileStageOut>
</job>
32
RFT Options<fileStageOut>
<transfer> <sourceUrl>file:///tmp/job.out</sourceUrl> <destinationUrl> gsiftp://host.domain:2811/tmp/stage.out </destinationUrl>
<rftOptions>
<subjectName> /DC=org/DC=doegrids/OU=People/CN=Stuart Martin 564728
</subjectName>
<parallelStreams>4</parallelStreams>
</rftOptions>
</transfer>
</fileStageOut>
33
RSL Variable
Enables late binding of values Values resolved by GRAM service
System-specific variables ${GLOBUS_USER_HOME} ${GLOBUS_LOCATION} ${GLOBUS_SCRATCH_DIR}
Alternative directory that is shared with compute node
Typically providing more space than user’s HOME dir
34
RSL Variable Example<job><executable>/bin/echo</executable><argument>HOME is ${GLOBUS_USER_HOME}</argument><argument>SCRATCH = ${GLOBUS_SCRATCH_DIR}</argument><argument>GL is ${GLOBUS_LOCATION}</argument><stdout>${GLOBUS_USER_HOME}/echo.stdout</stdout><stderr>${GLOBUS_USER_HOME}/echo.stderr</stderr>
</job>
35
RSL Extensions Support
4.0.3 does not support extension by default Update packages are available to add extension support
http://www.globus.org/toolkit/downloads/development/
globus_gram_job_manager-7.14 plus dependencies
All 4.1.x releases support extensions by default
36
RSL Extensions Example<job><executable>/bin/echo</executable><extensions>
<email_address>[email protected]</email_address><extensions>
</job> Simple string extension elements are converted
into single-element arrays Code example in pbs.pm:
if($description->email_address() ne ''){ print JOB '#PBS -M ', \ $description->email_address(), "\n";}
37
How to use Client Software
Command line programs WSDL interface
38
ManagedJobFactory portType
createManagedJob operation Creates either an MMJR or MEJR Input:
Initial Termination Time Job ID
UUID of the job resource, for job reliability/recoverability Subscribe Request
Client can include a request to subscribe for job state notifications with the job submission to avoid an extra operation call
Job Description / RSL Either a single or multi-job description
Output: newTerminationTime - new termination time of the job
resource managedJobEndpoint - EPR of the newly created job resource subscriptionEndpoint - EPR of the notification subscription
39
ManagedJob portType
Base port type for the MEJS and MMJS release operation
Release a holdState set in the job description Only one hold state can be set/released
Input: None Output: None
State change notifications State - job state (Active, Pending, Done, Cleanup…) Fault - fault causing a Failed state (if applicable) Exit Code - exit code of the job process Holding - boolean indicating if the job is in a hold state
40
ManagedJob portType
On destroy, or soft state termination…
The MJS will cleanup everything
1. Stop any outstanding tasks Cancel/terminate the execution Destroy RFT stage in, out requests
2. Process CleanUp state Submit request to RFT to remove
files/directories RSL attribute fileCleanUp
Remove job user proxy file
3. Destroy job resource
41
ManagedExecutableJobService
Executes the requested job process(es) specified in the RSL
Resource Properties (ManagedExecutableJobPortType)
serviceLevelAgreement - the RSL / Job Description state - the current job state faults - the fault causing a Failed state localUserId - the username of the resource owner userSubject - the GSI subject of the resource
owner holding - boolean indiciating the job is holding stdoutURL - the GridFTP URL to the stdout file stderrURL - the GridFTP URL to the stderr file credentialPath - the local path to the user proxy file exitCode - the exit code of the job proces (if
applicable)
42
ManagedMultiJobService
Processes a multi-job RSL submits the sub-jobs to the specified ManagedJobFactoryService.
Sub-jobs cannot be multi-jobs themselves. Resource Properties (ManagedMultiJobPortType)
serviceLevelAgreement - the multi-job RSL / Job Description state - the current overall state faults - the fault causing a Failed state localUserId - the username of the resource owner userSubject - the GSI subject of the resource owner holding - boolean indiciating all jobs are holding subJobEndpoint - list of endpoints to the sub-jobs
43
Our Goals Highly functional interface
grid service WSDLs C API Java API
Expressive job description language Basic command line clients
Should be useable from shell scripts Collaborate with others to create more
capable and complete clients E.g. Condor-G, TG’s Science Gateways, Portals
44
GRAM: Part 3
How to administer servers…
45
4.0 Quickstart Guide
Consult this guide first for basic GT setup Setting up first machine Setting up second machine Setting up a compute cluster - PBS www.globus.org/toolkit/docs/4.0/admin/docbook/quickstart.html
Then consult GRAM admin guide for additional details www.globus.org/toolkit/docs/4.0/admin/docbook/ch11.html
46
Typical GRAM service setup Host credentials
For client/service authentication For client authorization of the service Existing GT2/GT3 host certs can be used
Gridmap file Entries for each user allowed to execute job’s
Maps the grid ID to a local user account Same syntax as GT2, GT3 gridmap files
Installed sudo Method for GRAM to runs commands in the user’s account
47
sudo configuration sudo policies
Done by hand by rootRunas_Alias GRAMUSERS = ! root, ! wheel, …
globus ALL=(GRAMUSERS) NOPASSWD: /sandbox/globus/install/libexec/globus-gridmap-and-execute /sandbox/globus/install/libexec/globus-job-manager-script.pl *
globus ALL=(GRAMUSERS) NOPASSWD: /sandbox/globus/install/libexec/globus-gridmap-and-execute /sandbox/globus/install/libexec/globus-gram-local-proxy-tool *
globus-gridmap-and-execute Redundant if sudo is locked down tightly Enforce that GRAM only targets accounts in gridmap
So sudo policy need not enumerate all GRAM users at large/dynamic sites In fact, you can audit this tool and change GRAMUSERS to ALL if you like…
Replace this with your own authz tool (callout)
48
Local Resource Manager Adapters
GT provides/supports 4 RM adapters PBS, LSF, Condor, Fork
3rd party RM adapters exist SGE, LoadLeveler, GridWay Tell us about yours and we’ll add to GT web pages!
All 4 RM adapters are included in all binary and source installers
Only Fork is configured automatically Configuring an RM adapter
Add configure arguments ./configure --enable-wsgram-pbs …
49
File staging functionality
GridFTP Server Could be run on a separate host from GRAM service container to improve performance / scalability
cpu intensive globus_gram_fs_map_config.xml
Config the GridFTP server(s) to use for local file staging
RFT Requires PostgreSQL DB setup Usability: 4.1.x Defaults to embedded DB (Derby)
50
GRAM / GridFTP file system mapping
Associates compute resources and GridFTP servers Maps shared filesystems of the gram and gridftp hosts,
e.g. Gram host mounts homes at /pvfs/home gridftp host mounts same at /pvfs/users/home
GRAM resolves file:/// staging paths to local GridFTP URLs File:///pvfs/home/smartin/file1... resolves to: gsiftp://host.domain:2811/pvfs/users/home/smartin/file1
$GL/etc/gram-service/globus_gram_fs_map_config.xml Client will need to know mappings to stage files
separately from WS GRAM
51
Non-default Setup ./setup-gram-service-common
To change GRAM configuration Run in $GLOBUS_LOCATION/setup
GridFTP Server config Default is for localhost, port 2811 --gridftp-server=gsiftp://gridftp.host.org:1234
RFT Service config Default is localhost, port 8443 --stage-protocol=https --staging-host=host.domain.org --staging-port=4321
52
Setup: Container Credentials
Default: host credentials /etc/grid-security/containercert.pem /etc/grid-security/containerkey.pem
To configure for a user proxy Update container global security descriptor
Comment out <credential> element $GL/etc/globus_wsrf_core/global_security_descriptor.xml
Tell GRAM the subject to expect for authorization of the RFT service
./setup-gram-service-common --staging-subject= "/DC=org/DC=doegrids/OU=People/CN=Stuart Martin 564720”
Use “-self” argument with globusrun-ws Default GT auth in 4.1.1 will be “host” *or* “self”
53
GRAM: Part 4
Future Plans
54
4.2 Series WS GRAM 4.1.x is dev series for eventual stable
4.2.x stable series 4.1.0 released July 06
RSL extension support globus-job-*-ws scripts included by default Improved service throttling controls Persistence data stored in DB resource manager adapter API Removed unnecessary dependencies to Pre-WS GRAM
4.1.1 (no target date yet) Initial support for JSDL jobs Service auditing to DB
55
WS GRAM Standards Compliance
JSDL Target is 4.1.1 (definitely 4.2.0) Will preserve current interface, so 4.0.x job descriptions will work just fine
Adding new createManagedJobFromJSDLDocument operation
Globusrun-ws will choose appropriate create operation based on job description contents
OGSA-BES Target is 4.4 (spec is not finished, so 4.2 is unlikely)
Will preserve 4.0.x interface as well
56
Service Auditing
Follow along on bugzilla “roadmap” item http://bugzilla.globus.org/bugzilla/show_bug.cgi?
id=4409 Add yourself to cc list
Prototype written and deployed on TeraGrid In evaluation phase provides the capability for a TG grid user to get TG
usage info using a grid job id (from GRAM) Audit DB entries provide join between grid job id and
local TG accounting DB Will be included in 4.1.x series to be included in
4.2 Probably disable by default in GT releases
57
Advanced Reservation
Investigation is underway No firm plans yet, but high on our priority
list Follow along on bugzilla “roadmap” item
http://bugzilla.globus.org/bugzilla/show_bug.cgi?id=4045
58
Performance testing with OSG
Test scenario submit large (3500) job run through condor-g to WS GRAM
to LRM condor Job is create unique job dir; 2MB stageIn, 2MB
stageOut, cleanup job dir Solved reliability issue with default condor-g jobs
Included in 4.0.3 Found/fixed bugs in RFT which effected performance
by appox 250% for staging jobs From 5.2 jpm to 13 jpm Patches to 4.0.3 will be made available soon
We plan on writing up results and provide config recommendation for GT container and condor-g
59
WS GRAM Usage Statistics July 6 thru Aug 6th 2006
651517 jobs submitted 25 unique domains (e.g. .edu, .org, .gov) 356 unique IPs (Container installations with WS GRAM)
60
Documentation 4.0.x GRAM documentation
Guides: admin, user, developer, overview, public interface
http://www.globus.org/toolkit/docs/4.0/execution/wsgram/
4.1.x GRAM documentation http://www.globus.org/toolkit/docs/4.1/execution/wsgram/
Main 4.0.x documentation http://www.globus.org/toolkit/docs/4.0/ Download, release notes, links to all GT projects/ components
61
Writing New RM Adapters
http://www.globus.org/toolkit/docs/4.0/execution/wsgram/developer/scheduler-tutorial.html Scheduler perl modules (e.g. pbs.pm)
Submitting jobs, canceling jobs, setup and packaging
Scheduler Event Generator (SEG) Monitoring events from the scheduler for all job for all users; it runs under a privileged account
62
Bugzilla
If you’ve found a bug (not a question!) http://bugzilla.globus.org/ GRAM product, wsrf* components
63
Globus Development
GlobDev - Open development Globus governance model based on Apache
Developers (committers) control direction of software components (projects)
http://dev.globus.org GRAM project
http://dev.globus.org/wiki/GRAM Email lists: gram-user, gram-dev, gram-announce, gram-commit
GT project gt-user, gt-dev
64
Thanks to the GRAM developers!
Peter Lane - ANL Joe Bester - ANL Ravi Madduri - ANL Martin Feller - UofC Plus the entire GT dev team
65
Meet the Developers Session at Globus Alliance Booth (152A-
P7) September 12
8:00am - 9:00am "Java WS Core and Security (C, Java)" -- Olle Mulmo, Jarek Gawor, Rachana Anantakrishnan
11:30am -12:30pm "RLS" -- Rob Schuler, Ann Chervenak12:30pm -1:30pm "MDS" -- Mike D'arcy, Laura Pearlman3:00pm - 4:00pm ”Resource Management (GRAM, Virtual Workspaces and Dynamic Accounts)" – Stu Martin, Peter Lane, Tim Freeman, Kate Keahey6:00pm - 7:00pm "C WS Core" -- Joe Bester7:00pm - 8:00pm "Python WS Core" -- Joshua Boverhof
September 13
8:00am - 9:00am "GridShib" -- Von Welch, Ton Scavo, Tim Freeman
11:30am - 12:30pm "GT Installation and Administration" -- Charles Bacon12:30pm - 1:30pm "MyProxy" -- Jim Basney3:00pm - 4:00pm "GridFTP, XIO, RFT" -- John Bresnahan, Ravi Madduri
66
COME CELEBRATE WITH US!
In appreciation of your support of all things Globus over the past decade, you are cordially invited to the Globus 10th Birthday Party.
When: Monday, September 11, 2006 - 7:00pm, immediately following Ian Foster’s Globus State of the Union Keynote.
Where: The convention center concourse, in the center of the GlobusWORLD / GridWorld conference activity.
What: Food, drinks, music, friends and lots of fun!