E-infrastructure shared between Europe and Latin America Architecture of the WMS Manuel Rubio del...
-
date post
19-Dec-2015 -
Category
Documents
-
view
215 -
download
0
Transcript of E-infrastructure shared between Europe and Latin America Architecture of the WMS Manuel Rubio del...
![Page 1: E-infrastructure shared between Europe and Latin America Architecture of the WMS Manuel Rubio del Solar CETA-CIEMAT EELA Tutorial, Mérida, 07-10.11.2006.](https://reader036.fdocuments.net/reader036/viewer/2022062515/56649d2a5503460f949ff0fc/html5/thumbnails/1.jpg)
E-infrastructure shared between Europe and Latin America
Architecture of the WMS
Manuel Rubio del Solar
CETA-CIEMAT
EELA Tutorial, Mérida, 07-10.11.2006
![Page 2: E-infrastructure shared between Europe and Latin America Architecture of the WMS Manuel Rubio del Solar CETA-CIEMAT EELA Tutorial, Mérida, 07-10.11.2006.](https://reader036.fdocuments.net/reader036/viewer/2022062515/56649d2a5503460f949ff0fc/html5/thumbnails/2.jpg)
EELA Tutorial, Mérida, 14-16.11.2006
E-infrastructure shared between Europe and Latin America
This presentation will cover the following arguments:
Overview of WMS ArchitectureJob Description Language
OverviewWMProxy overview
Outline
= New in gLite 3.0New!
![Page 3: E-infrastructure shared between Europe and Latin America Architecture of the WMS Manuel Rubio del Solar CETA-CIEMAT EELA Tutorial, Mérida, 07-10.11.2006.](https://reader036.fdocuments.net/reader036/viewer/2022062515/56649d2a5503460f949ff0fc/html5/thumbnails/3.jpg)
EELA Tutorial, Mérida, 14-16.11.2006
E-infrastructure shared between Europe and Latin America
First PartArchitecture of the gLite WMS
![Page 4: E-infrastructure shared between Europe and Latin America Architecture of the WMS Manuel Rubio del Solar CETA-CIEMAT EELA Tutorial, Mérida, 07-10.11.2006.](https://reader036.fdocuments.net/reader036/viewer/2022062515/56649d2a5503460f949ff0fc/html5/thumbnails/4.jpg)
EELA Tutorial, Mérida, 14-16.11.2006
E-infrastructure shared between Europe and Latin America
Workload Manager Services
User requestWMS
traslator
![Page 5: E-infrastructure shared between Europe and Latin America Architecture of the WMS Manuel Rubio del Solar CETA-CIEMAT EELA Tutorial, Mérida, 07-10.11.2006.](https://reader036.fdocuments.net/reader036/viewer/2022062515/56649d2a5503460f949ff0fc/html5/thumbnails/5.jpg)
EELA Tutorial, Mérida, 14-16.11.2006
E-infrastructure shared between Europe and Latin America
• The The Workload Management SystemWorkload Management System (WMS) comprises a set of Grid middleware components responsible for distribution and management of tasks across Grid resources.
• The purpose of the Workload Manager (WM) is accept and satisfy requests for job management coming from its clients– meaning of the submission request is to pass the
responsibility of the job to the WM. WM will pass the job to an appropriate CE for execution
• taking into account requirements and the preferences expressed in the job description file
• The decision of which resource should be used is the outcome of a matchmakingmatchmaking process.
WMS Objectives
![Page 6: E-infrastructure shared between Europe and Latin America Architecture of the WMS Manuel Rubio del Solar CETA-CIEMAT EELA Tutorial, Mérida, 07-10.11.2006.](https://reader036.fdocuments.net/reader036/viewer/2022062515/56649d2a5503460f949ff0fc/html5/thumbnails/6.jpg)
EELA Tutorial, Mérida, 14-16.11.2006
E-infrastructure shared between Europe and Latin America
WMS Architecture
Job managementJob managementrequests (submission, requests (submission, cancellation) expressedcancellation) expressed
via a Job Descriptionvia a Job DescriptionLanguage (JDL)Language (JDL)
New! New!
New!
![Page 7: E-infrastructure shared between Europe and Latin America Architecture of the WMS Manuel Rubio del Solar CETA-CIEMAT EELA Tutorial, Mérida, 07-10.11.2006.](https://reader036.fdocuments.net/reader036/viewer/2022062515/56649d2a5503460f949ff0fc/html5/thumbnails/7.jpg)
EELA Tutorial, Mérida, 14-16.11.2006
E-infrastructure shared between Europe and Latin America
WMS Architecture
Job managementJob managementrequests (submission, requests (submission, cancellation) expressedcancellation) expressed
via a Job Descriptionvia a Job DescriptionLanguage (JDL)Language (JDL)
Finds an appropriateFinds an appropriateCE for each submission CE for each submission
request, taking into account request, taking into account job requests and preferences, job requests and preferences, Grid status, utilization policies Grid status, utilization policies
on resources on resources
New! New!
New!
![Page 8: E-infrastructure shared between Europe and Latin America Architecture of the WMS Manuel Rubio del Solar CETA-CIEMAT EELA Tutorial, Mérida, 07-10.11.2006.](https://reader036.fdocuments.net/reader036/viewer/2022062515/56649d2a5503460f949ff0fc/html5/thumbnails/8.jpg)
EELA Tutorial, Mérida, 14-16.11.2006
E-infrastructure shared between Europe and Latin America
WMS Architecture
Job managementJob managementrequests (submission, requests (submission, cancellation) expressedcancellation) expressed
via a Job Descriptionvia a Job DescriptionLanguage (JDL)Language (JDL)
Keeps submission Keeps submission requestsrequests
Requests are keptRequests are kept for a whilefor a while
if no resources are if no resources are immediately availableimmediately available
New! New!
New!
![Page 9: E-infrastructure shared between Europe and Latin America Architecture of the WMS Manuel Rubio del Solar CETA-CIEMAT EELA Tutorial, Mérida, 07-10.11.2006.](https://reader036.fdocuments.net/reader036/viewer/2022062515/56649d2a5503460f949ff0fc/html5/thumbnails/9.jpg)
EELA Tutorial, Mérida, 14-16.11.2006
E-infrastructure shared between Europe and Latin America
WMS Architecture
Job managementJob managementrequests (submission, requests (submission, cancellation) expressedcancellation) expressed
via a Job Descriptionvia a Job DescriptionLanguage (JDL)Language (JDL)
Keeps submission Keeps submission requestsrequests
Requests are keptRequests are kept for a whilefor a while
if no resources are if no resources are immediately availableimmediately available
Repository of resourceRepository of resource informationinformation
available to matchmakeravailable to matchmaker
Updated via notifications Updated via notifications and/or active and/or active
polling on resourcespolling on resources
New! New!
New!
![Page 10: E-infrastructure shared between Europe and Latin America Architecture of the WMS Manuel Rubio del Solar CETA-CIEMAT EELA Tutorial, Mérida, 07-10.11.2006.](https://reader036.fdocuments.net/reader036/viewer/2022062515/56649d2a5503460f949ff0fc/html5/thumbnails/10.jpg)
EELA Tutorial, Mérida, 14-16.11.2006
E-infrastructure shared between Europe and Latin America
WMS Architecture
Performs the actual Performs the actual job submission job submission and monitoring and monitoring
New! New!
New!
![Page 11: E-infrastructure shared between Europe and Latin America Architecture of the WMS Manuel Rubio del Solar CETA-CIEMAT EELA Tutorial, Mérida, 07-10.11.2006.](https://reader036.fdocuments.net/reader036/viewer/2022062515/56649d2a5503460f949ff0fc/html5/thumbnails/11.jpg)
EELA Tutorial, Mérida, 14-16.11.2006
E-infrastructure shared between Europe and Latin America
• The Network Server (NS) is a generic network daemon that provides support for the job control functionality. It is responsible for accepting incoming requests from the WMS-UI (e.g. job submission, job removal), which, if valid, are then passed to the Workload Manager.
• The Workload Manager Proxy (WMProxy) is a service providing access to WMS functionality through a Web Services based interface. Besides being the natural replacement of the NS in the passage to the SOA approach for the WMS architecture, it provides additional features such as bulk submission and the support for shared and compressed sandboxes for compound jobs.
NS and WMProxy
New!
![Page 12: E-infrastructure shared between Europe and Latin America Architecture of the WMS Manuel Rubio del Solar CETA-CIEMAT EELA Tutorial, Mérida, 07-10.11.2006.](https://reader036.fdocuments.net/reader036/viewer/2022062515/56649d2a5503460f949ff0fc/html5/thumbnails/12.jpg)
EELA Tutorial, Mérida, 14-16.11.2006
E-infrastructure shared between Europe and Latin America
• ISM represents one of the most notable improvements in the WM
• The ISM basically consists of a repository of resource information that is available in read only mode to the matchmaking engine
– the update is the result of the arrival of notifications active polling of resources some arbitrary combination of both
WMS Information Supermarket (ISM)
New!
![Page 13: E-infrastructure shared between Europe and Latin America Architecture of the WMS Manuel Rubio del Solar CETA-CIEMAT EELA Tutorial, Mérida, 07-10.11.2006.](https://reader036.fdocuments.net/reader036/viewer/2022062515/56649d2a5503460f949ff0fc/html5/thumbnails/13.jpg)
EELA Tutorial, Mérida, 14-16.11.2006
E-infrastructure shared between Europe and Latin America
• The Task Queue represents the second most notable improvement in the WM internal design– possibility to keep a submission request for a
while if no resources are immediately available that match the job requirements technique used by the AliEn and Condor systems
• Non-matching requests – will be retried either periodically
eager scheduling approach– or as soon as notifications of available resources
appear in the ISM lazy scheduling approach
WMS Task Queue
New!
![Page 14: E-infrastructure shared between Europe and Latin America Architecture of the WMS Manuel Rubio del Solar CETA-CIEMAT EELA Tutorial, Mérida, 07-10.11.2006.](https://reader036.fdocuments.net/reader036/viewer/2022062515/56649d2a5503460f949ff0fc/html5/thumbnails/14.jpg)
EELA Tutorial, Mérida, 14-16.11.2006
E-infrastructure shared between Europe and Latin America
WMS components handling the job during its lifetime and performs the submission
• Job Adapter (JA)– is responsible for
making the final touches to the JDL expression for a job, before it is passed to CondorC for the actual submission
creating the job wrapper script that creates the appropriate execution environment in the CE worker node
• transfer of the input and of the output sandboxes• CondorC
– responsible for performing the actual job management operations
• job submission, job removal• DAGMan
– meta-scheduler purpose is to navigate the graph determine which nodes are free of dependencies follow the execution of the corresponding jobs
WMS Job Submission Services
New!
![Page 15: E-infrastructure shared between Europe and Latin America Architecture of the WMS Manuel Rubio del Solar CETA-CIEMAT EELA Tutorial, Mérida, 07-10.11.2006.](https://reader036.fdocuments.net/reader036/viewer/2022062515/56649d2a5503460f949ff0fc/html5/thumbnails/15.jpg)
EELA Tutorial, Mérida, 14-16.11.2006
E-infrastructure shared between Europe and Latin America
• Log Monitor (LM)– is responsible for
watching the CondorC log file intercepting interesting events concerning active jobs
• Proxy Renewal Service– is responsible for assuring that,
for all the lifetime of a job, a valid user proxy exists within the WMS
MyProxy Server is contacted in order to renew the user's credential
• Logging & Bookkeeping (LB)– is responsible for
Storing events generated by the various components of the WMS
Delivering to the user information about the job‘s status
WMS Job Submission Services
![Page 16: E-infrastructure shared between Europe and Latin America Architecture of the WMS Manuel Rubio del Solar CETA-CIEMAT EELA Tutorial, Mérida, 07-10.11.2006.](https://reader036.fdocuments.net/reader036/viewer/2022062515/56649d2a5503460f949ff0fc/html5/thumbnails/16.jpg)
EELA Tutorial, Mérida, 14-16.11.2006
E-infrastructure shared between Europe and Latin America
WMS Job Submission Services
Jobs State Machine (1/9)Submitted job is entered by the user to the User Interface but not yet transferred to Network Server for processing
![Page 17: E-infrastructure shared between Europe and Latin America Architecture of the WMS Manuel Rubio del Solar CETA-CIEMAT EELA Tutorial, Mérida, 07-10.11.2006.](https://reader036.fdocuments.net/reader036/viewer/2022062515/56649d2a5503460f949ff0fc/html5/thumbnails/17.jpg)
EELA Tutorial, Mérida, 14-16.11.2006
E-infrastructure shared between Europe and Latin America
WMS Job Submission Services
Jobs State Machine (2/9)
Waiting job accepted by NS and waiting for Workload Manager processing or being processed by WMHelper modules.
![Page 18: E-infrastructure shared between Europe and Latin America Architecture of the WMS Manuel Rubio del Solar CETA-CIEMAT EELA Tutorial, Mérida, 07-10.11.2006.](https://reader036.fdocuments.net/reader036/viewer/2022062515/56649d2a5503460f949ff0fc/html5/thumbnails/18.jpg)
EELA Tutorial, Mérida, 14-16.11.2006
E-infrastructure shared between Europe and Latin America
WMS Job Submission Services
Jobs State Machine (3/9)Ready job processed by WM but not yet transferred to the CE (local batch system queue).
![Page 19: E-infrastructure shared between Europe and Latin America Architecture of the WMS Manuel Rubio del Solar CETA-CIEMAT EELA Tutorial, Mérida, 07-10.11.2006.](https://reader036.fdocuments.net/reader036/viewer/2022062515/56649d2a5503460f949ff0fc/html5/thumbnails/19.jpg)
EELA Tutorial, Mérida, 14-16.11.2006
E-infrastructure shared between Europe and Latin America
WMS Job Submission Services
Jobs State Machine (4/9)Scheduled job waiting in the queue on the CE.
![Page 20: E-infrastructure shared between Europe and Latin America Architecture of the WMS Manuel Rubio del Solar CETA-CIEMAT EELA Tutorial, Mérida, 07-10.11.2006.](https://reader036.fdocuments.net/reader036/viewer/2022062515/56649d2a5503460f949ff0fc/html5/thumbnails/20.jpg)
EELA Tutorial, Mérida, 14-16.11.2006
E-infrastructure shared between Europe and Latin America
WMS Job Submission Services
Jobs State Machine (5/9)
Running job is running.
![Page 21: E-infrastructure shared between Europe and Latin America Architecture of the WMS Manuel Rubio del Solar CETA-CIEMAT EELA Tutorial, Mérida, 07-10.11.2006.](https://reader036.fdocuments.net/reader036/viewer/2022062515/56649d2a5503460f949ff0fc/html5/thumbnails/21.jpg)
EELA Tutorial, Mérida, 14-16.11.2006
E-infrastructure shared between Europe and Latin America
WMS Job Submission Services
Jobs State Machine (6/9)
Done job exited or considered to be in a terminal state by CondorC (e.g., submission to CE has failed in an unrecoverable way).
![Page 22: E-infrastructure shared between Europe and Latin America Architecture of the WMS Manuel Rubio del Solar CETA-CIEMAT EELA Tutorial, Mérida, 07-10.11.2006.](https://reader036.fdocuments.net/reader036/viewer/2022062515/56649d2a5503460f949ff0fc/html5/thumbnails/22.jpg)
EELA Tutorial, Mérida, 14-16.11.2006
E-infrastructure shared between Europe and Latin America
WMS Job Submission Services
Jobs State Machine (7/9)Aborted job processing was aborted by WMS (waiting in the WM queue or CE for too long, expiration of user credentials).
![Page 23: E-infrastructure shared between Europe and Latin America Architecture of the WMS Manuel Rubio del Solar CETA-CIEMAT EELA Tutorial, Mérida, 07-10.11.2006.](https://reader036.fdocuments.net/reader036/viewer/2022062515/56649d2a5503460f949ff0fc/html5/thumbnails/23.jpg)
EELA Tutorial, Mérida, 14-16.11.2006
E-infrastructure shared between Europe and Latin America
WMS Job Submission Services
Jobs State Machine (8/9)Cancelled job has been successfully canceled on user request.
![Page 24: E-infrastructure shared between Europe and Latin America Architecture of the WMS Manuel Rubio del Solar CETA-CIEMAT EELA Tutorial, Mérida, 07-10.11.2006.](https://reader036.fdocuments.net/reader036/viewer/2022062515/56649d2a5503460f949ff0fc/html5/thumbnails/24.jpg)
EELA Tutorial, Mérida, 14-16.11.2006
E-infrastructure shared between Europe and Latin America
WMS Job Submission Services
Jobs State Machine (9/9)
Cleared output sandbox was transferred to the user or removed due to the timeout.
![Page 25: E-infrastructure shared between Europe and Latin America Architecture of the WMS Manuel Rubio del Solar CETA-CIEMAT EELA Tutorial, Mérida, 07-10.11.2006.](https://reader036.fdocuments.net/reader036/viewer/2022062515/56649d2a5503460f949ff0fc/html5/thumbnails/25.jpg)
EELA Tutorial, Mérida, 14-16.11.2006
E-infrastructure shared between Europe and Latin America
Service architecture
““User User interface”interface”
““possible possible operations”operations”
Find the list of resources suitable to run a specific job
Submit a job/DAG for execution on a remote Computing Element
Check the status of a submitted job/DAG
Cancel one or more submitted jobs/DAGs
Retrieve the output files of a completed job/DAG (output sandbox)
Retrieve and display bookkeeping information about submitted jobs/DAGs
Retrieve and display logging information about submitted jobs/DAGs
Retrieve checkpoint states of a submitted checkpointable job
Start a local listener for an interactive job
![Page 26: E-infrastructure shared between Europe and Latin America Architecture of the WMS Manuel Rubio del Solar CETA-CIEMAT EELA Tutorial, Mérida, 07-10.11.2006.](https://reader036.fdocuments.net/reader036/viewer/2022062515/56649d2a5503460f949ff0fc/html5/thumbnails/26.jpg)
EELA Tutorial, Mérida, 14-16.11.2006
E-infrastructure shared between Europe and Latin America
Command Line Interface
• The most relevant commands to interact with the WMS (NS):– edg-job-submit <jdl_file>– edg-job-list-match <jdl_file>– edg-job-status <job_Id>– edg-job-get-output <job_Id>– edg-job-cancel <job_Id>
• In gLite 3.0:– glite-job-submit <jdl_file>– glite-job-list-match <jdl_file>– glite-job-status <job_Id>– glite-job-output <job_Id>– glite-job-cancel <job_Id>
![Page 27: E-infrastructure shared between Europe and Latin America Architecture of the WMS Manuel Rubio del Solar CETA-CIEMAT EELA Tutorial, Mérida, 07-10.11.2006.](https://reader036.fdocuments.net/reader036/viewer/2022062515/56649d2a5503460f949ff0fc/html5/thumbnails/27.jpg)
EELA Tutorial, Mérida, 14-16.11.2006
E-infrastructure shared between Europe and Latin America
Command Line InterfaceCommand Line
Interface• Job Submission
– Perform the job submission to the Grid.$ edg-job-submit [options] <jdl_file>
$ glite-job-submit [options] <jdl_file>
– where <jdl file> is a file containing the job description, usually with extension .jdl.
Now, all examples with edg-*
![Page 28: E-infrastructure shared between Europe and Latin America Architecture of the WMS Manuel Rubio del Solar CETA-CIEMAT EELA Tutorial, Mérida, 07-10.11.2006.](https://reader036.fdocuments.net/reader036/viewer/2022062515/56649d2a5503460f949ff0fc/html5/thumbnails/28.jpg)
EELA Tutorial, Mérida, 14-16.11.2006
E-infrastructure shared between Europe and Latin America
Command Line InterfaceCommand Line
InterfaceIf the request has been correctly submitted this is the tipical output that you can get:
edg-job-submit test.jdl
====================glite-job-submit Success =====================The job has been successfully submitted to the Network Server.Use edg-job-status command to check job current status. Your job identifier (edg_jobId) is:- https://lxshare0234.cern.ch:9000/rIBubkFFKhnSQ6CjiLUY8Q==============================================================
In case of failure, an error message will be displayed instead, and an exit status different form zero will be retured.
![Page 29: E-infrastructure shared between Europe and Latin America Architecture of the WMS Manuel Rubio del Solar CETA-CIEMAT EELA Tutorial, Mérida, 07-10.11.2006.](https://reader036.fdocuments.net/reader036/viewer/2022062515/56649d2a5503460f949ff0fc/html5/thumbnails/29.jpg)
EELA Tutorial, Mérida, 14-16.11.2006
E-infrastructure shared between Europe and Latin America
Command Line InterfaceCommand Line
InterfaceIt is possible to see which CEs are eligible to run a job specified by a given JDL file using the command
edg-job-list-match test.jdl
Connecting to host lxshare0380.cern.ch, port 7772Selected Virtual Organisation name (from UI conf file): dteam*********************************************************************COMPUTING ELEMENT IDs LISTThe following CE(s) matching your job requirements have been found:adc0015.cern.ch:2119/jobmanager-lcgpbs-infiniteadc0015.cern.ch:2119/jobmanager-lcgpbs-longadc0015.cern.ch:2119/jobmanager-lcgpbs-short**********************************************************************
![Page 30: E-infrastructure shared between Europe and Latin America Architecture of the WMS Manuel Rubio del Solar CETA-CIEMAT EELA Tutorial, Mérida, 07-10.11.2006.](https://reader036.fdocuments.net/reader036/viewer/2022062515/56649d2a5503460f949ff0fc/html5/thumbnails/30.jpg)
EELA Tutorial, Mérida, 14-16.11.2006
E-infrastructure shared between Europe and Latin America
Command Line InterfaceCommand Line
InterfaceAfter a job is submitted, it is possible to see its status using the glite-job-status command.
edg-job-status https://lxshare0234.cern.ch:9000/X-ehTxfdlXxSoIdVLS0L0w
*************************************************************BOOKKEEPING INFORMATION:Printing status info for the Job:https://lxshare0234.cern.ch:9000/X-ehTxfdlXxSoIdVLS0L0wCurrent Status: ScheduledStatus Reason: unavailableDestination: lxshare0277.cern.ch:2119/jobmanager-pbs-infinitereached on: Fri Aug 1 12:21:35 2003*************************************************************
![Page 31: E-infrastructure shared between Europe and Latin America Architecture of the WMS Manuel Rubio del Solar CETA-CIEMAT EELA Tutorial, Mérida, 07-10.11.2006.](https://reader036.fdocuments.net/reader036/viewer/2022062515/56649d2a5503460f949ff0fc/html5/thumbnails/31.jpg)
EELA Tutorial, Mérida, 14-16.11.2006
E-infrastructure shared between Europe and Latin America
Command Line InterfaceCommand Line
InterfaceA job can be canceled before it ends using the command glite-job-cancel.
edg-job-cancel https://lxshare0234.cern.ch:9000/dAE162is6EStca0VqhVkog
Are you sure you want to remove specified job(s)? [y/n]n :y=================== glite-job-cancel Success====================The cancellation request has been successfully submitted for the following job(s)- https://lxshare0234.cern.ch:9000/dAE162is6EStca0VqhVkog===========================================================
![Page 32: E-infrastructure shared between Europe and Latin America Architecture of the WMS Manuel Rubio del Solar CETA-CIEMAT EELA Tutorial, Mérida, 07-10.11.2006.](https://reader036.fdocuments.net/reader036/viewer/2022062515/56649d2a5503460f949ff0fc/html5/thumbnails/32.jpg)
EELA Tutorial, Mérida, 14-16.11.2006
E-infrastructure shared between Europe and Latin America
Command Line InterfaceCommand Line
InterfaceAfter the job has finished (it reaches the DONE status), its output can be copied to the UI
edg-job-get-output https://lxshare0234.cern.ch:9000/snPegp1YMJcnS22yF5pFlg
Retrieving files from host lxshare0234.cern.ch*****************************************************************JOB GET OUTPUT OUTCOMEOutput sandbox files for the job:- https://lxshare0234.cern.ch:9000/snPegp1YMJcnS22yF5pFlghave been successfully retrieved and stored in the directory:/tmp/jobOutput/snPegp1YMJcnS22yF5pFlg*****************************************************************
By default, the output is stored under /tmp, but it is possible to specify in which directory to save theoutput using the - -dir <path name> option.
![Page 33: E-infrastructure shared between Europe and Latin America Architecture of the WMS Manuel Rubio del Solar CETA-CIEMAT EELA Tutorial, Mérida, 07-10.11.2006.](https://reader036.fdocuments.net/reader036/viewer/2022062515/56649d2a5503460f949ff0fc/html5/thumbnails/33.jpg)
EELA Tutorial, Mérida, 14-16.11.2006
E-infrastructure shared between Europe and Latin America
Second PartJob Description Language
![Page 34: E-infrastructure shared between Europe and Latin America Architecture of the WMS Manuel Rubio del Solar CETA-CIEMAT EELA Tutorial, Mérida, 07-10.11.2006.](https://reader036.fdocuments.net/reader036/viewer/2022062515/56649d2a5503460f949ff0fc/html5/thumbnails/34.jpg)
EELA Tutorial, Mérida, 14-16.11.2006
E-infrastructure shared between Europe and Latin America
Job Description Language
The JDL is used in gLite to specify the job’s characteristics and constrains, which are used during the match-making processmatch-making process to select the
best resources that satisfy job’s requirements.
![Page 35: E-infrastructure shared between Europe and Latin America Architecture of the WMS Manuel Rubio del Solar CETA-CIEMAT EELA Tutorial, Mérida, 07-10.11.2006.](https://reader036.fdocuments.net/reader036/viewer/2022062515/56649d2a5503460f949ff0fc/html5/thumbnails/35.jpg)
EELA Tutorial, Mérida, 14-16.11.2006
E-infrastructure shared between Europe and Latin America
The JDL syntax JDL syntax consists on statements like:
Attribute = value;Attribute = value;
Comments must be preceded by a sharp character ( ## ) or have to follow the C++ syntax
WARNING: The JDL is sensitive to blank
characters and tabs. No blank charactersor tabs should follow the
semicolon at the end of a line.
Job Description Language (cont.)
![Page 36: E-infrastructure shared between Europe and Latin America Architecture of the WMS Manuel Rubio del Solar CETA-CIEMAT EELA Tutorial, Mérida, 07-10.11.2006.](https://reader036.fdocuments.net/reader036/viewer/2022062515/56649d2a5503460f949ff0fc/html5/thumbnails/36.jpg)
EELA Tutorial, Mérida, 14-16.11.2006
E-infrastructure shared between Europe and Latin America
Job Description Language (cont.)
In a JDL, some attributes are mandatory while others are optional.An “essential” JDL is the following:Executable = “test.sh”;
StdOutput = “std.out”;StdError = “std.err”;
InputSandbox = {“test.sh”};OutputSandbox = {“std.out”,”std.err”};
If needed, arguments to the executable can be passed:Arguments = “Hello World!”;
![Page 37: E-infrastructure shared between Europe and Latin America Architecture of the WMS Manuel Rubio del Solar CETA-CIEMAT EELA Tutorial, Mérida, 07-10.11.2006.](https://reader036.fdocuments.net/reader036/viewer/2022062515/56649d2a5503460f949ff0fc/html5/thumbnails/37.jpg)
EELA Tutorial, Mérida, 14-16.11.2006
E-infrastructure shared between Europe and Latin America
If the argument contains quoted strings, the quotes must be escaped with a backslash
e.g. Arguments = “\”Hello World!\“ 10”;
Special characters such as &, |, >, < are only allowed if specified inside a quoted string or preceded by triple \
(e.g. Arguments = "-f file1\\\&file2";)
Job Description Language (cont.)
![Page 38: E-infrastructure shared between Europe and Latin America Architecture of the WMS Manuel Rubio del Solar CETA-CIEMAT EELA Tutorial, Mérida, 07-10.11.2006.](https://reader036.fdocuments.net/reader036/viewer/2022062515/56649d2a5503460f949ff0fc/html5/thumbnails/38.jpg)
EELA Tutorial, Mérida, 14-16.11.2006
E-infrastructure shared between Europe and Latin America
Workload Manager Service
•The JDL allows the description of the following request types supported by the WMS:
Job: a simple application
DAG: a direct acyclic graph of dependent jobs
• With WMSProxy
Collection: a set of independent jobs
• With WMSProxy
![Page 39: E-infrastructure shared between Europe and Latin America Architecture of the WMS Manuel Rubio del Solar CETA-CIEMAT EELA Tutorial, Mérida, 07-10.11.2006.](https://reader036.fdocuments.net/reader036/viewer/2022062515/56649d2a5503460f949ff0fc/html5/thumbnails/39.jpg)
EELA Tutorial, Mérida, 14-16.11.2006
E-infrastructure shared between Europe and Latin America
Jobs
•The Workload Management System currently supports the following types for Jobs :
•Normal a simple batch, a set of commands to be processed as single unit
•Interactive a job whose standard streams are forwarded to the submitting client
•MPICH a parallel application using MPICH-P4 implementation of MPI
•Partitionable a job which is composed by a set of independent steps/iterations
•Checkpointable a job able to save its state
•Parametric a job where one or more of its attributes are parameterized
Support for MPI and parametric jobs is only available when the submission to the WMS is done through the WMProxy service
a set of independent sub-jobs, each one taking care of a step or of a sub-set of steps, and which can be executed in parallel
the job execution can be suspended and resumed later, starting from the same point where it was first stopped
![Page 40: E-infrastructure shared between Europe and Latin America Architecture of the WMS Manuel Rubio del Solar CETA-CIEMAT EELA Tutorial, Mérida, 07-10.11.2006.](https://reader036.fdocuments.net/reader036/viewer/2022062515/56649d2a5503460f949ff0fc/html5/thumbnails/40.jpg)
EELA Tutorial, Mérida, 14-16.11.2006
E-infrastructure shared between Europe and Latin America
JDL: Relevant Attributes
JobTypeJobType (optional)
Normal (simple, sequential job), Interactive, MPICH, Checkpointable, Partitionable, Parametric
Or combination of them Checkpointable, InteractiveCheckpointable, MPI
E.g. JobType = “Interactive”; JobType = “Interactive”;
JobType = {“Interactive”,”Checkpointable”}; JobType = {“Interactive”,”Checkpointable”};
““Interactive” + “MPI” not yet permitted Interactive” + “MPI” not yet permitted
![Page 41: E-infrastructure shared between Europe and Latin America Architecture of the WMS Manuel Rubio del Solar CETA-CIEMAT EELA Tutorial, Mérida, 07-10.11.2006.](https://reader036.fdocuments.net/reader036/viewer/2022062515/56649d2a5503460f949ff0fc/html5/thumbnails/41.jpg)
EELA Tutorial, Mérida, 14-16.11.2006
E-infrastructure shared between Europe and Latin America
ExecutableExecutable (mandatory)
This is a string representing the executable/command name.The user can specify an executable which is already on the remote CE Executable = Executable = {“/opt/EGEODE/GCT/egeode.sh”};{“/opt/EGEODE/GCT/egeode.sh”};
The user can provide a local executable name, which will be staged from the UI to the WN.
Executable = {“egeode.sh”};Executable = {“egeode.sh”};
InputSandbox = {“/home/larocca/egeode/InputSandbox = {“/home/larocca/egeode/
egeode.sh”};egeode.sh”};
JDL: Relevant Attributes (cont.)
![Page 42: E-infrastructure shared between Europe and Latin America Architecture of the WMS Manuel Rubio del Solar CETA-CIEMAT EELA Tutorial, Mérida, 07-10.11.2006.](https://reader036.fdocuments.net/reader036/viewer/2022062515/56649d2a5503460f949ff0fc/html5/thumbnails/42.jpg)
EELA Tutorial, Mérida, 14-16.11.2006
E-infrastructure shared between Europe and Latin America
ArgumentsArguments (optional)This is a string containing all the job command line arguments.
E.g.: If your executable sum has to be started as:
$ sum N1 N2 –out result.out
Executable = “sum”;Executable = “sum”;
Arguments = “N1 N2 –out result.out”;Arguments = “N1 N2 –out result.out”;
JDL: Relevant Attributes (cont.)
![Page 43: E-infrastructure shared between Europe and Latin America Architecture of the WMS Manuel Rubio del Solar CETA-CIEMAT EELA Tutorial, Mérida, 07-10.11.2006.](https://reader036.fdocuments.net/reader036/viewer/2022062515/56649d2a5503460f949ff0fc/html5/thumbnails/43.jpg)
EELA Tutorial, Mérida, 14-16.11.2006
E-infrastructure shared between Europe and Latin America
EnvironmentEnvironment (optional)
List of environment settings needed by the job to run properlyE.g. Environment = Environment = {“JAVA_HOME=/usr/java/j2sdk1.4.2_08”};{“JAVA_HOME=/usr/java/j2sdk1.4.2_08”};
InputSandboxInputSandbox (optional)
List of files on the UI local disk needed by the job for proper runningThe listed files will be automatically staged to the remote resource
E.g. InputSandbox InputSandbox ={“myscript.sh”,”/tmp/cc.sh”};={“myscript.sh”,”/tmp/cc.sh”};
JDL: Relevant Attributes (cont.)
![Page 44: E-infrastructure shared between Europe and Latin America Architecture of the WMS Manuel Rubio del Solar CETA-CIEMAT EELA Tutorial, Mérida, 07-10.11.2006.](https://reader036.fdocuments.net/reader036/viewer/2022062515/56649d2a5503460f949ff0fc/html5/thumbnails/44.jpg)
EELA Tutorial, Mérida, 14-16.11.2006
E-infrastructure shared between Europe and Latin America
OutputSandboxOutputSandbox (optional)
List of files, generated by the job, which have to be retrieved from the CE
E.g. OutputSandbox ={ “std.out”,”std.err”,OutputSandbox ={ “std.out”,”std.err”,
“ “image.png”};image.png”};
JDL: Relevant Attributes (cont.)
![Page 45: E-infrastructure shared between Europe and Latin America Architecture of the WMS Manuel Rubio del Solar CETA-CIEMAT EELA Tutorial, Mérida, 07-10.11.2006.](https://reader036.fdocuments.net/reader036/viewer/2022062515/56649d2a5503460f949ff0fc/html5/thumbnails/45.jpg)
EELA Tutorial, Mérida, 14-16.11.2006
E-infrastructure shared between Europe and Latin America
RequirementsRequirements (optional)
Job requirements on computing resources Specified using attributes of resources published in the Information ServiceIf not specified, default value defined in UI configuration file is consideredDefault. Requirements = Requirements =
other.GlueCEStateStatus == other.GlueCEStateStatus == "Production“;"Production“;
Requirements=other.GlueCEUniqueID == “adc006.cern.ch:2119/jobmanager-pbs-infinite”
Requirements=Member(“ALICE-3.07.01”, other.GlueHostApplicationSoftwareRunTimeEnvironment);
JDL: Relevant Attributes (cont.)
![Page 46: E-infrastructure shared between Europe and Latin America Architecture of the WMS Manuel Rubio del Solar CETA-CIEMAT EELA Tutorial, Mérida, 07-10.11.2006.](https://reader036.fdocuments.net/reader036/viewer/2022062515/56649d2a5503460f949ff0fc/html5/thumbnails/46.jpg)
EELA Tutorial, Mérida, 14-16.11.2006
E-infrastructure shared between Europe and Latin America
References
JDL Attributes
http://server11.infn.it/workload-grid/docs/DataGrid-01-TEN-0142-0_2.pdf
https://edms.cern.ch/document/590869/1
http://egee-jra1-wm.mi.infn.it/egee-jra1-wm/api_doc/wms_jdl/index.html
• LCG-2 User Guide Manual Series
https://edms.cern.ch/file/454439/LCG-2-UserGuide.html
![Page 47: E-infrastructure shared between Europe and Latin America Architecture of the WMS Manuel Rubio del Solar CETA-CIEMAT EELA Tutorial, Mérida, 07-10.11.2006.](https://reader036.fdocuments.net/reader036/viewer/2022062515/56649d2a5503460f949ff0fc/html5/thumbnails/47.jpg)
EELA Tutorial, Mérida, 14-16.11.2006
E-infrastructure shared between Europe and Latin America
Third PartWorkload Manager Proxy
![Page 48: E-infrastructure shared between Europe and Latin America Architecture of the WMS Manuel Rubio del Solar CETA-CIEMAT EELA Tutorial, Mérida, 07-10.11.2006.](https://reader036.fdocuments.net/reader036/viewer/2022062515/56649d2a5503460f949ff0fc/html5/thumbnails/48.jpg)
EELA Tutorial, Mérida, 14-16.11.2006
E-infrastructure shared between Europe and Latin America
WMProxy
• WMProxy (Workload Manager Proxy) – is a new service providing access to the gLite Workload
Management System (WMS) functionality through a simple Web Services based interface.
– has been designed to handle a large number of requests for job submission gLite 1.5 => ~180 secs for 500 jobs
goal is to get in the short term to ~60 secs for 1000 jobs
– it provides additional features such as bulk submission and the support for shared and compressed sandboxes for compound jobs.
– It’s the natural replacement of the NS in the passage to the SOA approach.
![Page 49: E-infrastructure shared between Europe and Latin America Architecture of the WMS Manuel Rubio del Solar CETA-CIEMAT EELA Tutorial, Mérida, 07-10.11.2006.](https://reader036.fdocuments.net/reader036/viewer/2022062515/56649d2a5503460f949ff0fc/html5/thumbnails/49.jpg)
EELA Tutorial, Mérida, 14-16.11.2006
E-infrastructure shared between Europe and Latin America
New request types
• Support for new types strongly relies on newly developed JDL converters and on the DAG submission support– all JDL conversions are performed on the server– a single submission for several jobs
• All new request types can be monitored and controlled through a single handle (the request id)– each sub-jobs can be however followed-up and controlled
independently through its own id
• “Smarter” WMS client commands/API – allow submission of DAGs, collections and parametric jobs
exploiting the concept of “shared sandbox”– allow automatic generation and submission of collections and
DAGs from sets of JDL files located in user specified directories on the UI
![Page 50: E-infrastructure shared between Europe and Latin America Architecture of the WMS Manuel Rubio del Solar CETA-CIEMAT EELA Tutorial, Mérida, 07-10.11.2006.](https://reader036.fdocuments.net/reader036/viewer/2022062515/56649d2a5503460f949ff0fc/html5/thumbnails/50.jpg)
EELA Tutorial, Mérida, 14-16.11.2006
E-infrastructure shared between Europe and Latin America
WMProxy C++ client commands
The commands to interact with WMProxy Service are:
glite-wms-job-submit <jdl_file>
glite-wms-job-list-match <jdl_file>
glite-wms-job-cancel <job_Ids>
glite-wms-job-output <job_Ids>
In our examples:glite-wms-job-*
areedg-job-*
![Page 51: E-infrastructure shared between Europe and Latin America Architecture of the WMS Manuel Rubio del Solar CETA-CIEMAT EELA Tutorial, Mérida, 07-10.11.2006.](https://reader036.fdocuments.net/reader036/viewer/2022062515/56649d2a5503460f949ff0fc/html5/thumbnails/51.jpg)
EELA Tutorial, Mérida, 14-16.11.2006
E-infrastructure shared between Europe and Latin America
References
• gLite 3.0 User Guide– https://edms.cern.ch/file/722398/1.1/gLite-3-UserGuide.pdf
• R-GMA overview page– http://www.r-gma.org/
• GLUE Schema– http://infnforge.cnaf.infn.it/glueinfomodel/
• JDL attributes specification for WM proxy– https://edms.cern.ch/document/590869/1
• WMProxy quickstart– http://egee-jra1-wm.mi.infn.it/egee-jra1-wm/wmproxy_client_quickst
art.shtml
• WMS user guides– https://edms.cern.ch/document/572489/1
![Page 52: E-infrastructure shared between Europe and Latin America Architecture of the WMS Manuel Rubio del Solar CETA-CIEMAT EELA Tutorial, Mérida, 07-10.11.2006.](https://reader036.fdocuments.net/reader036/viewer/2022062515/56649d2a5503460f949ff0fc/html5/thumbnails/52.jpg)
EELA Tutorial, Mérida, 14-16.11.2006
E-infrastructure shared between Europe and Latin America
Questions…