EGEE is a project funded by the European Union under contract IST-2003-508833 EGEE Tutorial Turin -...

EGEE is a project funded by the European Union under contract IST-2003-508833

EGEE TutorialTurin - ITALY

18 – 19 January 2005

www.eu-egee.org

Job Services and Data Services

Giuseppe La RoccaINFN Catania

EGEE Tutorial – Turin – Job Services and Data Services – January 18-19, 2005 - 2

Overview

Files on the RLS - Name Convention

Job Description Language - Overview

Relevant attributes

JDL Examples

Job without Data Requirements

Job with Output Data

Job with Input and Output Data

Other examples using the lcg-* commands

Bibliography

Summary and Conclusions


Files on the RLS - Name Convention

Logical File Name 1

Logical File Name n

GUID

Physical File SURL n

Physical File SURL 1

.

. ..

Globally Unique Identifier (Globally Unique Identifier (GUIDGUID)) A non-human-readable unique identifier for a file, e.g.“guid:f81d4fae-7dec-11d0-a765-00a0c91e6bf6”

Site URL (Site URL (SURLSURL)) (or Physical/Site File Name (PFN/SFNPFN/SFN))The location of the actual file on a storage system, e.g. “sfn://lxshare0209.cern.ch/data/alice/ntuples.dat”

Logical File Name (Logical File Name (LFNLFN)) An alias created by a user to refer to some file, e.g. “lfn:cms/track1”

Transport URL (Transport URL (TURLTURL))A temporary URL which can be used to access a particular data file contained in a Storage Element e.g.“gsiftp://lxshare0209.cern.ch//data/alice/ntuples.dat”


Job Description Language (1)

In LCG-2 Job Description Language (JDL)Job Description Language (JDL) are used to describe jobs for execution on Grid.

The JDL adopted within the LCG-2 middleware is based upon Condor’s CLASSified Advertisement language (ClassAd)CLASSified Advertisement language (ClassAd).

A ClassAd is a record-like structure composed of a finite number of attribute separated by semi-colon (;)A ClassAd is highly flexible and can be used to represent arbitrary services

The JDL is used in LCG-2 to specify the desired job characteristics and constrains, which are used in by match-match-making processmaking process to select the resources that the job will use.


The JDL syntax JDL syntax consists on statements like:

Attribute = value;Attribute = value;

Comments must be preceded by a sharp character ( ## ) or have to follow the C++ syntax

WARING: The JDL is sensitive to blank

characters and tabs. No blank characters

or tabs should follow the semicolon

at the end of a line.



In a JDL, some attributes are mandatory while others are optional.

A essential JDL is the following:

Executable = “test.sh”;

StdOutput = “std.out”;

StdError = “std.err”;

InputSandbox = {“test.sh”};

OutputSandbox = {“std.out”,”std.err”};

If needed, arguments to the executable can be passed:

Arguments = “Hello World!”;



If the arguments contains quoted strings, the quotes must be escaped with a backslash

e.g. Arguments = “\”Hello World!\“ 10”;

Special characters such as &, |, >, < are only allowed if specified inside a quoted string or preceded by triple \

(e.g. Arguments = "-f file1\\\&file2";)



The supported attributes are grouped in two categories:

Job Attributes Define the job itself

ResourcesTaken into account by the RB for carrying out the matchmaking algorithm (to choose the “best” resource where to submit the job)

Computing Resource– Used to build expressions of Requirements and/or Rank attributes by

the user

– Have to be prefixed with “other.”

Data and Storage resources– Input data to process, SE where to store output data, protocols

spoken by application when accessing SEs



JDL: Relevant Attributes (1)

JobTypeJobType (optional)

Normal (simple, sequential job), Interactive, MPICH, Checkpointable, Partitionable

Or combination of them Checkpointable, Interactive

Checkpointable, MPI

E.g. JobType = “Interactive”; JobType = “Interactive”;

JobType = {“Interactive”,”Checkpointable”}; JobType = {“Interactive”,”Checkpointable”};


TypeType (mandatory, default “Job”)

This is a representing the type of the request described by the JDL.

Possible values are:Job

DAG

Reservation

Co-allocation

E.g. Type = “Job”;Type = “Job”;



ExecutableExecutable (mandatory)

This is a string representing the executable/command name.

The user can specify an executable that lies already on the remote CE

Executable = {“/opt/EGEODE/GCT/egeode.sh”};Executable = {“/opt/EGEODE/GCT/egeode.sh”};

The user can provide a local executable name, which will be staged from the UI to the WN.

Executable = {“egeode.sh”};Executable = {“egeode.sh”};

InputSandbox = {“/home/larocca/egeode/InputSandbox = {“/home/larocca/egeode/

egeode.sh”};egeode.sh”};



ArgumentsArguments (optional)

This is a string containing all the job command line arguments.

E.g.: If your executable sum has to be started as:

$ sum N1 N2 –out result.out

Executable = “sum”;Executable = “sum”;

Arguments = “N1 N2 –out result.out”;Arguments = “N1 N2 –out result.out”;



EnvironmentEnvironment (optional)

List of environment settings needed by the job to run properly

E.g. Environment = {“JAVABIN=/usr/local/java”};Environment = {“JAVABIN=/usr/local/java”};

InputSandboxInputSandbox (optional)

List of files on the UI local disk needed by the job for running

The listed files will automatically staged to the remote resource

E.g. InputSandbox ={“myscript.sh”,”/tmp/cc,sh”};InputSandbox ={“myscript.sh”,”/tmp/cc,sh”};



OutputSandboxOutputSandbox (optional)

List of files, generated by the job, which have to be retrieved

E.g. OutputSandbox ={ “std.out”,OutputSandbox ={ “std.out”,

” ”std.err”,std.err”,

“ “image.png”};image.png”};

VirtualOrganisationVirtualOrganisation (optional)

This is a string representing the name of the VO the submitting user is currently working for.

E.g. VirtualOrganisation =“gilda”;VirtualOrganisation =“gilda”;



RequirementsRequirements (optional)

Job requirements on computing resources

Specified using attributes of resources published in the Information Service

If not specified, default value defined in UI configuration file is considered

Default. Requirements = Requirements = other.GlueCEStateStatus == other.GlueCEStateStatus == "Production“;"Production“;



RankRank (optional)

Floating-point expression used to ranks CEs that have already met the Requirements expression.

The Rank expression can contain attributes that describe the CE in the Information System (IS).

The evaluation of the rank expression is performed by the Resource Broker (RB) during the match-making phase.

A higher numeric value equals a better rank.

If not specified, default value defined in the UI configuration file is considered

Default: Rank = -Rank = - other.GlueCEStateFreeCPUs; other.GlueCEStateFreeCPUs;



InputDataInputData (optional)

This is a string or a list of strings representing the Logical File Name (LFN) or Grid Unique Identifier (GUID) needed by the job as input.

The list is used by the RB to find the CE from which the specified files can be better accessed and schedules the job to run there.

InputData = {“lfn:cmstestfile”,InputData = {“lfn:cmstestfile”,

““guid:135b7b23-4a6a-11d7-87e7-9d101f8c8b70”};guid:135b7b23-4a6a-11d7-87e7-9d101f8c8b70”};



DataAccessProtocolDataAccessProtocol (mandatory if InputData has been specified)

The protocol or the list of protocols which the application is able to “speak” with for accessing files listed in InputData on a given SE.

Supported protocols in LCG-2 are currently gridftpgridftp, filefile and rfiorfio.

DataAccessProtocol = {“file”,“gridftp”,“rfio”};DataAccessProtocol = {“file”,“gridftp”,“rfio”};



OutputSEOutputSE (optional)

This string representing the URI of the Storage Element (SE) where the user wants to store the output data.

This attribute is used by the Resource Broker to find a CE being “close” to this SE and schedule the job there.

OutputSE = “grid009.ct.infn.it”;OutputSE = “grid009.ct.infn.it”;



OutputDataOutputData (optional)

This attribute allows the user to ask for the automatic upload and registration of datasets produced by the job on the Worker Node (WN).

This attribute contains the following three attributes:

OutputFileStorageElementLogicalFileName



OutputFileOutputFile (mandatory if OutputData has been specified)

This is a string attribute representing the name of the output file, generated by the job on the WN, which has to be automatically uploaded and registered by the WMS.

StorageElementStorageElement (optional)

This is a string representing the URI of the Storage Element where the output file specified in the OutputFile has to be uploaded by the WMS.

LogicalFileNameLogicalFileName (optional)

This is a string representing the LFN the user wants to associate to the output file when registering it to the Replica Catalogue.



OutputData = {OutputData = {[[

OutputFile = “dataset1.out”;OutputFile = “dataset1.out”;LogicalFileName = “lfn:test-result1”;LogicalFileName = “lfn:test-result1”;

],],[[

OutputFile = “dataset2.out”;OutputFile = “dataset2.out”;LogicalFileName = “lfn:test-result2”;LogicalFileName = “lfn:test-result2”;StorageElement = “grid009.ct.infn.it”;StorageElement = “grid009.ct.infn.it”;

],],[[

OutputFile = “dataset3.out”;OutputFile = “dataset3.out”;]]

};};



Examples


Job Without Data Requirements(1)

[Type = “job”;

JobType = “Normal”;

Executable = "script.sh";Arguments = “Giuseppe”;

StdOutput = “sim.out”;StdError = “sim.err”;InputSandbox = {"script.sh"};OutputSandbox = {“sim.err”,“sim.out”};


# A site with more than 4 CPU is required.

Requirements=(other.GlueCEInfoTotalCPUs>4);

# If more than one resource matches, the resource with the largest # number of CPU is chosen.

Rank = (other.GlueCEStateFreeCPUs);

]

Job Without Data Requirements(2)


script.sh

#!/bin/sh

/bin/echo Hello $1 and Welcome to the EGEE Tutorial!


Examples


Job with Output Data (1)

[ Type = “job”;


Executable = “scriptOutput.sh";Arguments = “Giuseppe”;

VirtualOrganisation = “gilda”;

StdOutput = “sim.out”;StdError = “sim.err”;InputSandbox = {"scriptOutput.sh"};


OutputSandbox = {“sim.err”, “sim.out”};

OutputData = {[

Outputfile = “TorinoXX.out”;LogicalName = “lfn:TorinoXX.out”;StorageElement = “grid009.ct.infn.it”;

]};

Requirements=(other.GlueCEInfoTotalCPUs>4); Rank=(other.GlueCEStateFreeCPUs); RetryCount = 0;

]

Job with Output Data (2)


scriptOutput.sh

#!/bin/sh

/bin/echo Hello $1 and Welcome to the EGEE Tutorial! > TorinoXX.out


Examples


Job with Input Data (1)

[ Type = “job”;


Executable = “scriptInput.sh";Arguments = “Francesco”;

VirtualOrganisation = “gilda”;

StdOutput = “std.out”;StdError = “std.err”;


InputSandbox = {“scriptInput.sh"};OutputSandbox = {“std.err”, “std.out”};

InputData = “lfn:myoutdata.1”;DataAccessProtocol = {“gridftp”,”rfio”};

Requirements=(other.GlueCEInfoTotalCPUs>4);Rank=(other.GlueCEStateFreeCPUs); RetryCount = 0;

]

Job with Input Data (2)


scriptInput.sh

#!/bin/sh

lcg-cp –vo gilda lfn:myoutdata.1 file:`pwd`/dataset1.out

echo “Before updating..”

cat dataset1.out

#Adding new entry on the dataset1.out file.

/bin/echo Hello $1 and Welcome to the EGEE Tutorial! >> dataset1.out

echo “After updating..”

cat dataset1.out


Examples


StoreData.jdl

Type="Job";JobType="Normal";

Executable="StoreData.shStoreData.sh";Arguments="myfile myfile grid009.ct.infn.itmyfile myfile grid009.ct.infn.it";VirtualOrganisation="gilda";

StdOutput="std.out";StdError="std.err";

InputSandbox={"StoreData.shStoreData.sh","myfile"};OutputSandbox={"std.out","std.err"};

Requirements=(other.GlueCEInfoTotalCPUs > 4);Rank=other.GlueCEStateFreeCPUs;


#!/bin/shARGS=3ERROR_BADARGS=10 # Bad arguments errorERROR_FILE=20 # File to copy on RLS doesn't existERROR_STORAGE=30 # Storage Element doesn't exist in this

Catalog.

# Check for proper no. of command line args.if [ $# -ne $ARGS ]then

echo "Usage: `basename $0` [text-file2copy] [lfn-file2copy] [se-name]"exit $ERROR_BADARGS

StoreData.sh (1)


else # Setting the variables for the script.FILE2COPY=$1LFN2COPY=$2SENAME=$3

fi# Check if the ${FILE2COPY} exists on the path.if [ ! -e `pwd`/${FILE2COPY} ]then

echo "Error: `basename $0.` ${FILE2COPY} does not exist on the path."exit ${ERROR_FILE}

fi

StoreData.sh (2)


# Checking if the SENAME is a good SE where to store data.lcg-infosites --vo gilda closeSE | grep ${SENAME}if [ $? -eq 0 ]then

# Uploading the copy on the RLS.lcg-cr --vo gilda -d ${SENAME} -l ${LFN2COPY} file:`pwd`/${FILE2COPY}

elseecho Sorry, but you have to specific a good Storage Element.exit $ERROR_STORAGE

fi

StoreData.sh (3)


Type="Job";JobType="Normal";

Executable=“UpdateData.shUpdateData.sh";Arguments="myfile Julia grid009.ct.infn.itmyfile Julia grid009.ct.infn.it";VirtualOrganisation="gilda";

StdOutput="std.out";StdError="std.err";

InputSandbox={“UpdateData.shUpdateData.sh"};OutputSandbox={"std.out","std.err"};

Requirements=(other.GlueCEInfoTotalCPUs > 4);Rank=other.GlueCEStateFreeCPUs;

UpdateData.jdl


UpdateData.sh (1)

#!/bin/shARGS=3ERROR_BADARGS=10 # Bad arguments error.ERROR_BADLFN=20 # LFN doesn't exist on the RLS.ERROR_BADSTORAGE=30 # Storage Element doesn't exist in this Catalog.

# Check for proper no. of command line args.if [ $# -ne $ARGS ]then

echo "Usage: `basename $0` [lfn-file2copy] [argument] [se-name]"exit ${ERROR_BADARGS}

else # Setting the arguments for the script.LFN2RETRIEVE=$1ARGUMENT=$2SENAME=$3

fi


# Check if the ${LFN2RETRIEVE} exists on the RLS.lcg-lr --vo gilda lfn:${LFN2RETRIEVE}if [ $? -eq 1 ]then

echo "Sorry, but the LFN that you have specified does not exists on the RLS".exit ${ERROR_BADLFN}

else# Retrieve the file from the catalog...lcg-cp --vo gilda lfn:${LFN2RETRIEVE} file:`pwd`/myfile# ...adding new data to myfile.echo Hello ${ARGUMENT} and Welcome to the EGEE Tutorial!! >> `pwd`/myfile

fi

# Checking if the SENAME is a good SE where to store data.lcg-infosites --vo gilda closeSE | grep ${SENAME}

UpdateData.sh (2)


if [ $? -eq 1 ]then

echo Sorry, but you have to specific a good Storage Element.exit ${ERROR_STORAGE}

else# Setting the variables for the new copy.FILE2COPY=`pwd`/myfileLFN2COPY=${LFN2RETRIEVE}# Removing the old copy form the RLS...lcg-del --vo gilda -a lfn:${LFN2COPY}# ... and staging the new data on the RLS.lcg-cr --vo gilda -d ${SENAME} -l ${LFN2COPY} file:${FILE2COPY}

echo "Your data has been correctly uploaded to the RLS.” echo “Have a nice day!!"fi

UpdateData.sh (3)


JDL Attributeshttp://server11.infn.it/workload-grid/docs/DataGrid-01-TEN-0142-0_2.pdf

LCG-2 User Guide Manual Series http://egee-na4.ct.infn.it/documentation/LCG-2-Userguide.pdf

EDG Tutorialhttp://www.dutchgrid.nl/DataGrid/introduction/edg-tutorial-handout.pdf

EDG Users’ Guidehttp://marianne.in2p3.fr/datagrid/documentation/EDG-User-Guide-2.0.pdf


Summary & Conclusions

We explained the main attributes to create and submit basic JDL on the GRID.

Jobs which interacts with the RLS through the JDL

Jobs which interacts with the RLS using the lcg-* commands.


Questions…

EGEE is a project funded by the European Union under contract IST-2003-508833 EGEE Tutorial Turin -...

Documents

Transcript of EGEE is a project funded by the European Union under contract IST-2003-508833 EGEE Tutorial Turin -...