Status of NA4 Generic Applications Roberto Barbera NA4 Generic Applications Coordinator

EGEE is a project funded by the European Union under contract IST-2003-508833

Status of NA4 Generic Applications

Roberto Barbera

NA4 Generic Applications Coordinator

NA4 Meeting, CERN, 13.10.2004

www.eu-egee.org

NA4 Meeting, CERN, 13.10.2004 - 2

Outline

• Status of GILDA

• New demonstrative applications in GILDA

• Status of “official” Generic Applications

• How to move from GILDA to EGEE-0

• Summary

NA4 Meeting, CERN, 13.10.2004 - 3

The present GILDA Testbed(https://gilda.ct.infn.it/testbed.html)

NA4 Meeting, CERN, 13.10.2004 - 4

The present GILDA sponsors

NA4 Meeting, CERN, 13.10.2004 - 5

The GILDA Tutorials/Demonstrations (https://gilda.ct.infn.it/tutorials.html)

• Catania, 4-8 October 2004, home page, agenda• Vilnius, 5-6 October 2004, agenda• London, 6 October 2004• Madrid, 6-7 October 2004, agenda• Heidelberg, 11-14 October 2004• CERN, 16 October 2004 • Prague, 26 October 2004, home page • Warsaw, 4-6 November 2004, home page, agenda• The Haague, 15-17 November 2004• Merida, 15-19 November 2004 • Bochum, 7-10 December 2004• Istanbul, 9-10 December 2004 • Prague, 13 December 2004

NA4 Meeting, CERN, 13.10.2004 - 6

The GILDA Video Tutorials(https://gilda.ct.infn.it/video.html)

• How to join GILDA

• Certificate: conversion and manipulation

• The GILDA Grid Demonstrator

• The GILDA Grid Tutor: how to install it

• The GILDA Grid Tutor: how to use it

NEW !

NA4 Meeting, CERN, 13.10.2004 - 7

New functionalities of GILDA

• Complete support for MPI jobs

• Complete support for DAG jobs DAG UI: grid-demo1.ct.infn.it DAG RB: grid007.ct.infn.it CMS example demonstrated at the last EDG Review successfully reproduced Integration of DAG’s in TRIANA re-started

• SciLab (http://www.scilab.org, a MathLab clone) installed on all sites and successfully tested

• GEANT4 installed on all sites and successfully tested

• GNU G95 Fortran 90/95 compiler (http://www.g95.org) under test in Catania. Will be available on all sites in a few days along with a series of working examples

• Complete support for DGAS accounting system foreseen in a few weeks

NA4 Meeting, CERN, 13.10.2004 - 8

MPI jobs: site configuration recipe(valid for PBS as LRM)

• Edit /etc/ssh/sshd_config and add the following lines at the end:

HostbasedAuthentication yes

IgnoreUserKnownHosts yes

IgnoreRhosts yes

• Copy the file /etc/ssh/shosts.equiv from CE on all WN’s.

• On the CE generate an updated version of /etc/ssh/ssh_known_hosts by running /opt/edg/sbin/edg-pbs-knownhosts.

• Copy that file into all your WN’s.

• Restart the server with: /sbin/service sshd restart

NA4 Meeting, CERN, 13.10.2004 - 9

MPI jobs: JDL example(see http://grid-it.cnaf.infn.it/index.php?mpihowto&type=1)

Type = "Job";

JobType = "MPICH";

NodeNumber = 4;

Executable = "MPItest.sh";

Arguments = "cpi 4";

StdOutput = "test.out";

StdError = "test.err";

InputSandbox = {"MPItest.sh","cpi"};

OutputSandbox = {"test.err","test.out","executable.out"};

Requirements = other.GlueCEInfoLRMSType == "PBS" || other.GlueCEInfoLRMSType == "LSF";

NA4 Meeting, CERN, 13.10.2004 - 10

MPI jobs: execution script example (1/2) (see http://grid-it.cnaf.infn.it/index.php?mpihowto&type=1)

#!/bin/sh # this parameter is the binary to be executed EXE=$1 # this parameter is the number of CPU's to be reserved for parallel execution CPU_NEEDED=$2 # prints the name of the master node echo "Running on: $HOSTNAME" echo "*************************************" if [ -f "$PWD/.BrokerInfo" ] ; then TEST_LSF=`edg-brokerinfo getCE | cut -d/ -f2 | grep lsf` else TEST_LSF=`ps -ef | grep sbatchd | grep -v grep` fi if [ "x$TEST_LSF" = "x" ] ; then # prints the name of the file containing the nodes allocated for parallel execution echo "PBS Nodefile: $PBS_NODEFILE" # print the names of the nodes allocated for parallel execution cat $PBS_NODEFILE echo "*************************************" HOST_NODEFILE=$PBS_NODEFILE else # print the names of the nodes allocated for parallel execution echo "LSF Hosts: $LSB_HOSTS" # loops over the nodes allocated for parallel execution HOST_NODEFILE=`pwd`/lsf_nodefile.$$ for host in ${LSB_HOSTS} do echo $host >> ${HOST_NODEFILE} done fi

NA4 Meeting, CERN, 13.10.2004 - 11

MPI jobs: execution script example (2/2) (see http://grid-it.cnaf.infn.it/index.php?mpihowto&type=1)

echo "*************************************" # prints the working directory on the master node echo "Current dir: $PWD" echo "*************************************" for i in `cat $HOST_NODEFILE` ; do echo "Mirroring via SSH to $i" # creates the working directories on all the nodes allocated for parallel execution ssh $i mkdir -p `pwd` # copies the needed files on all the nodes allocated for parallel execution /usr/bin/scp -rp ./* $i:`pwd` # checks that all files are present on all the nodes allocated for parallel execution echo `pwd` ssh $i ls `pwd` # sets the permissions of the files ssh $i chmod 755 `pwd`/$EXE ssh $i ls -alR `pwd` echo "@@@@@@@@@@@@@@@" done # execute the parallel job with mpirun echo "*********************************" echo "Executing $EXE" chmod 755 $EXE ls -l mpirun -np $CPU_NEEDED -machinefile $HOST_NODEFILE `pwd`/$EXE > executable.out echo "*********************************"

NA4 Meeting, CERN, 13.10.2004 - 12

New demonstrative applications on GILDA

• GATE (demonstrated at the last GEANT4 Workshop by Lydia Maigne)

• MPI examples

• Raster-3D (chemical application: 3D modelling/rendering of molecules)

• SciLab examples (general application: data presentation using SciLab, the MathLab clone)

• GEANT4 examples (general application: examples of GEANT4 applications)

NA4 Meeting, CERN, 13.10.2004 - 13

MPI example

NA4 Meeting, CERN, 13.10.2004 - 14

Raster-3D example

NA4 Meeting, CERN, 13.10.2004 - 15

SciLab example

NA4 Meeting, CERN, 13.10.2004 - 16

GEANT4 example

NA4 Meeting, CERN, 13.10.2004 - 17

Status of Generic Applications(Earth Science)

External presentation

NA4 Meeting, CERN, 13.10.2004 - 18

Status of Generic Applications(Computational Chemistry)

Scientific area Computational chemistry

Contact Antonio Laganà, Dipartimento di Chimica, University of Perugia, Italy, [email protected]

VO manager Osvaldo Gervasi, [email protected]

VO services VO administration, Resource Broker and Replica Location Service located at CNAF

Application(s) deployed “A priori” atomic and molecular simulations.

Status on GILDA First cluster in Perugia added to GILDA next week. First brainstorming meeting scheduled for october 18-19 at CNAF.

Status on EGEE-0

Main requirements Deployment of licensed software on the grid infrastructure. Commercial software will initially be deployed on the Perugia cluster.

Main issues related to deployment

NA4 Meeting, CERN, 13.10.2004 - 19

Status of Generic Applications(Astroparticle Physics)

Scientific area Astro-particle Physics

Contact Harald Kornmayer, FZK, Germany, [email protected]

VO manager Harald Kornmayer, KZK, Germany, [email protected]

VO services VO administration, Resource Broker and Replica Location Service located at SARA

Application(s) deployed MAGIC telescope Montecarlo simulations

Status on GILDA Realisation of RPM(s) of Magic Montecarlo in progress.

Status on EGEE-0

Number of nodes accessible on EGEE infrastructure

Desired infrastructure usage

Main requirements

Main issues related to deployment

NA4 Meeting, CERN, 13.10.2004 - 20

Move from GILDA to EGEE-0(UI points to another default RB)

• In the file /var/obj/conf/server/source/ of your LCFGng server add the needed information in the lines:

#ifdef UI_RESBROKER #undef UI_RESBROKER #define UI_RESBROKER < RB to point to > #endif …..…..#ifdef UI_LOGBOOK #undef UI_LOGBOOK #define UI_LOGBOOK https://< RB to point to >:7846 #endif

NA4 Meeting, CERN, 13.10.2004 - 21

Move from GILDA to EGEE-0(allow a new VO to run on a site – step 1)

• In the file site-cfg.h on your LCFGng server add the needed information in the lines:

#define SE_VO_<vo name>

....................... .......................

#define SA_PATH_<vo name> <vo name>

....................... .......................

#define WN_AREA_<vo name> /opt/exp_software/<vo name>

NA4 Meeting, CERN, 13.10.2004 - 22

• In the file ComputingElement-cfg.h on your LCFGng server add the needed information in the lines:

#ifdef SE_VO_<vo name>

EXTRA(ceinfo.SUPPORTEDVOS) <vo name>

#endif

....................... .......................


EXTRA(dirperm.ents) <vo name>sgmce

dirperm.path_<vo name>sgmce SW_PATH/<vo name>

dirperm.owner_<vo name>sgmce <vo name>sgm:<vo name>

dirperm.perm_<vo name>sgmce 0775

dirperm.type_<vo name>sgmce d

#endif


NA4 Meeting, CERN, 13.10.2004 - 23


• In the files: ComputingElement-novoms-cfg.h StorageElementClassic-cfg.h StorageElement-cfg.h

on your LCFGng server add the needed information in the line:

#include "poolaccounts-<vo name>-cfg.h"

where poolaccounts-<vo name>-cfg.h looks like


EXTRA(poolaccounts.usernames) <vo name>sgm <vo name>001 \

<vo name>002 ……<vo name>100

EXTRA(poolaccounts.nolock) <vo name>sgm

#endif

NA4 Meeting, CERN, 13.10.2004 - 24


• In the file flatfiles-dirs-SECLASSIC-cfg.h on your LCFGng server add the needed information in the lines:


EXTRA(dirperm.ents) <vo name>

dirperm.path_<vo name> SE_FILE_CACHE_AREA/<vo name>

dirperm.owner_<vo name> root:<vo name>

dirperm.perm_<vo name> 0775

dirperm.type_<vo name> d

#endif

NA4 Meeting, CERN, 13.10.2004 - 25


• In the file lcginfo-seclassic-cfg.h on your LCFGng server add the needed information in the lines:


#define PATH_<vo name> <vo name>:SA_PATH_<vo name>

#define PATH_DYN_<vo name> <vo name>:CE_CLOSE_SE_MOUNTPOINT/SA_PATH_<vo name>

#else

#define PATH_<vo name>

#define PATH_DYN_<vo name>

#endif

.......................

#ifdef SE_DYNAMIC_CLASSIC

+lcginfo.args_classic SE_HOSTNAME PATH_DYN_ALICE PATH_DYN_ATLAS PATH_DYN_CMS PATH_DYN_LHCB PATH_DYN_DTEAM PATH_DYN_<vo name>

#endif

.......................

EXTRA(lcginfo.value_GlueSARoot) PATH_<vo name>

EXTRA(lcginfo.names_GlueSAAccessControlBaseRule) <vo name>

EXTRA(lcginfo.names_GlueSAPolicyFileLifeTime) <vo name>

.......................


lcginfo.prefix_GlueSAAccessControlBaseRule_<vo name> PATH_<vo name>

lcginfo.value_GlueSAAccessControlBaseRule_<vo name> <vo name>

lcginfo.prefix_GlueSAPolicyFileLifeTime_<vo name> PATH_<vo name>

lcginfo.value_GlueSAPolicyFileLifeTime_<vo name> permanent

#endif

NA4 Meeting, CERN, 13.10.2004 - 26


• In the file mkgridmap-cfg.h on your LCFGng server add the needed information in the lines:


EXTRA(mkgridmap.groups) <vo name> mkgridmap.uri_gilda ldap://<vo server name>:<vo server port>/ou=<vo name ou>,o=<vo name o>,c=<vo name c> mkgridmap.user_<vo name> .<vo name>

#endif

Example: ldap://grid-vo.cnaf.infn.it:10389/ou=Testbed-gilda,o=gilda,c=it

NA4 Meeting, CERN, 13.10.2004 - 27


• In the file se-cfg.h on your LCFGng server add the needed information in the lines:


EXTRA(se.vos) <vo name>

#endif

NA4 Meeting, CERN, 13.10.2004 - 28


• In the file seinfo-cfg.h on your LCFGng server add the needed information in the lines:


EXTRA(seinfo.vos) <vo name>

seinfo.SARoot_<vo name> <vo name>:/<vo name>

seinfo.SAPolicyMaxFileSize_<vo name> 10000

seinfo.SAPolicyMinFileSize_<vo name> 1

seinfo.SAPolicyMaxData_<vo name> 1000000 seinfo.SAPolicyMaxNumFiles_<vo name> 10000 seinfo.SAPolicyMaxPinDuration_<vo name> 10000

#endif

NA4 Meeting, CERN, 13.10.2004 - 29


• In the file user-edguser-cfg.h on your LCFGng server add the needed information in the lines:

#ifdef SE_VO_<vo name> EXTRA(auth.usersuppgroups_edguser) <vo name>

#endif

NA4 Meeting, CERN, 13.10.2004 - 30


• In the file voenv-cfg.h on your LCFGng server add the needed information in the lines:


EXTRA(voenv.vo) <vo name>

voenv.swroot_<vo name> /opt/<vo name>

#endif

NA4 Meeting, CERN, 13.10.2004 - 31

• In the file vomswmgr-dirs-cfg.h on your LCFGng server add the needed information in the lines:


EXTRA(dirperm.ents) <vo name>sgm

dirperm.path_<vo name>sgm INFO_PATH/<vo name>

dirperm.owner_<vo name>sgm <vo name>sgm:<vo name>

dirperm.perm_<vo name>sgm 0755

dirperm.type_<vo name>sgm d

#endif


NA4 Meeting, CERN, 13.10.2004 - 32

Summary

• GILDA testbed is expanding and it is now an inter-continental facility

• Some applications’ requirements in terms of commercial software (MathLab, F90) are being addressed in GILDA using free/open source clones (SciLab, G95)

• The portfolio of demonstrative applications is growing every day and can stand very nice and complex demonstrations

• Official Generic Applications are doing well and they are on track for the First Review

• The technical procedure to move from GILDA to EGEE-0 is very simple. Accepting a new VO on a given site takes just a few minutes (just 12 steps to the Heaven )

• We envisage to deploy the accepted Generic Applications on EGEE-0 by the end of the year (to be in time with DNA4.3.1)

Status of NA4 Generic Applications Roberto Barbera NA4 Generic Applications Coordinator

Documents

Transcript of Status of NA4 Generic Applications Roberto Barbera NA4 Generic Applications Coordinator