Overview of GridSAM and its Deployment

Web: www.omii.ac.uk Email: [email protected]

Overview of GridSAM and its Deployment

Steve Crouch, [email protected]

29/07/09


Contents

• What is GridSAM and what does it offer?o For resource owners and end users

• Example scenario• Packaging options• The job submission pipeline• Open development• Installation and configuration• Future developments


GridSAM Overview

• What is GridSAM to the resource owners?o A web service to uniformly expose a computational

resource• Condor• Portable Batch Scheduler (PBS)• Globus• Sun GridEngine• Platform Load Sharing Facility (LSF)• Single machine through Fork or SSH

o Acts as a client to these resources• What is GridSAM to end-users?

o A means to access computational resources in an open standards-based uniform way

o A set of end-user command-line tools and client-side APIs to interact with GridSAM Web Services

• Submit and monitor compute jobs• Cross-protocol file transfer (gsiftp, ftp, sftp, WebDav, http, https,

soon SRB, iRODS) via Commons-VFS (http://sourceforge.net/projects/commonsvfsgrid)


For Resource Owners…

ComputationalResourceManager

ComputationalResourceManager

DRMDRM

…

One of:PBS (Torque/OpenPBS/PBSPro)LSF, Condor, Sun GridEngine, Globus, Fork

GridSAMService

GridSAMService

X509certificate

Linux Many flavours:RHEL 3,4,5, Fedora 7,8, Scientific Linux 4Java: JDK 1.5.0+

Linux + JavaLinux + Java

Tomcat/AxisTomcat/Axis

Tomcat: 5.0.23, 5.0.28, 5.5.23Axis: v1.2.1

Persistence provided by one of:Hypersonic, PostgreSQL, or existing MySQL


For End-Users…

GridSAMService

GridSAMService

JSDL

HTTPS/HTTP

WS-Security:X509User/Password

Service InterfaceService Interface

Any/all of:GridSAM native interface, OGSA-BES v1.0, HPC Basic Profile v1.0

…

MyProxy (for Globus/GridFTP)

MyProxy (for Globus/GridFTP)

Globus-

styl

e

Proxy

Certif

icat

e

+ MyProxycredentials

Windows/Linux + Java

Windows/Linux + Java

GridSAM ClientGridSAM Client

Many flavours:RHEL 3,4,5, Fedora 7,8, Debian, Ubuntu, Scientific Linux 4, Windows XP, Windows VistaJava: JDK 1.5.0+

AxisAxisX509certificate

Generic BES/HPC Basic ProfileClient

Generic BES/HPC Basic ProfileClient

OSGA-BESOSGA-BESHPC Basic Profile

HPC Basic Profile


Example Scenario - CPOSS ENGAGE Project

CondorCondor …

UCL Legion (PBS Torque + Moab)

UCL Legion (PBS Torque + Moab)

…

UK NGSUK NGS

…

Job Managemen

tSystem

Job Managemen

tSystem

Workflow

WorkflowPortalPortal

GridSAM

GridSAM

GridSAMGridSAM

ActiveBPEL

ActiveBPEL


Package Options: CGT

• Campus Grid Toolkit (CGT) – automated client or server install

• Client/server solution that containso Apache Tomcat/Axis/WSS4j (WS-Security)o GridSAMo Optional - Application Hosting Environment (AHE)o PostgreSQL or HyperSonic database

• Easy, quick installation via further autoconfiguration:o Service:

• Initial security setup for testing• Hooks into existing computational resources

o Client:• Downloadable from existing CGT server and pre-configured for that

server• End-to-end, ‘out of the box’ working examples

• Currently version 1.1.3 (with GridSAM 2.1.6)• Installation videos – see http://www.omii.ac.uk/wiki/CGT


Package Options: Development Kit

• Development Kit – heavily assisted client or server installation

• Service comprised of:o Web Services Container: Tomcat/Axis with optional WS-

Securityo Data persistence: readily-installable PostgreSQL or

HyperSonic databases, or pre-installed MySQLo Configurable Distributed Resource Manager (DRM) system

• Client essentially ‘container’ of libraries and security configuration

• Optionally downloadable and installable components for client and server:

o GridSAM: Job submission and monitoringo AHE: Application Hosting Environmento Grimoires: UDDI + metadata registryo …

• Currently version 3.4.4


GridSAM e.g. with Condor

• A staged event-driven architectureo Submission pipeline is constructed as a network of

stages connected by event queueso Each stage performs a specific action upon incoming

events

Example Pipeline: Condor


Open Community Development

• GridSAM is Open Source, Open Community Development

• GridSAM SourceForge project:

o 99.03% activity, 1 release/month

o SVN source code repositoryo Developer & discuss

mailing lists

http://sourceforge.net/projects/gridsam/


Installation Process – e.g. Development Kit• Pre-requisites:

o Download omii-server-3.4.4.tar from OMII-UK website (Software -> Downloads) & unpack

o Ensure JDK 1.5.0+ is installed & JAVA_HOME set & Java on PATH

o e.g. for PBS• Ensure submit and monitor PBS commands available on PATH• Installable on PBS server node, or elsewhere with ssh access to PBS

server node

• cd to omii-server-3.4.4 unpacked directory and run ./OMIIstackInstall.pl once each for:

1. Database install: provide PostgreSQL version, install location & port

2. Tomcat/Axis container: Tomcat version, install location, database port, test certificate details

3. Database setup for optional components4. Optional components: select GridSAM only

• Install complete


Config. Process – e.g. Development Kit• For example, for PBS (any flavour):

o GridSAM is a PBS client for submission and monitoring

o Edit usermap_config.xml and supply DN -> local user account mappings

o Edit batch_config.xml for local PBS configuration• PBS batch queue name to use• User with sudo privileges to enable job submission as

submitting usero Ensure PBS qstat, pbsnodes, qnodes, qsub on PATH

when starting containero Restart container

• For CGT, you specify install requirements in an easily changed specification file prior to installation


Planned Future Developments

• For end-users:o Refactored documentation and improved OGF

standards coverageo Full support for JSDL DataStaging credential

specificationo Full support for JSDL Resource selection across PBS,

Condor & Fork DRMso JSDL Parameter Sweep Extensiono Support for SRB and iRODS

• For resource owners:o Packaging option as a standalone, manually

configurable web archive (WAR) fileo Job submission through multiple remote ssh accounts

• Direct PBS deployment throughout NGS sites

Overview of GridSAM and its Deployment

Documents

Transcript of Overview of GridSAM and its Deployment