W w w. h p c - e u r o p a. o r g The HPC-Europa project and GridSphere Dawid Szejnfeld e-mail:...
-
Upload
ilene-boyd -
Category
Documents
-
view
216 -
download
0
Transcript of W w w. h p c - e u r o p a. o r g The HPC-Europa project and GridSphere Dawid Szejnfeld e-mail:...
w w w . h p c - e u r o p a . o r g
The HPC-Europa project and GridSphere
Dawid Szejnfeld
e-mail: [email protected]
Poznan Supercomputing and Networking Center (PSNC)
Poland
GridSphere and Portlets workshop, Edinburgh, 4 March 2005
w w w . h p c - e u r o p a . o r g
Outline• HPC-Europa
• JRA2: Single Point of Access• Multi-Grid Interface• Job Submission Interface• Implementation issues• Generic Job Submission Portlet• Status & Next Steps
w w w . h p c - e u r o p a . o r g
HPC-Europa• HPC-Europa - Pan-European
Research infrastructure on High Performance Computing for the Science of 21st century
• Goal: to provide advanced computational services in an integrated way to the European research community
• Budget: ~20 mln EURO• 14 Partners across Europe• Project activities
– Transnational Access Programme– Networking Activities– Joint Research Activities (JRA1,
JRA2)
w w w . h p c - e u r o p a . o r g
JRA2: Single Point of Access• Motivation
– To provide a uniform access to resources of all centers, transparently and regardless of user physical location
• Main objectives– ease of use,– improving availability,– global optimization of resources utilization
• To achieve these goals– JRA2 builds a HPC-Europa portal providing uniform, flexible and
intuitive user access to Grid resources from anywhere– Develops and/or adapts tools and services needed to establish full
operational Grid environment– Will develop a meta-broker responsible for users’ access to all
consortium’s resources
w w w . h p c - e u r o p a . o r g
HPC-Europa SPA
Grid Portal / Grid Resource Broker
HPC centerHPC center HPC center
HPC center
Grid Middleware AGrid Middleware B Grid Middleware C Grid Middleware D
VO
w w w . h p c - e u r o p a . o r g
Middleware in HPC Centers
OpenPBS LSFSGE
Pre-WS Globus 3.2GRMS (GridLab Resource Management System)
PSNC
CondorGRIA (Grid Resources for Industrial Applications)
NTUA
LoadLevelerSGE
Globus 3.2JOSH (JOb Scheduling Hierarchically)
EPCC
LoadLevelerOpenPBS
NQS
Globus 3.2eNANOS (Grid Resource Broker)
CEPBA
OpenPBSERS-NQSPBS Pro
UNICORE-
HLRS
LoadLevelerOpenPBS with MAUI
UNICORE-
CINECA
LRMSUnderlying Grid technology
High-level middlewareCenter
w w w . h p c - e u r o p a . o r g
GridSphere• We have decided to use the GridSphere as a portal framework for the SPA
on the basis of Grid portals' evaluation• The main reasons of our choice are as follows:
– portlet-based & JSR 168 compliant– experience in Grids
• useful tools, services, and core & Grid portlets
• pluggable access to Grid services using the Portlet Services concept
• abstract & generic Grid Portlets model
– good support & documentation– relatively straightforward to use (customization mechanism and a
flexible easy to use interface for a wide range of end users)– open source
w w w . h p c - e u r o p a . o r g
Multi-Grid Interface (MGI)• Extension of the GridSphere’s Grid Portlets concept by
– Dynamic loading of multiple plugins for a single interface – Interface based on the GGF JSDL and analysis of functionality of
several Grid systems• Definition of the common interface
– For each functionality, e.g. job submission, resource information etc.– Based on standards where possible (e.g. JSDL)– Taking into account both gathered requirements and the functionality of
Grid middleware deployed in HPC centers• Capability check
– description of implemented capabilities in the form of the XML document– used to disable not available options (in portal) or to select sites which
can accept given job description (meta-broker)
w w w . h p c - e u r o p a . o r g
Portal layers
PortletsPortlets
Portlet Components
Portlet Services
Service Plugin
Service Plugin
Service Plugin
Presentation Layers
Business Logic
Portlet Resources
Portlet Resources
Portlet Resources
w w w . h p c - e u r o p a . o r g
MGI Architecture - Job Submission View
• Components of a common job submission interface
– JSS - Job Submission Service
– JSSI - Job Submission Service Interface
– JSP - Job Submission Plugin
– JSPI - Job Submission Plugin Interface
• Other common interfaces
• Auxiliary components
w w w . h p c - e u r o p a . o r g
Auxiliary Components• Information System
– Central information indexing service
– Provides a list of available HPC centers, URLs of Grid middleware, etc.
• Application Repository– Logical names, meta-data, binaries compiled for various systems,
required environment (preinstalled libraries, environment variables, etc.), possible configurations & parameters
• Data management system– Management of logical file names
– Goal of the HPC-Europa NA3
• Plugin Factory– Initializes and loads needed plugins
w w w . h p c - e u r o p a . o r g
Job Submission Interface (JSI)• Job submission operations
– submitJob (UNICORE, GRMS, JOSH, eNANOS, GRIA)
– cancelJob (UNICORE, GRMS, JOSH, eNANOS, GRIA)
– getJobInfo (UNICORE, GRMS, JOSH, eNANOS, GRIA)
– findResources (UNICORE, GRMS, eNANOS)
– submitJobToBestResource (GRMS, JOSH, eNANOS, GRIA)
– findBestResource (JOSH)
• Data elements descriptions– Job Description XML schema– Job Information XML schema
• Auxiliary operations– getMiddlewareName – getMiddlewareDesc– getMiddlewareMethods– getMiddlewareSchema
• Job states– GRID_QUEUED, BROKER_QUEUED,
LOCAL_QUEUED, PREPROCESSING, RUNNING, POSTPROCESSING, SUSPENDED, FINISHED, FAILED, CANCELLED
w w w . h p c - e u r o p a . o r g
Job Description
• Based on the GGF Job Submission Description Language (JSDL) specification
• Subset of JSDL defines the common interface
• The specification is still evolving
• Several extensions have been added
w w w . h p c - e u r o p a . o r g
Capabilities Provided by Plugins• Some plugins may implement only a part of the common
interface• Two descriptions must be returned by plugins
– description of available methods– description of data structures (i.e. job description)
• The XML document based on the Job Description schema• Two main kinds of information included
– elements of the document define which parts of interface are implemented by service plugins
– for each element a list of feasible values is defined
• The Job Submission Portlet uses this information to disable appropriate fields and limit lists of feasible values
w w w . h p c - e u r o p a . o r g
Capabilities Provided by Plugins - example
<JobDescription>.....
<Resource id=""><HostName/><CPUArchitecture/><CPUSpeed operator="" units=""/><CPUTime operator=""/><CPUCount operator=""/><ResourceCount operator=""/><PhysicalMemory operator="" units=""/><VirtualMemory operator="" units=""/><OperatingSystem>
<OperatingSystemType/><OperatingSystemVersion/><OperatingSystemDescription/><hpc:Enumerations>
<hpc:OperatingSystemTypeEnumeration>LINUX, FreeBSD, IRIX, Solaris, HPUX, Windows_XP
</hpc:OperatingSystemTypeEnumeration></hpc:Enumerations>
</OperatingSystem></Resource>.....</JobDescription>
• The NetworkBandwidth element is not handled by a plugin (is not translated to the system-specific job description language)
• The list of feasible operating systems (values of the OperatingSystemType element) is limited to a few most common systems
w w w . h p c - e u r o p a . o r g
Implementation issues – JSR 168• Intention to create portable web application• Lack of services in the JSR 168 specification
– Gridsphere JSR services implementation is not compatible with spec
– Solution - external mechanism of creating services
• Spring – application framework– Easy to use (spring.jar added to application lib directory)
– portlet-services web app - local and shared services (LocalServiceManager and SharedServiceManager)
– Services as java beans, can be initialized with simple and complex values
• Configuration files– /WEB-INF/portlet-services.xml and shared-portlet-services.xml
w w w . h p c - e u r o p a . o r g
Implementation issues – services
PortletLocalServiceManager
Web app
Local Service
Portlet-services web app
SharedServiceManager
shared-portlet-services.xml
portlet-services.xml
Shared Service
w w w . h p c - e u r o p a . o r g
Generic Job Submission Portlet
w w w . h p c - e u r o p a . o r g
Generic Job Submission Portlet (GRMS)
w w w . h p c - e u r o p a . o r g
Generic Job Submission Portlet - Resource Requirements
w w w . h p c - e u r o p a . o r g
Generic Job Submission Portlet - Resource Requirements(GRMS)
w w w . h p c - e u r o p a . o r g
Status & Next Steps• Status
– Common job submission interface designed
– Job submission portlet and plugin management started
• Next steps– Development and testing of the job submission portlet
– Development and testing of the interoperability of authentication models
– Design interfaces for resource information, accounting, data management and additional tools (e.g. application repository)
– Demonstration planned in June