Grid Computing Grid Computing at PSNCat PSNC

Jarosław NabrzyskiPoznań Supercomputing and Networking Center (PSNC) andInformation Sciences Institute, Poznan University of Technologyemail: naber@man.poznan.pl

www.GridLab.org

Talk OutlineTalk Outline

What is Grid Computing about? Background - Polish Information

Infrastructure PIONIER Program

Poznan Supercomputing and Networking Center

PSNC’s Grid Research International collaboration IST projects

The future (6FP)

The Grid ProblemThe Grid Problem

Flexible, secure, coordinated resource sharing among dynamic collections of individuals, institutions, and resources

From “The Anatomy of the Grid: Enabling Scalable Virtual Organizations”

Enable communities (“virtual organizations”) to share geographically distributed resources as they pursue common goals -- assuming the absence of… central location, central control, omniscience, trust relationships.

The Grid - The Grid - Not Not aa New Idea New Idea

Late 70’s – Networked operating systems Late 80’s – Distributed operating systems Early 90’s – Heterogeneous computing Mid of 90’ - metacomputing (C.C. Catlett and L.

Smarr) Transparent access to heterogeneous and geographically

distributed computational resources, Giving a user access to the required computing power

Metacomputing -> then the Grid: 1999, C. Kesselman and I. Foster

How are Grids Different?How are Grids Different?

Autonomy Heterogeneity Focus on the user

These three differences create many of the problems but also make the system much more usable than its predecessors

Elements of the ProblemElements of the Problem

Resource sharing Computers, storage, sensors, networks, … Sharing always conditional: issues of trust,

policy, negotiation, payment, … Coordinated problem solving

Beyond client-server: distributed data analysis, computation, collaboration, …

Multi-institutional virtual orgs Community overlays on classic org structures Large or small, static or dynamic

PSNCPSNC

Founded in 1993 by the State Committee for Scientific Research

One of the 5 supercomputing centres in Poland

4 departments: APPS, HPC, NET, SERVICES

Staff: 75 people

PSNC’s Main ActivitiesPSNC’s Main Activities

POZMAN - POZMAN - metropolitanmetropolitan area area network operatornetwork operator Operator of the Polish National Research Network Operator of the Polish National Research Network

POL-34/155 (now PIONIER network)POL-34/155 (now PIONIER network) Management of high performance computing Management of high performance computing

resources, network file servers, archive systemsresources, network file servers, archive systems Development and deployment of new broadband Development and deployment of new broadband

network servicesnetwork services GRID computingGRID computing

Resource managementResource management Grid Information ServicesGrid Information Services Grid Application Development ToolkitsGrid Application Development Toolkits

PSNC’s international PSNC’s international activitiesactivities

European Grid Forum (EGrid) (http://www.egrid.org/) 1st EGrid Workshop in Poznań (April 2000) EGrid Testbed – European Grid Forum Testbed

(demo at SC’2000) Global Grid Forum

(http://www.gridforum.org/) GGF Steering Group (J. Nabrzyski, AD) 1st Global Grid Forum (Amsterdam 2001) –

Program Chair 5th GGF to be held in Edinburgh, July 2002

EGrid Testbed Cactus worm Dynamic grid

computing 12 sites Presented at

SC2000

Cactus warmDynamic grid computingRemote steering and visualizationMobile user access35 supercomputers at 25+ sites10+ teraflops!

EC fundedEC funded Grid project space Grid project space

GRIDLAB

GRIA

EGSO

DATATAG

CROSSGRID

DATAGRID

Applications

GRIP EUROGRID

DAMIENMiddleware & Tools

Underlying Infrastructures

ScienceIndustry / business

- Links with European National efforts

- Links with US projects (GriPhyN, PPDG, iVDGL,…)

PSNC

CYFRONETPSNCICM

35M35MЄЄ ECEC-funding for above initiatives-funding for above initiativesThe volume of EC projects in which Poland The volume of EC projects in which Poland participates is participates is 13M13MЄЄ

ENACTSPSNC

ICMICM

GridLab – A Grid Application GridLab – A Grid Application Toolkit and TestbedToolkit and Testbed

PSNC - Project Coordinator 11 institutions/vendors from Europe, 3

from the US, The total budget:5.086 Euro 3 years project, started January 2002 Contact: Jarek Nabrzyski – Project

Manager, naber@man.poznan.pl

GridLab ParticipantsGridLab Participants

Poznań Supercomputing and Networking Center, Poland (PSNC) – Project Coordinator, Albert Einstein Institute (AEI), Germany, SZTAKI, Hungary Masaryk University, Czech Republik, University of Lecce, Italy Konrad Zusse Centrum (ZIB), Germany University of Wales, UK Vrije University, Netherlands, Sun Microsystems Gridware GmbH, Germany Compaq EMEA, France University of Athens, Greece Argonne National Laboratory (Ian Foster’s group) ISI (Carl Kesselman’s group) University of Wisconsin (Miron Livny’s group) Several subcontracting sites provide additional testbed resources Many other partners will work unfunded (GGF Apps WG connections)

Grand Challenge Grand Challenge ApplicationsApplications

Black hole simulations, Gravitational wave detection, Other generic applications.

GridLab AimsGridLab Aims

Get Computational Scientists using the “Grid” and Grid services for real, everyday, production work (AEI Relativists, EU Network, Grav Wave Data Analysis, Cactus User Community).

Make it easier for applications to make flexible, efficient, robust, use of the resources available to their virtual organizations.

Dream up, prototype, and test new application scenarios which make adaptive, dynamic, wild, and futuristic uses of resources.

SolutionSolution

“Grid Application Toolkit” or GATProvides a layer between applications and emerging grid technologies. Provides an application developer orientated API, allowing the flexible use of different tools and services, as well as providing protection from developing software.

“GridLab Testbed/VO”Diverse controllable environment for developing and testing applications and tools, software maintained by people who know it.

End Users

GAT Tool Developers

Grid Infrastructure

Developers

GAT-APIDevelopers

What GridLab Isn’t …What GridLab Isn’t …

Don’t want to develop low level Grid Infrastructure

Don’t want to repeat work which has already been done (want to incorporate and assimilate it … Globus APIs, OGSA, ASC Portal (GridSphere/Orbiter), GPDK, GridPort, DataGrid, GriPhyn)

Architektura GridLabArchitektura GridLab

Grid Grid ScenarioScenario QuickTime™ and a

Photo decompressorare needed to see this picture.

We see something,but too weak.

Please simulateto enhance signal!

WashU Potsdam

Thessaloniki

OK! Resource EstimatorSays need 5TB, 2TF.Where can I do this?

RZG

NCSA

1Tbit/secHong KongQuickTime™ and a

Photo decompressorare needed to see this picture.

Resource Broker:LANL is best match…

Resource Broker:NCSA + Garching

OK, but need 10Gbit/sec…

LANLNow..

LANL

Workpackages Workpackages

WP1: Grid Application Toolkit (AEI) This is a key component of GridLab - link between Grid

middleware and applications, usable by any conforming application or middleware component. Requiring input from, and connecting to, most other workpackages and components.

WP2: Cactus Grid Application Toolkit (AEI) Provides an extended GAT interface for Cactus, a very

general toolkit framework supporting different Grid applications, from astrophysics to chemical engineering. Cactus will be one of the primary application drivers for the GAT, and the project generally.

WP3: Work-flow Application Toolkit (CARDIFF) Will develop Grid capabilities for a widely used dataflow

programming environment, Triana, used in gravitational wave and other data analysis areas.

Workpackages (cont.)Workpackages (cont.)

WP4: Grid Portals (AEI) Will be highly application driven, aimed at providing uniform,

flexible and intuitive user access to Grid resources from anywhere, as well as administration tools for maintaining a Grid environment.

WP5: Testbed management (MU) Will administrate and maintain an active development testbed

across roughly a dozen EU sites (leveraging the work of the EGrid), deploying technologies as they are developed by the project. This workpackage will also coordinate with sites in the USA-based NCSA Alliance and others to test and develop interoperability.

WP6: Security (PSNC) Will develop the required security mechanisms and will ensure

the integration of all the technologies developed under other WPs, taking into account the various local security requirements and state of the art solutions.

Workpackages (cont.)Workpackages (cont.)

WP7: Adaptive Application Components (VU) develops a set of components and APIs to be plugged into the

toolkit, for example to take monitoring information and implement basic techniques for short-term forecasting and behavior adaptation/optimization.

WP8: Data Handling and Visualization (ZIB) will provide Grid aware techniques for data management, analysis,

and visualization, needed especially for applications that make use of multiple sites in a dynamic, time dependent manner, leaving data unpredictably scattered across the Grid.

WP9: Resource Management (PSNC) will develop resource need estimators, resource brokers, and other

tools, for both Grid users and the applications themselves to make intelligent decisions about which Grid resources should be used at any instant in the lifetime of a simulation.

Workpackages (cont.)Workpackages (cont.)

WP10: Information Services (ISUFI) will extend existing Grid middleware toolkits with

dynamic features needed by applications to select appropriate Grid resources and to provide simulation information to collaborative user groups.

WP11: Monitoring (SZTAKI) will develop new components that will fit in the

general Grid monitoring architecture to support application steering, adaptive monitoring, and automatic analysis and prediction of performance data.

WP12: Access for mobile users (PSNC, ZIB) will develop and test Grid access and monitoring

technologies through a variety of mobile devices,

Workpackages (cont.)Workpackages (cont.)

WP13: Information Dissemination and Exploitation (PSNC) will ensure the active dissemination of the project results

through a variety of channels, including active participation in international organizations (e.g. GGF), co-development with other Grid projects in the USA and EU, participation in international conferences, training programs, instruction of GridLab technologies into various communities, and introduction into the commercial vendor world.

WP14: Project Management (PSNC) day-to-day scientific, financial and administrative

management of the project, including careful orchestration and monitoring of work across groups, major project decisions, liaisons with external projects and with the international advisory board, reporting

6FP, Grids and PSNC6FP, Grids and PSNC

Europe needs the Distributed EU-wide Supercomputing Facility,

PSNC (and others) has already started to promote this idea,

EoI in preparation, Strong co-operation with colleagues

from UK, Project proposal (70-100 MEuro) in

preparation

Computational Needs not Computational Needs not MetMet

Great scientific and engineering talent in EU Many EU programs now beginning, with training

mission, but: No EU-wide facilities No Collaborative Infrastructure No HPC Training (except for do-it-yourself) Rely on US connections

What do others do?? In many cases, they do toy problems EU science and engineering fall behind Germany, UK, France are slight exceptions, but no

large scale Many countries (mainly NAS, but not only) simply cut

off from modern science

Our Vision for the EU FacilityOur Vision for the EU Facility

Science and Engineering Drivers Expertise Centers (CFD, Astrophysics,

Computational Biology, Aerospace, etc.) Sheparding/building communities across Europe

Present science/engineering calculations require TB/Tflop/sec, but just scratching the surface! Need much more!

Enabling new science Computational Science Drivers

Need Grid Software infrastructure Cycles, networks, visualization centers

Partnerships with existing centers, networks, etc

Vision of the FacilityVision of the Facility Distributed Nature

Pushes networking technologies through apps Works to integrate European efforts, scientific,

cultural, etc. East and West at 100Gb/sec! Multi-lambda backplane (GEANT?)

Unique EU facility in the HPC world Leapfrog Teragrid with Big Machine, but also Draw on unique, more diverse EU research

strengths Beginning of a Worldwide Grid Effort

Unique challenges to integrating diverse EU grids Already strong support from US, NSF, DOE

principals Scientists/engineers know they have

expanding facility over coming decades (can affect their projects)

Our VisionOur Vision

Major Center

Major Center

Satellite

Satellite

Satellite

Satellite

Satellite

Satellite

More info / summaryMore info / summary

www.gridlab.org www.man.poznan.pl www.kbn.gov.pl