GDI-Grid Spatial Data Infrastructure Grid Christian Kiehle Andreas Krüger Christopher Kunz OGF22...

GDI-GridSpatial Data Infrastructure Grid

Christian KiehleAndreas KrügerChristopher Kunz

OGF22Feb 26, 2008

GDI-Grid - Spatial Data Infrastructure Grid

Agenda

1. Introduction to the German Grid initiative

2. Introduction to the GDI-Grid – Spatial Data Infrastructure Grid (GDI-Grid) Project

3. Grid-enabling Open Source OGC Webservices

4. Gridification of OGC Webservice Chains

5. Security and Scheduling Challenges

6. Discussion

Christopher Kunz | February 26th, 2008 | Slide 2


Motivation of this session

Project just started

One of the first new community projects in D-Grid

Unsolved challenges ahead

Presenting work in progress

Looking for ideas

Your input is highly appreciated!



The D-Grid

The German Grid initiative

Long-term strategic initiative started in 2005

D-Grid Call 1 (2005 – 2008) Infrastructure development and testing Use by first scientific organizations

D-Grid Call 2 (2007 – 2010) inclusion of several new scientific disciplines

Funding by German Federal Ministry of Education and Research



D-Grid – timeline


2003: first working groups

09/2005: D-Grid 1 – early adopters > 100 organizations > 200 researchers

07/2007: D-Grid 2 – new communities > 200 organizations > 400 researchers

2008: D-Grid 3 Goal: sustainable production grid No more funding – feasible business model Integration of new communities


D-Grid integration structure


Te

xt-G

rid

Me

di-

Gri

d

D-Grid Integration Project

As

tro

-Gri

d

C3

-Gri

d

HE

P-G

rid

GD

I-G

rid

WIS

EN

T

. . . Communities . . .

Access Layer, Portals

Generic Grid Middleware and Grid Services


D-Grid Integration Project

Goal: Establish standardized Grid Infrastructure in Germany

Requirement: support for three Grid middlewares Globus Toolkit gLite UNICORE

Reference installation for all three middlewares Comprehensive documentation Specified product and middleware versions Unified environment that‘s proven to work



D-Grid Integration Project: Site Map

25 facilities Backbone: 8 core facilities 17 additional computing facilities

Vital statistics 10,000 CPU cores 3.3 PB disk-based storage 5.5 PB tape-based storage


Hannover

Bremerhaven

Oldenburg

Göttingen

Potsdam

OberpfaffenhofenFreiburg

Heidelberg

Darmstadt

Marburg

Dortmund

Wuppertal

Köln

Siegen

SanktAugustin

München

Hamburg

Berlin

Dresden

StuttgartKarlsruhe

Jülich

= computing facility= Core D-Grid facility= Backbone interconnect


GDI-Grid

„Geodaten-Infrastruktur-Grid“ - „Spatial Data Infrastructure Grid“

Project funded by German Federal Ministry of Education and Research

funded from 2007/07 until 2010/06

Strong focus on sustainable development Public-private partnership 12 principal partners

8 scientific institutions 4 industry partners

Biggest community project within D-Grid 2 3M € budget, 15 funded researchers



GDI-Grid – Project Goals

Optimization of SDI and GIS applications to… Satisfy rapidly increasing demand for computing power Improve accessibility of decentralized storage and compute resources Integrate basis technologies of both application fields …especially GT4 and OGC

Enable grid resources to seamlessly process geospatial data

Development of generic services for… Integration Fusion Management Processing

…of geospatial data on the D-Grid infrastructure




GDI-Grid – Project Partners


GDI-Grid – Scenarios and Proof of Concept

Proof of Concept with three distinct scenarios Highly relevant for practical use Large range of GIS applications covered High need for processing power Need for high responsivity Conventional GIS applications already available

Scenarios mandated by EU directives



GDI-Grid - Scenarios

Flood simulation Based on sensor data / empirical evidence Flood prediction Creation of flood maps

Noise propagation Simulation of noise propagation in cities Large-scale simulation in regular intervals Highly detailed simulation setting Relevant for city planning

Emergency routing Routing for disaster emergency services Alternative routes (road blocks, traffic jam) High responsivity and adaptivity Utilization of sensor data in routing process


Grid-enabling Open Source OGC Webservices

Dr. Christian Kiehle, lat/lon GmbH<[email protected]>


Dr. C. Kiehle | February 26th 2008 | Slide 15

Agenda

• deegree – Open Source Framework for Spatial Data Infrastructures (SDI)

• Challenges for Grid-enablement

• User Requirements

• Conclusions and Outlook



www.deegree.org

• .org Joint-Project:

University of Bonn, Department of Geography, Working Group Geo Information Systems (GIS), Germany.

lat/lon GmbH, Bonn, Germany. Free Software based on GNU Lesser Public License

• .aim Interoperability Spatial Data Management Spatial Data Infrastructure (SDI) Management

• .how ISO/TC211 models & Open Geospatial Consortium (OGC) interfaces Encapsulation of data sources / Abstraction About 1800 Java classes (core library) + various client technologies

• .now Most comprehensive implementation of OGC specifications OGC Reference implementation for WMS, WCS.

http://www.deegree.org/



Feature Portrayal Model

Aus: Kiehle, C. Lokales abstrakt - Geodaten standardisiert integrieren. IX 12/2006.



OpenGIS® Web Services

• Standardized Interfaces for Storage Metadata Visualization Processing



Web Processing Service

• Interface for Processes Description Execution

• OGC compliant (in contrast to SOAP / WSDL)

• Generic approach• Three interfaces:

GetCapabilities DescribeProcess (processIdentifier) Execute (request)

• Basis for service chaining / orchestration



Spatial Webservices provide...

Aus Kiehle, C. (2006): Business Logic for Geoprocessing of Distributed Geodata. Computers & Geosciences 32(10): 1746-1757.

doi:10.1016/j.cageo.2006.04.002

• ...capabilities to build Spatial Data Infrastructures

Decision Support Early Warning Systems Spatial - / Environmental Information

Systems Homeland Security ...

• ... a “common sense solution”

Interfaces Data Models Application Profiles Best Practices



Agenda







Grid-enablement of OGC WS

• Why Grid-enablement? Large-scale data bases (e.g. country

wide elevation models) Processing of datasets Processing of 3D-/4D data Virtualization Accounting & Billing

• Challenges Adopt technology, keep standards as

they are Hide complexity from end-users /

system integrators The Grid is (just another) computing

environment. --> Should not require a new paradigm for spatial data processing

WSRF not (yet) supported by OGC

• OGC paradigm provides a model of real-world phenomena

Translate model into Grid-compatible communication schemes

OGC Service may be clients on top of grid services

• Possible solutions Encapsulate Grid-middleware Encapsulate grid processes inside

OGC WPS Integrate WSRF into OGC



Grid-Computing vs. OGC WS

Based on Hobona et al. 2007



Architectural Challenges



Architectural Challenges II

• Define Translator Services• OGC WS as Grid-frontend



Agenda







User requirements

• Who is interested in Grid-enabled OGC WS? Customers with (very) large scaled data sets Environmental Scientists End Users?

• Survey by project partner ESRI / conterra

• Resulting requirements Processing capabilities Externalize data storage Virtual Organizations



Agenda







Conclusions & Outlook

© Heier, C.; Kiehle, C. 2007



Questions?

Dr. Christian [email protected]

http://www.lat-lon.de/

GDI-Grid

Grid-enabling Open Source OGC Webservice

Gridification of OGC Webservice Chains

Andreas KrügerProf. Dr. Thomas H. Kolbe

Institute for Geodesy and Geoinformation Science Technische Universität Berlin{krueger,kolbe}@igg.tu-berlin.de


Agenda

Scenario Noise Pollution Simulation

Gridification in Context of Spatial Data

Infrastructure

Categories of Gridification

Orchestration of Services

32


33

Scenario – Noise Pollution Simulation

Noise PollutionSimulation

Noise PollutionSimulation

Requirements on data:

• as input a 3D block model of buildings with 3D Digital Terrain Model

• enrichment with noise emission levels, traffic frequency, …

Problem:

• normally available heterogeneous data sources:

2D area-wide geobasis data 2,5D raster data noise emission data

Required services:

• WCS, WFS for data delivery

• WPS for 3D model generation

• WPS for Noise Pollution Simulation

• WMS for delivery of Noise Pollution Maps


34

Noise Pollution Simulation – OGC Webservices

WPS3D Geodata Generation

WPS3D Geodata Generation

WFS2D Geobasis Data

(e.g. National Cadastre Data)

WFS2D Geobasis Data

(e.g. National Cadastre Data)

WMSNoise

Pollution Map

WMSNoise

Pollution Map

WPSNoise Pollution

Simulation

WPSNoise Pollution

Simulation

WCS2,5D Raster DataLaserscan Data /

Digital Terrain Model

WCS2,5D Raster DataLaserscan Data /

Digital Terrain Model

Chain of OGC compliant Webservices

Spatial Data Infrastructure

WFS

Noise Emission Data

WFS

Noise Emission Data

Grid

Gridification


35

Agenda



Infrastructure




36

• Gridification:

Adaptation of existing applications and services to requirements of a grid environment.

• Goals of gridified Spatial Data Infrastructure applications:

Possibility of combination with other grid services Potential of access from / to non-gridified OGC Webservice High performance for processing large amounts of spatial data

• Requirements:

Adaptation to grid technology

Stateful services

WSDL

Security (Certificates, public / private key)

Maintain interfaces and functionality of OGC Webservice

Maximized utilisation of grid resources

Optimised distribution of data / services for processing spatial data

Harmonised interconnection of services

Gridification in Context of Spatial Data Infrastructure

Grid-Service

OGSI-Interfaces

OGC web service functionality


37

Parallelisation

• Scenario Noise Pollution Simulation:

A good choice: Tiling

Each service of the scenario works on continuous spatial data of wide areas

Each service has object-oriented models as input and output

Problem for harmonised parallelisation between services:

Adjustment calculation at a data join needs calculation of overlapping borders of tiles

Needs on additional overlapping border size differs between several services

Parallelisation of object types

Parallelisation of operations

Tiling

Height() Width() Extrude() Height() Width() Extrude()


38

• Pros: Realisable harmonised

parallelisation between services Allows full utilisation of grid

resources

• Cons: How works the connection

between grid and Spatial Data Infrastructure ?

Non-transparent mapping of OGC Webservice chains in grid environment

No combination of gridified and non-gridified services

Cooperation between services only by using OGSI

• Pros: Realisable harmonised

parallelisation between services Allows for full utilisation of grid

resources OGC compliant service chains

at the Spatial Data Infrastructure side

cooperation between grid and Spatial Data Infrastructure

Enables chains of gridified and non-gridified services

• Cons: Complex realisation of this

solution

• Pros: Orchestration still at OGC

Webservice level in the Spatial Data Infrastructure

For an user same handling of services, since communication only by OGC Webservice interfaces at Spatial Data Infrastructure side

Enables chains of gridified and non-gridified services

• Cons: Autonomous kind of

gridification at each OGC Webservice

Non-harmonised parallelisation of more than one OGC Webservice

OWS1

OWS2

OWS3

OWS1‘

OWS2‘

OWS3‘

Spatial Data Infrastructure and Grid


GridWith Parallelisation

OWS2

OWS1

OWS3

2

3

1

2

3

1

2

3

1


39

Agenda



Infrastructure




40

Categories of Gridification and Notation

OGC Webservice realised as one grid service Parallelisation by redundant execution of same service in

grid environment

OGC Webservice stays a Spatial Data Infrastructure service Calculation processes / handling of datasets distributed on

several instances mapped as grid services After processing distributed dataset joined at each OGC

webservice

OGC Webservice stays a Spatial Data Infrastructure service

Calculation processes / handling of datasets distributed on several instances mapped as grid services

Each instance transfers its part of dataset directly to an instance of another service

• 3) Transcendent Gridification:

OWSn

OWSG

Grid

OWS1

Grid OWSg

• 2) Encapsulated Gridification:

• 1) Gridification by Embedding into a Grid Service:

GSOWS

Grid


41

Agenda



Infrastructure




42

Orchestration of Grid Services

• Set of services with different parallelisation

• Set of known workflows for a scenario

Workflow-Engine

WFS

OWSg

OWSG

OWSG

OWSg OWSg

OWSg

OWSG OWSG

OWSGOWSg OWSg

OWSG

OWSG OWSG

OWSG

Spatial Data Infrastructure Grid

WFS

WPS

WCS

WCS


43

Questions / Contact

Thank you for your attention.

Andreas KrügerInstitute for Geodesy and Geoinformation Science

Technische Universität [email protected]: +49 (30) 314 24409

Thomas H. KolbeInstitute for Geodesy and Geoinformation Science

Technische Universität [email protected]

Phone: +49 (30) 314 23274

Contact:

mailto:[email protected]

mailto:[email protected]


44

Problem:

• High data traffic at centralised Spatial Data Infrastructure services Bottlenecks

• Additional costs because of Split and Join at each OGC Webservice

Grid

GridGrid

Encapsulated Gridification

• OGC Webservice stays a Spatial Data Infrastructure service

• Each service instance is an separate grid service

• 3D Geodata Generation works internally on separate data tiles at the service instances

• Data transfer between two OGC Webservices centralised as one collected dataset inside the Spatial Data Infrastructure

WPSg

3D GeodataGeneration

WFSg

2D Geoobjects

……

WCSg

Laserscan DTM


45

Transcendent Gridification

…

WCSG

DTM 2,5D

Grid

Distributedconsistent 3D-Model

• Data distribution one time at the beginning and handed over the whole service chain

• Geoobjects provided on distributed interfaces as tiles on several grid nodes

• Every instance gets a single tile, proceeds the calculatons and provides the resulting tile by an own interface

• Data split only one time at the beginning, and merging one time at the end of the chain

Bottlenecks and needless data distribution/data merging can be omitted

But: A well harmonised parallelisation exists not in every situation

WCS2WCS2WCS1WCS1…WFS2WFS2WFS1WFS1

WFSG

Geoobjects 2D

WFSnWFSn WCSnWCSn

…WPS2WPS2WPS1WPS1 WPSnWPSn

WPSG

3D Geodata Generation


46

Transcendent Gridification – Noise Pollution Simulation


Grid

WPS1G


WFS2D Geobasis

Data

WFS2D Geobasis

Data

WCS2,5D Raster

Data

WCS2,5D Raster

Data

WMSNoise Pollution Maps

WMSNoise Pollution Maps

WPS2G

Noise Pollution Simulation

WPS2‘

WPS2‘

WPS2‘

WPS1‘

WPS1‘

WPS1‘only possible if

input

parallelisation

signature

WPS2

=

output

parallelisation

signature

WPS1


47

Transcendent Gridification – Noise Pollution Simulation

• Additional Capabilities:

Input / Output signature of parallelisation for orchestration

Access to service endpoints of grid instances

• OGC compliant Publish-find-bind concept

…

Grid

WPS2WPS2WPS1WPS1 WPSnWPSn

…WPS2WPS2WPS1WPS1 WPSnWPSn

WPSG


WPSG

Noise Pollution Simulation

CapabilitiesCapabilities

CapabilitiesCapabilities

find(getCapabilities)bind bind bind

publish

publish

Input / Output signature

Input / Output signature

Spatial Data Infrastructure GridSecurity and Scheduling Challenges

Christopher Kunz<[email protected]>

OGF22Feb 25, 2008



Agenda

Security Challenges GSI vs. traditional models Current situation and concepts in D-Grid Feasibility and scalability of PKI-based AuthN in SDI environments Data storage and handling Possible alternatives

Scheduling and architecture Current situation and concepts in D-Grid Architectural concepts for GDI-Grid Necessary redundancies Where to scale What to scale Compatibility challenges

Discussion


Security – current context in D-Grid

D-Grid supports 3 middlewares: Globus Toolkit 4 gLite 3 UNICORE 5

GDI-Grid will focus on Globus Toolkit 4

Mandatory use of Grid Security Infrastructure (GSI) Single Sign On Delegation of Rights Certificate based authentication, proxy credentials GSI chain must not be broken by any component Credentials must not be transferred over the wire



Security – current situation in SDI

Very heterogenous structure

Proprietary authentication and authorization mechanisms

Data providers are geo scientists, not IT people

Different products require different access mechanisms

No cross-product cross-vendor single sign on

No delegation of rights



Making ends meet…

These contrary architectures need to converge

GSI must be implemented in or in front of GIS services

Paradigm shift in Grid infrastructure?

Paradigm shift in SDI / GIS world?



Authentication / Authorization

At the user entry point, GSI is necessary

SDI / GIS applications do not implement GSI

Is GSI implementation for end user apps feasible?

Different approach: web-based UI GridSphere portal GSI implemented, workflow management possible



Certificate Infrastructure Challenges

Certificates for Grid use are regulated CA has to be conformant to GridPMA (in our case, EuGridPMA) regulations Resource providers must only accept PMA-compliant CAs Subsequently, only certificates issued by those CAs are allowed in the Grid

Commercial CAs are not GridPMA compliant Grid resource providers cannot accept certificates issued by them Little or no interest in GridPMA compliance Certificates issued by commercial vendors are not usable in Grid infrastructures

CAs will issue certificate only to „known“ entities Generally, one CA per country, local Regional Authorities (RA) In our context, DFN (German Science Network) and GridKA operate CAs Both supply certificates to member organizations only



PKI Scalability issues

D-Grid project is urged to develop sustainable business model

SDI / GIS products have large user base Thousands of users Heterogeneous user groups: Commercial, academic, private, institutional

Certificate Authority needs to issue a certificate to each end user

Are academic CAs ready for this amount of certifications? Who pays for this?



GSI implementation


All components need to implement GSI

At the very least: transport-level security

Acceptance of proxy credentials as an authentication token

For other components: connectors / adaptors needed

MAGE toolkit offers easy integration of GSI

Creates Globus container services


GDI-Grid and data


Geospatial data is integral part of GIS / SDI

Very costly to collect and maintain Data is nearly always proprietary and non-free Highly valuable asset

Data is essential for geoprocessing Input for computation jobs Input for illustrative tasks (maps, 3D scenes)

Data management important challenge for GDI-Grid


GDI-Grid: Data management

Typical situation in Grid environments Data stored on Storage Resource prior to job submission or (limited) Input data submitted together with job

In both cases: Input data is duplicated Commercial vendors do not permit this Contractual problem No technical solution (yet)

Many different license models / enforcement

Different data types / storage containers

Is a technical solution possible?



GDI-Grid and D-Grid: The big picture


Two distinct Grid infrastructures GDI-Grid contains community specific

infrastructure D-Grid acts as service provider for

computation

Scheduling in both Grids necessary

Different scheduling approaches

User interface: different possibilities

Only GDI-Grid infrastructure is under project control


The D-Grid view


D-Grid receives job callouts from GDI-Grid

Well-defined entry point necessary

D-Grid metascheduler deploys jobs to clusters

Cluster nodes pull relevant input data from GDI-Grid storage

Result data is pushed back to GDI-Grid storage


The GDI-Grid view

Transparent Interaction with D-Grid

End user does not notice difference to conventional GIS application

Alternative: Engine/portal for specification of workflows

Finished workflows passed to scheduler

Workflow elements sent to relevant WPS

WPS calls out to D-Grid



Scalability

Where to scale? SDI processing should profit from Grid integration Thus, processing tasks should be segmented/paralellized Scaling as late as possible Ideally after WPS

What to scale? Part of the GDI-Grid project For suitable data: tiling and computation of 1/Nth For other data: parallel computation of different Levels of Detail (LoD) Computation of different layers in parallel

How to scale? Current situation:

processing tools distributed as part of Grid job standardized software already available on Grid nodes

Adaptation to GDI-Grid situation necessary



Scalability and Heterogenity

SDI applications are very heterogenous Conventional Windows/Unix GUI interactive applications Non-GUI interactive applications (command-line etc.) Increasingly web-based end-user applications OGC web services

D-Grid is very homogenous Standardized architecture Unix only No GUI applications No interactive applications Limited possibility for custom libraries/environments

How do we converge? Many questions – technical and organizational Virtualization might be key



Discussion

Thank you!

More information http://www.d-grid.de/ - D-Grid web site http://www.gdi-grid.de/ - GDI-Grid web site

Contact information Christian Kiehle <[email protected]> Andreas Krüger <[email protected]> Christopher Kunz <[email protected]> General information <[email protected]>


GDI-Grid Spatial Data Infrastructure Grid Christian Kiehle Andreas Krüger Christopher Kunz OGF22...

Documents

Transcript of GDI-Grid Spatial Data Infrastructure Grid Christian Kiehle Andreas Krüger Christopher Kunz OGF22...