2. WP9 – Earth Observation Applications

ESA DataGrid Review Frascati, 10 June 2002

1. Welcome and introduction (15m)

2. WP9 – Earth Observation Applications (50m)

3. ESA and WP9: infrastr, appl, TB 0/1 (50m)

4. Demonstration of EO applications (45m)

5. ESA and WP9: scale up, effort, recovery plan (50m)

6. Side ESA actions related to GRID (30m)

7. Reviewer separate meeting (60m)

8. Conclusions (30m)

ESA DataGrid Review – 10 June 2002 – n° 2

Summary item 2

WP9 – Earth Observation Applications (50m)

DataGrid EO requirement (LF, 15m)

DataGrid WP9 tasks, WP9 Plan (JL, 15m)

ESA DataGrid Review – 10 June 2002 – n° 3

Earth Observation Community GRID interactive scenario

Common access to EO missions cataloguesAcquisition plan, order, delivery

Parametric data fusion and models integration

Collaborative publishing of results

On demand high level products generation

ESA DataGrid Review – 10 June 2002 – n° 4

EO and Network Computing data models:EO is more than “parallel task execution”!!

Distributed Computing Integration of data from various instruments

and missions

High-Throughput Computing Interferometry …

On-Demand Computing Generation of EO user products…

Data-Intensive Computing Archive data re-processing, climate modeling…

Collaborative Computing Scientists application interactions, Instrument

cal/val …

Ian Foster and Carl Kesselman, editors, “The Grid: Blueprint for a New Computing Infrastructure,” Morgan Kaufmann, 1999

ESA DataGrid Review – 10 June 2002 – n° 5

The Grid from a Services View

:

:E.g.,

Resource-specific implementations of basic servicesE.g., Transport protocols, name servers, differentiated services, CPU schedulers, public keyinfrastructure, site accounting, directory service, OS bypass

Resource-independent and application-independent services authentication, authorization, resource location, resource allocation, events, accounting, remote data access, information, policy, fault detection

DistributedComputing

Toolkit

Grid Fabric(Resources)

Grid Services(Middleware)

ApplicationToolkits

Data-Intensive

ApplicationsToolkit

CollaborativeApplications

Toolkit

RemoteVisualizationApplications

Toolkit

ProblemSolving

ApplicationsToolkit

RemoteInstrumentation

ApplicationsToolkit

Applications Chemistry

Biology

Cosmology

High Energy Physics

EO and Environment

...

Sci

ence

& A

pp

licat

ion

Sys

tem

ESA DataGrid Review – 10 June 2002 – n° 6

Number crunching: interferometry, subsidence,DEM generation

Pomona (Cal): subsidence velocity fields40 ERS1/2 images (92-99), Ambiguity: 28 mm

Digital Elevation Model

GRID requirements:• large data files (10+ GB) • stages with intensive

processing • science driven value adding

ESA DataGrid Review – 10 June 2002 – n° 7

• Provide a single access point to space systems to emergency & rescue organisations in case of disasters

• Participating Space Agencies: CNES, CSA, ESA, ISRO, NOAA, …

• Missions: RADARSAT; ERS, (Envisat); SPOT; IRS; NOAA, …

Charter for Disaster Management

ESA DataGrid Review – 10 June 2002 – n° 8

ESRIN

MATERA (I)

NEUSTREL.ITZ (D)

KIRUNA (S)- ESRANGE

MASPALOMAS (E)

TROMSO (N)

MATERA (I)

METADATABROWSEWEB

SEAWIFS

SPOT IRS-P3

LANDSAT 7TERRA/MODIS

STANDARDPRODUCTION CHAINS

MULTIMISSIONDATABASES FOR REMOTE ACCESSAND USER SERVICES

USERS

HISTORICALARCHIVES

USERS

PRODUCTS

AVHRR

ESA EO facilities real-time infrastructure

ESA DataGrid Review – 10 June 2002 – n° 9

No Projects1-25 Projects26-5051-100100+ Projects

Countries

P.I. geographic distribution

AOs: Stimulating scientific research world-

wide

AOs: Stimulating scientific research world-

wide

AO-1 (1986)

AO-2 (1994)

AO-3 (1998)

AO- ENVISAT (2000)0

100

200

300

400

500

600

700

Nr

of

pro

jec

ts

3500+ science Users of ESA data

3500+ science Users of ESA data

120 New Cat-1 Projects in 2001120 New Cat-1 Projects in 2001

700 Envisat AOs to start in 2002700 Envisat AOs to start in 2002

Envisat science community

ESA DataGrid Review – 10 June 2002 – n° 10

Why GRID in EO? (1)

EO Community: Progressive refinement of data from many data sources to produce higher quality products

Product generation chain involving distributed organisations and users

Collaborative: distributed users and data – large international cooperation

Discovery: large numbers of products & resources

Interoperabiltiy of catalogue and metadata already in operation

Web based data services

ESA DataGrid Review – 10 June 2002 – n° 11

Why GRID in EO? (2)

Massive, non-stop data volumes New instruments, sensors & product types

Distributed archives

Historical dataset reprocessing

Complex numerical processing algorithms

Near real-time turnover

ESA DataGrid Review – 10 June 2002 – n° 12

Needed GRID technologies

Resource-independent and application-independent services (middleware)

authentication, authorization, resource location, resource allocation, remote data access,

accounting, security, quality of services, fault detection, real time services, …

Specialized protocols, procedures, data standards, operational environments, interfaces to EO legacy systems…

EO dedicated portal and user access…

ESA DataGrid Review – 10 June 2002 – n° 13

OZONE: a case of Global Environmental Monitoring

GOME analysis detected ozone thinning over Europe 31 Jan 2002

GRID requirements:• Multi instrument data fusion• Distributed data sources,

science and institutional users

• Complex data processing (1d data = 20 dprocessing

• Near real time deliv.

ESA DataGrid Review – 10 June 2002 – n° 14

Application of DataGrid in EO

Focus on One EO application (Ozone) but explore:

Collaborative environment, parallel data processing, … Interface to legacy and COTS systems Develop generic components Re-use components to add new applications Integrate compatible technologies Integrate other data types (new envisat instruments)

Testing in “controlled” GRID environment (ESRIN-ENEA) and in “wide-European” environment

Feedback to DataGrid developers and Architecture Group