Slide 1 ECMWF, 5 September 2007 Slide 1 The SIMDAT project Baudouin Raoult Head of Data and Services...

ECMWF, 5 September 2007

Slide 1

Slide 1

The SIMDAT project

Baudouin RaoultHead of Data and Services Section

ECMWF


Slide 2

Slide 2

SIMDAT and the VGISC project

SIMDAT is a four years EU funded project

SIMDAT was an opportunity to fund the development of the VGISC- DWD invited ECMWF to join and take a lead rôle in the SIMDAT project, in

order to develop the VGISC.

- For ECMWF it was an opportunity to develop expertise in modern technologies (GRID, Service oriented architecture,…)

The partners are DWD, ECMWF, EUMETSAT, Météo France and UK Met Office

The software was designed according to all WIS related documents present on the WMO web site, as well as previous VGISC documents- In particular the functional roles of GISCs, DCDPs and NCs, as well as

their interactions

- The virtualisation requirements of the VGISC


Slide 3

Slide 3

The VGISC architecture


Slide 4

Slide 4

Access to existing data repositories in a non-intrusive fashion


Slide 5

Slide 5

Connectivity to another site via secure connection layer


Slide 6

Slide 6

A scalable system: addition of new sites


Slide 7

Slide 7

Catalogue Synchronisation: each site has a copy of the global catalogue

Synchronisation


Slide 8

Slide 8

Data from anywhere can be accessed from everywhere

Retrievals


Slide 10

Slide 10

Portal: Searching…


Slide 11

Slide 11

Portal: Search results


Slide 12

Slide 12

Portal: Browsing by Category


Slide 13

Slide 13

Portal: Legible Metadata


Slide 14

Slide 14

Sites are organised in a mesh network

Each site is connected to 2-3

peers at most

Each site can reach any sites

which is part of the network

Provides a scalable solution


Slide 15

Slide 15

Data policies and Access controlWe want authentication

- Who is this user?

We want to support data policies

- e.g. WMO Resolution 40

We want access control

- According to his/her profile, can this user access data with this data policy?

We want a decentralised solution

- No central security server

- User from anywhere canaccess data from everywhere


Slide 16

Slide 16

Decentralised access control

This site holds some data

… and defines under which policy this data is available (e.g. for research and education)


Slide 17

Slide 17


This site registers a user

… and defines its profile (e.g. a researcher)


Slide 18

Slide 18


The sites must establish a relation of trust

- The trust relation is defined by having the same understanding of the term “researcher”

- More that one sites can be part of a trust relation

- The trust relation is enforced by exchanging X509 certificates

A scalable solution:

- Each site manages its own users


Slide 19

Slide 19

How to publish data in SIMDATDownload the software

Install the catalogue node and the portal (optional)

- Edit the configuration file

- Start the software

Install the data repository

- Edit the configuration file

- Start the software

Edit some metadata files to describes your datasets

- Drop the files in a directory

Write a shell script that translate SIMDAT requests into requests to your existing database


Slide 20

Slide 20

From architecture to usage


Slide 21

Slide 21

Collaboration

With the WMO community

- Experience gain in the project are fed back to WMO via expert teams.

Evaluation copies of the software

- Australia, Canada, China, Finland, Japan, Korea, Morocco, Norway, Russia, Sweden, Zimbabwe

- The software is available under an Open Source license

Other communities are interested

- EUMETNET OPERA project

- Global Monitoring for Environment and Security (GMES)

- International Polar Year (IPY)


Slide 22

Slide 22

Status: 11 sites connected

Satellite data

ERA40 dataTIGGE data

Climate Time Series

Aviation data (TAF, METAR)Lightning data

Model outputReal-time GTS data

Model outputClimate Time Series

Model outputObservation

Model outputSatellite data

Model outputWave Observation

Research datasets

Oceanographicdata

More than 27,000 datasets discoverable


Slide 23

Slide 23

Data availableBoM NWP outputs, Observations GRIB, ASCII

CMA NWP output, satellite data, TIGGE GRIB, BUFR

DWD Climate Time Series, NWP outputs XML, HTML, GRIB

ECMWF NWP outputs (ERA15, ERA40, TIGGE), Sample GTS observations (1 day)

GRIB1, GRIB2, BUFR, PDF

EUMETSAT METEOSAT 8 images JPEG

JMA NWP outputs, MTSAT images, Observations

GRIB, BUFR, animated GIFs

KMA Climate Time Series, NWP outputs XML, GRIB

Météo France NWP outputs Real-time GTS data GRIB, ASCII

NCAR ~ 6000 datasets NetCDF, other

RNODC Oceanographic data (BATHY, SHIP, TESAC)

ASCII

UKMO Aviation data (METAR, TAF) XML


Slide 24

Slide 24

Support of WIS requirements

Routine collection of observation data Work in progress.

Automated dissemination of all observed data and products, both real‑time and non real-time : Push model

Work in progress.

Ad-hoc requests for data and products : Pull model

Done.

Support of different user profiles and data policies

Done.

Support of data and network security Done. Based on SSL

Support the integration of diverse datasets Done. Support for Any data type, from any data repository.

Reliable infrastructure Done.

Technologically sustainable and appropriate to local expertise

Done. Based on Standards. Run on a PC. Open Source components.

Modular, flexible and scalable Done.


Slide 25

Slide 25

Conclusion

SIMDAT is a fully decentralised system- All sites are true peers and have equal rights

- Decentralised user and data policies management

- Integrated catalogue and data retrieval functions (one-stop-shop)

- Designed for operational use

Interfacing with any existing data repositories- Without any impact on the local infrastructure or disruption of operational activities

- Support for any data types (GTS bulletins, Model outputs, Satellite images, climate time-series, ….)

Generates interest in meteorology and other environmental communities- A synergy has been established

Data available from 11 sites worldwide- Software running continuously, catalogues are constantly updated

Slide 1 ECMWF, 5 September 2007 Slide 1 The SIMDAT project Baudouin Raoult Head of Data and Services...

Documents

Transcript of Slide 1 ECMWF, 5 September 2007 Slide 1 The SIMDAT project Baudouin Raoult Head of Data and Services...