www.auscover.org.au

- AusCover - The Terrestrial Remote Sensing

and Spatial Data Facility for TERN

21 March 2011

Matt Paget and Edward King

CMAR Canberra

TERRESTRIAL ECOSYSTEM RESEARCH NETWORK TERRESTRIAL ECOSYSTEM RESEARCH NETWORK

www.auscover.org.au

Discovering AusCover datasets and products

• For all things dataset and product related:– www.auscover.org.au -> AusCover Portal

• Visualisation portal is up to this point– http://portal.auscover.org.au

• Metadata for datasets and products can be discovered here:– http://mest.auscover.org.au

• For remote sensing experts, direct access to datasets is possible– OPeNDAP, THREDDS, Geoserver, HTTP

• All currently available datasets and products– http://data.auscover.org.au

• Servers hosting AusCover data:– http://data.auscover.org.au/servers

www.auscover.org.au

Accessing AusCover datasets and products

Online

Coming soon

Source: http://www.aarnet.edu.au

BoM

NCI CSIRO

UQ

iVEC

VPAC

Data Fabric

www.auscover.org.au

AusCover strategic vision

TERN strategic priority 5: Sustainable production of an accessible and nationally consistent long-time series set of satellite images, satellite-based biophysical map products, next generation remote sensing research data and product validation for Australian conditions.

Web-enabled data delivery for gridded remote sensing products

and associated validation data

www.auscover.org.au

Data Needs and Inputs to Ecosystem & Climate Modelling for Land-Surface Condition & Change Detection

Time-series of Vegetation Biophysical Characteristics

• Cover type & extent• Phenology• Fraction of absorbed

photosynthetically active radiation (FAPAR)

• Leaf area index (LAI)• Disturbance – Degradation:

type/extent/intensity• Biomass• Net primary productivity• Future Climate Predictions ?

Other Land-Surface Characteristics

• Evapotranspiration• Land surface temperature• Near-surface meteorology• Precipitation• Albedo• Radiation fluxes• Energy balance• Surface roughness• Soil physics and chemistry• Topography• Runoff• Snow and ice• Fires and biomass burning

Source: ISLSCP – GCOS ECVs

www.auscover.org.au

Principles

• Best available remote sensing and validation datasets and products

• Multiple access pathways for users• Evolution away from proprietary data formats to open

standards for formats and access• Discoverability via metadata standards and catalogue

federation• Scalable and distributed data management system• Building sustainable relationships and collaborative links

with TERN facilities, other NCRIS facilities, Aust Govt. and national computing facilities.

www.auscover.org.au

System design

• Storage

• Metadata

• Web-enabled data servers

• Whole of system tools

www.auscover.org.au

Storage

• We are not resourced to build a monolithic “peak facility” to store all AusCover data

• The maintenance task would be overwhelming• Data is almost always best curated and cared for locally

by its creators• This suggests that some sort of distributed storage

model will be both necessary and desirable

CollectionCatalogue

(portal)DataServer

DataServer

DataServer

www.auscover.org.au

Metadata…

1 10 100 1000 10000 100000 Granules

Col

lect

ions

www.auscover.org.au

Metadata standards

• There are LOTS to choose from• Adopt simple BUT sufficient to:

– Maximise inclusiveness– Guarantee minimum level of service

• Mandatory minimum standard for collection metadata– ANZLIC profile of ISO19115 (geospatial data)– Fit for purpose + widely supported– No need to distinguish ourselves from other communities

• Support a range of levels for granule metadata– Strive for CF (climate and forecast) Conventions– Optionally ACDD (attribute convention for data discovery)

translation

translation

www.auscover.org.au

Whole of system tools

Whole of system tools

•Discovery – http://mest.auscover.org.au•Visualisation – http://portal.auscover.org.au•Data subsetting and aggregation –proof of concept prototype (http://www.marine.csiro.au/remotesensing/imos/)

Data access •Currently available products – http://data.auscover.org.au•Browse all AusCover servers - http://data.auscover.org.au/servers

Collaborations •Australian institutions – UQ, BoM, GA, iVEC, CSIRO;•TERN facilities – Ecoinformatics, Supersites, TERN Central;•NCRIS platforms – IMOS, ALA, AuScope;•eResearch strategies – ANZLIC, ANDS, NeAT

www.auscover.org.au

Present systems

• BoM– Technical lead is Tim Pugh– Existing Opendap/Thredds server

• UQ– Technical lead is Leslie Elliot– Server powered up and install/config of software is in progress

• CSIRO– Technical lead is Ron Jones– Beta servers built for Portal and MEST– Development Opendap, Thredds, Geoserver, Ncwms and Erddap servers

• Perth– Opendap/Thredds server supporting both TERN and IMOS

• GA– Pathway to push data to CSIRO– MODIS swath processing to be initiated at the ANU NCI

• Darwin– Pathway to push data to CSIRO– MODIS swath processing to be initiated at the ANU NCI

www.auscover.org.au

Next steps

Q1 2011 Publish data formats and metadata cookbooks for data providers.

With appropriate guidance, the expectation is for data providers to do the majority of data formatting and metadata creation.

Q2 2011 Refine implementation of data services at participating organisations.

We aim to have reasonably consistent yet adaptable systems. The expectation is that the addition of new nodes will be relatively easy.

Identified tasks but no timeline yet

Enhancement of the whole-of-system tools:• (semi-)Automated harvesting of metadata from datasets;• Implementation of usage and data download statistics;• Improved integration of the metadata and visualisation portals;•Continued co-development of the granule aggregation and subsetting tool with IMOS/eMII.

Ongoing Stay abreast of TERN, NCRIS and eResearch capabilities.

Web-enabled data delivery and system interoperability is a dynamic field and we’re at the leading edge.

www.auscover.org.au

Threats/Risks

• Failure to secure viable storage– Multi-provider strategy

• Inability to straddle the GIS/Global divide– Adopting loose strategy in terms of mandated formats

• Failure to establish/support management/ingest workflows & transparency– Work with data sets team and community

• Nodes fail to adopt – Outreach, feedback, documentation, roadshows, help

• Granule manager technology superseded– Currently no other solution exists (but it might), we’re watching

www.auscover.org.au

AusCover, TERN and NCRIS

www.auscover.org.au

Generic and Facility architecture

www.auscover.org.au

Data and Metadata Team role

• Provision of remote sensing data sets.

• Providing advice and a process for creation of metadata for data sets

• Guiding the reformatting of data files for improved web-enabled access and inclusion of metadata for “future-proofing”

• Building relationships with Australia’s large computing facilities to both allow a platform for serving AusCover datasets and to promote coordination and consistency with other NCRIS Capabilities

• Developing new server-side subsetting and aggregating tools in a partnership with IMOS to enhance the usability of large remote sensing datasets.

www.auscover.org.au

Servers and services

Selected data servers(standard layer services)

•Tier 1 (core): OPeNDAP, THREDDS, Geoserver.•Tier 2 (value-adding): ncWMS, ERDDAP.

Reasons for their selection •Use of, and alignment to, open standards for data access (eg, DAP, OGC, KML).

•Scalability with respect to a distributed network of data servers around Australia.

•Common use within other NCRIS platforms.

Key requirements for effective discovery, management and use of data

•Searchable collection metadata records (for each distinct dataset served by AusCover).

•Evolution away from proprietary file formats towards open-standard file formats for datasets.

•Early and accurate recording of metadata including provenance, algorithm and theoretical basis, maturity, validation, contacts and online resources.