ICEDS – Creating a continental-scale on-line geographic information service

Jeremy Morley, Nuno Gil, Jan-Peter Muller, Cristiano Giovando, Kashif MunirDept. Geomatic Engineering, UCL

Oliver Greening, Rob ScottESYS plc

On-line GIS

• Fast developing area• New distributed processing technologies:

– Web services components (SOAP etc)– Grid processing

• Aided by standards– Data transfer formats– Data access methods– OpenGeospatial Consortium central + ISO TC/211

• ICEDS: Integrated CEOS European Data Server– Demonstrator sponsored by British National Space

Centre International Co-operation Programme

ICEDS Aims

• To promote Web access to large datasets– Europe (to Urals) and Africa– SRTM & Landsat mosaics

• To prepare:– An OpenGeospatial Web Map Service (WMS)– Web Coverage Service (WCS)– Portal website

• To provide feedback on data preparation process• To use open access (i.e. open source / freeware /

free to download) software– MapServer: http://mapserver.gis.umn.edu/

OGC Web mapping

• Open Geospatial Consortium (new name for OpenGIS Consortium from September 2004)

• Provides practical software standards for data and processing interchange of geospatial information

• Web Map Service (WMS)– Provides rendered maps (i.e. a portrayal service)

• Web Coverage Service (WCS)– Provides user-definable extracts of raster (coverage) data (i.e. live data

connection)• Web Terrain Service (WTS)

– Provides rendered images of 3D terrain• Web Feature Service

– Provides access to vector and annotation data• Interface is simply through URLs on HTTP

– Stateless– No need for low-level SOAP infrastructure, but experiments with this will

commence shortly

Hardware

• Primary ICEDS server:– Linux RAID Server with ~2TB capacity– Mandrake Linux– Apache WebServer and Tomcat servlet container

• Processing carried out on a P4 laptop machine and P3 desktop with 20GB local disk.

• Test web-server: IIS 6 on Win2003 Server - Dell Precision 210 - Pentium 3, 497MHz, 512MB RAM, 60GB local disk.

• Sun Sparc Ultra 60, with a 20GB local disk, for editing in IDL/ENVI and ArcInfo

• Further RAID server for additional data storage (Landsat, SRTM, ArcCoverages, Grids)

Software

• UCL GE Dept. software:– ENVI 4.0 (SRTM and Landsat processing and editing)– ArcView 3 and ArcGIS 8.3 (coverages and vector data processing)

• Free software:– DLGV32 Pro NASA release (SRTM colourised hillshading)– MicroDEM

• Libraries and tools– GDAL libraries: GDALtileindex (shapefile indexing)– Global Mapper (full version of DLGV32 – low cost)– Free libraries for use with MapServer (including GDAL)

• Map servers tested– MapServer 4.4.0 beta1– IONIC RedSpiderWeb 3.1 (WMS setup and website design)

• Various IONIC tools (indexers, editors, pyramid-creator)– DeeGree v.1

• Deegree tools: runtileimage2

Current datasets

• Hosted datasets:– 3 arcsec. unedited SRTM DEM

• Downloadable from Eros Data Center (EDS)– GSHHS global coastlines dataset (based on WVS)

• Downloadable from NOAA website– JPL Landsat 5 full-res mosaic of Africa

• 141GB on RAID Server• Courtesy of Nevin Bryant (JPL)

– NASA MODIS "Blue Marble" global composite

• Cascaded datasets:– NASA MODIS "Blue Marble" global composite

• http://onearth.jpl.nasa.gov/gis-wms.cgi?– NASA LANDSAT 7 Global mosaic

• http://onearth.jpl.nasa.gov/gis-wms.cgi?– ESRI dataset of inland waters (from DEMIS)

• http://www.demis.nl/mapserver/request.asp?– “Bathymetry” derived from MODIS “Blue Marble” data

• http://maps1.intergraph.com/wms/world/request.asp?

• Any OGC compliant services worldwide can be cascaded

Data & server preparation

Guidelines document

• A guidelines ‘cookbook’ is available on the ICEDS website http://iceds.ge.ucl.ac.uk

• The latest release (v 1.1) details the deployment of the Minnesota MapServer– Full details of server software setup and implementation– Full details of data preparation and hosting

• This document will be kept up to date as the service evolves. Expect a new draft release in September 2005

• Linear stretch proven to be better– Upper limit 98th percentile– To handle the possible

(numerous) presence of black pixels around the image edges, without compromising the stretch quality, the lower limit is automatically adjusted. Default value is the 2nd percentile

JPL Landsat 5 Africa mosaic contrast enhancement

JPL Landsat 5 Africa mosaic –retiling & pyramidal structure

• Retiling operation was performed in– 5ºx5º contrast stretched composite tiles (WMS)– 5ºx5º original tiles in separate bands (WCS)

• Pyramidal structure for WMS:– 0.0002778°/pixel from scale 1:1 to 1:1 000 000

(14302 tiles)– 0.002778°/pixel from scale 1:1 000 000 to 1:8 000 000

(1020 tiles)– 0.02778°/pixel from scale 1:8 000 000 to max (1 tile)

• Each level has its own index file to allow the map-server application to find and get the right files

SRTM retiling & hill shading

• For portrayal in WMS, preferred portrayal for SRTM was colourised & hill shaded.– Preferred software: DLGV32 or Global Mapper– Handles SRTM HGT; can take user-defined colour table

• Unedited SRTM data contains elevations over ocean – masked out in hill shaded images

• An additional issue: raw HGT files from EDC contain additional overlap pixels

• Overlap pixels need to be removed for processing in Global Mapper / DLGV32

SRTM in DLGV32 / Global Mapper

• DLGV32 reads the native *.hgt files and displays the data with a user-defined elevation colour LUT and greyscale hillshading

• Missing pixels are displayed as transparent - when exported they appear with the background colour (red)

• Possible to export the data into different raster image or elevation formats

Hill shading colour table

• A colour table has been defined reflecting atlas-like colour ranges

• The “lowest green” ranges from +20m to -∞m, to include all possible areas below sea level

• The resulting “green-sea” is then masked with vector layers

India SRTM hill shading

• Same colour table used for Africa and EuropeArea of Tibet has an elevation consistently greater than 5,000m and appears all in purple, the RGB values combination previously assigned to the highest elevation range

The global coastlines dataset

• ~1:100 000 scale coastlines dataset (NGDC GSHHS)

• Made by Hawaii University, centred on 180°E !!

• Edited so mosaic centred on Greenwich meridian (ArcINFO processing CLIP, CLEAN, BUILD, UNION, etc)

• Polygon for the sea mask extracted

The masking process

• Vector polygon layer displayed in white (in most WMS clients, white is usually portrayed as transparent)

• The background colour (red) shows the voids present in the SRTM.

• The coastlines dataset has a lower resolution than SRTM, but sufficient to include many small islands (better than VMAP0 coastlines dataset -> 1km!)

SRTM shading & masking

• Hillshaded and masked SRTM 1°x1° tiles

• EU25: 1,118 files = 4.5GB• Uncompressed GeoTiff tiles

– faster access– geo-info preserved

• Index file created

• Pyramidal structure to be accessed by the map-server

• Lower res./bigger tiles created

Africa SRTM Masking

SRTM tiles intersecting coastline shown for NE & S areas

DEM tiled in 4 regions (NW, NE, S, Azores)

Azores included in Europe and Africa – political versus geographic affiliations

Inland tiles do not need masking

MapServer setup

• Recently IONIC and Deegree servers were replaced by the University of Minnesota MapServer v 4.4.0-beta1 (http://mapserver.gis.umn.edu/)

• UMMS pros– Compiles on most versions of UNIX/Linux, MacOS and Windows– Made available via a CGI application (no need to have Tomcat)– Very flexible data setup with the GDAL utility (no need to re-engineer data

and easy to move to new server software)– Open source software supports WMS & WCS– DescribeCoverage request is supported on the WCS– WCS capable of easily serving 16-bit GeoTIFF coverages (SRTM)

• WMS & WCS setup– WMS of hillshaded STRM pyramid & Landsat African stretched mosaic– WCS of 16-bit GEOTIFF SRTM and 8-bit Landsat mosaic

• Web application interface same as IONIC & Deegree demo

http://iceds.ge.ucl.ac.uk/cgi-bin/wms?map=wms.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetCapabilities

MapServer versus IONIC & Deegree

• IONIC in current pre-release WCS version cannot access a layer from a hierarchical directory of files– Requires a spatial image database (such as Oracle): beyond scope of

ICEDS project

• IONIC Red Spider is proprietary– Dedicated support, quicker development times– Cost! However, IONIC kindly donated licence(s).

• Deegree is fussy, for example in the relationship between pyramid and full-resolution layers

• Deegree 16-bit support inclusion would require major changes• MapServer has a very flexible data setup

– Uses shape file as index– Data can be stored in disk in a simple directory structure or in a spatial

database– Has utilities to ease data configuration: e.g. editing of configuration files

Connecting to the service

Current portal

SRTM (unedited) data gaps visible in red

• Simple client based on the client/server architecture developed with IDL– Code is easily portable via an IDL virtual machine which is freely

available

• Allows on-line connection to the

ICEDS WCS server– Signed 16-bit data retrieval– Display 3D perspective views of

SRTM DEM– Vertical exaggeration control

parameter– Possibility to choose desired

area by BBox

WCS client

WMS Connection from ArcGIS

WCS Connection from Cadcorp SIS

Comsine: WCS demonstration - orthorectification

wcs_url = 'http://iceds.ge.ucl.ac.uk/cgi-bin/wcs?map=wcs.map&SERVICE=WCS&REQUEST=GetCoverage&COVERAGE=srtm&FORMAT=GEOTIFFINT16' res = max(tif_pixsize_x, tif_pixsize_y) wcsRequest = wcs_url + "&CRS=EPSG:" + str(epsg) + "&BBOX=" + bbox + "&RESX=" + str(res) + "&RESY=" + str(res) logger.info("WCS REQUEST " + wcsRequest) if (dem == None): demFilename = str(time.clock()) + ".tif" urllib.urlretrieve(wcsRequest, filename) cleanup = 1

Current portal

Current portal

Current portal

Additional annotation cascaded from GLOBE server

Current portal

Cascaded layers only. GLOBE annotation plus JPL OnEarth Landsat 7 mosaic

Current portal

JPL Landsat 5 low-res., stretched mosaic

Current portal

Hill-shaded, colourised SRTM DEM

Current portal

MODIS Blue Marble

Current portal

Cascaded OnEarth Landsat 7 mosaic (NB: geoloc. shift c.f. SRTM)

Current portal

JPL Landsat 5 full-res., stretched mosaic

Current portal

Current portal

ICEDS view of Cordoba

ICEDS optimal architecture

InternetInternet

portalhost sitehost site

InternetInternet

ICEDS Portalat http://iceds.ge.ucl.ac.uk

Web Browser (Thin Client)Web Browser (Thin Client) Applications (Thick Client)Applications (Thick Client)

Services

Data Providers

Data

WMS WCSWFS WMS

Integrated services: - gazetteer- catalog- registry

UCL G.E.JPLUNESA……

ICEDS Portal objectives

• CEOS Geo-Portal publishing services compliant to OGC CEOS Geo-Portal publishing services compliant to OGC specs and ISOTC211 standardsspecs and ISOTC211 standards

• Supports GSDI, ESDI, National, Regional and Local SDIs Supports GSDI, ESDI, National, Regional and Local SDIs • Promotes migration to an environment that enables spatial Promotes migration to an environment that enables spatial

data maintained by various collaborating communities to be data maintained by various collaborating communities to be readily accessed, integrated, fused and applied to support readily accessed, integrated, fused and applied to support critical decision-makingcritical decision-making

• Possible link to Reuters AlertNet humanitarian aid web Possible link to Reuters AlertNet humanitarian aid web mapping servicemapping service

• Free direct access to SRTM and Landsat data, through an Free direct access to SRTM and Landsat data, through an easy and intuitive interfaceeasy and intuitive interface

ICEDS Next Phase

• Next phase to address user needs and further develop the services – just begun

• Main effort will run from Feb until Nov 2005, with final presentations in Jan 2006.– Publicise the service to user groups in Africa– Collate and analyse use cases, and extract requirements for further

development– Parallel onward development of the services and portal– Accommodate new requirements for Web Service capability from

use cases as budget, time and feasibility allows

Initial use cases

• National Remote Sensing Agencies in Africa– Web Services for publishing of national geospatial data archives

• Reuters Alertnet & NGOs– Evolution of the Alertnet WMS for improved map making and

access to example data sets (e.g. those related to the recent Asian Tsunami event)

– Awaiting input from Reuters

• CEOS WGCV– Web Services to provide data of WGCV test sites– Addressed by Prof J-P Muller

• Not restricted to African users– We are interested to hear from any group who use this service, and

have opinions on how to improve it

Contacting African Users

• WGISS contacts – Lorant and John– Links back to UNEP Nairobi, awaiting contact with UNEP

• UNOOSA – David Stevens– Has agreed to support ICEDS project, and particularly likes the availability

of the cookbook and scripts for local implementations– Has reservations about users ability to access the service, due to poor

local infrastructure

• AARSE – Jide Kufoniyi– Director of Regional Centre for Training in Aerospace Surveys (RECTAS),

Obafemi Awolowo University Nigeria– Strongly supportive of the project and is propagating the ICEDS URL

throughout AARSE membership, and other African user groups

• UN FAO – Jelle Hielkema– Very interested in ICEDS in Tromso w.r.t. Madagascar, awaiting his input.

Candidate technology developments

• Better printing support - online interactive map composition to enable high resolution PDF files to be produced for download to the client (combining any available WMS data sets)

• Data exploration - additional WMS datasets added by a user, and manual annotation with standard menu of graphical icons

• Support for identification of temporal changes (flickering between data sets, transparency, animation and flicker of multiple data sets in a user’s display list. To include display list management functions

• Web Terrain Service – rendered imagery of 3D terrain• WGCV-TMSG extensions• Portal to link separate ICEDS-based services

Candidate Data Sets

• Near-global SRTM JPL unedited DEM in WMS/WCS format (sourced from USGS)

• Near-global SRTM JPL edited in WMS/WCS format (also USGS)• Near-global SRTM SAR amplitude derived water body mask in WMS

format (source NGA to be supplied)• Near-global X-SRTM coverage map and colourised hill-shaded image

mosaics in WMS format (source DLR)• Global ASTER stereo DEM coverage map in WMS format (source

JPL/USGS)• Global ASTER stereo image (not yet processed to DEM) coverage

map in WMS format (source JPL/USGS)• Global Landsat 5 Image mosaics (1”, bands 1-3, 4, where available

(source JPL)• Coverage maps from other higher resolution sensors where available

as WMS coverage.• Hot links to CEOS-WTF test site datasets (source EDC/GSFC)

Promotion plans

• CEOS WGISS 18,19 and Plenary• African Journal of GeoInformation (AARSE

Journal), and presentation to AARSE 2006 conference in Cairo.

• UK EO Community – BNSC ICP2 workshop in November 2005

• Digital Earth – presentation to conference in Tokyo April 2005

• OGC technical and planning committee meeting in Frascati May/June 2005

Conclusions

• Implementations of continental-scale high-resolution data serving using OGC technologies

• Providing data & portrayal (WCS/WMS) services• User-selectable data freely and “instantly” available at full

resolution• Guidelines, scripts and software for setting up your own

server available now• Significantly more functionality and data to be added• Use cases with African users will be used to assess service

and refine requirements for development• Initial user assessment very positive, but access limited for

many users due to poor infrastructure• Updated service, guidelines and scripts/software will be

made available September – November 2005