Data preparation, tools & formats · • Not a task to be underestimated! • Plan • Document...
21
Data preparation, tools & formats Warren Vick Europa Technologies Ltd.
Transcript of Data preparation, tools & formats · • Not a task to be underestimated! • Plan • Document...
Data preparation,tools & formats
Warren VickEuropa Technologies Ltd.
Better Mapping 2017
What’s your poison?
• We use a variety of methods and technologies• Straw poll
• Hand drawn• Non-GIS software (whole process)• Esri• PB / MapInfo• Open source• Other
• Your choice?
Better Mapping 2017
Ken Field, 2017
Better Mapping 2017
What is the best GISfor cartography?
• There is no single product or platform• Where possible, use the right tool(s) for the job
Better Mapping 2017
Data Preparation
• Crossing the data chasm• Not a task to be underestimated!• Plan• Document
(especially if a repeated process)
Source data Data ready & ideal for use
Effort Results
Disproportionateeffort
Better Mapping 2017
What might need to be changed?
• Data Format• Data Type (raster / vector)• Data Projection• Inherent styling• Geometry joins / splits
(Good example in my second presentation)• Topology
Better Mapping 2017
Formats
• There are many formats for geospatial data:• Based on published standards• De facto vendor standards
• Data suppliers support limited number of formats• Choose the best format for your software• Some may be better than others:
• Active vs. transfer formats• Know the limitations of your chosen format
• 2Gb file size limits are common
Better Mapping 2017
Projection
• Choosing the right projection for a map isvery important
• Nearly all GIS software can reproject on the fly…• … but this is a computational burden• If you have a known destination projection,
it is best to process to it• Not all reprojection algorithms are the same…• … which can lead to misalignment…
Better Mapping 2017
Better Mapping 2017
Spatial ETL• Extract / Transform / Load• Promoting greater interoperability• Building a bridge over our data chasm• Common tasks:
• Reprojection• Transformations• Restyling
• Other tasks:• Geocoding• Merging / conflict management• Publishing / dissemination
Better Mapping 2017
Developing a workflow
• Programming/scripting vs. Visual workflows• Programming/scripting is difficult but becoming
more accessible. e.g. Python• Visual programming is being made available on
many technology platforms• Building blocks (readers, transformers & writers)
with an intuitive drag-and-drop user interface• Examples…
Better Mapping 2017
Alistair Aitchisonhttps://alastaira.wordpress.com/2011/04/11/an-unlikely-replacement-for-the-bing-maps-3d-control/
FME, Safe Software (www.safe.com)
Better Mapping 2017
Geoprocessing in ArcGIS, Esri (www.esri.com)
Just like programmers creating code,add comments to workflows so that they
may be better understood by others.
Better Mapping 2017
Spatial Model Builder, ERDAS Imagine, Hexagon (www.hexagonegeospatial.com)
Better Mapping 2017
Graphical ModellerQGIS (www.qgis.org)
Better Mapping 2017
Introducing GDAL & OGR
• Open source software (www.gdal.org)• GDAL for raster processing• OGR for vector data processing• Multiple pronunciations
Gee-Dhal vs. Goo-Dull, even Gee-Dull!• Included with QGIS package (www.qgis.org)• Command line tools
OSGeo4W Shell
Better Mapping 2017
Introducing GDAL & OGR
• Open source software (www.gdal.org)• GDAL for raster processing• OGR for vector data processing• Multiple pronunciations
Gee-Dhal vs. Goo-Dull, even Gee-Dull!• Included with QGIS package (www.qgis.org)• Command line tools
OSGeo4W Shell
Better Mapping 2017
OS Terrain 50
Hillshade Slope Base colour
Using gdaldem command
Better Mapping 2017
Using gdaldem command
LIDAR – Digital Surface Model (DSM)
Better Mapping 2017
GDAL/OGR Cheat SheetGDAL• Get vector information• Print vector extent• List vector drivers• Convert between vector formats• Print count of features with attributes matching
a given pattern• Read from a zip file• Clip vectors by bounding box• Clip one vector by another• Reproject vector• Add an index to a shapefile• Add a spatial index• Merge features in a vector file by attribute
("dissolve")• Merge features ("dissolve") using a buffer to
avoid slivers• Merge vector files• Get the difference between two vector files
OGR• Get raster information• List raster drivers• Force creation of world file• Convert between raster formats• Reproject raster• Clip raster by bounding box• Crop raster dimensions to vector bounding box• Merge rasters• Stack grayscale bands into a georeferenced RGB• Export a raster for Google Earth• Raster calculation (map algebra)• Create a hillshade from a DEM• Apply color ramp to a DEM
Derek Watkins (http://blog.dwtkns.com)https://github.com/dwtkns/gdal-cheat-sheet
