The Geodata Warehouse of the Swiss Federal SDIConceptual View of the FSDI «Ingest once, use...
Transcript of The Geodata Warehouse of the Swiss Federal SDIConceptual View of the FSDI «Ingest once, use...
![Page 1: The Geodata Warehouse of the Swiss Federal SDIConceptual View of the FSDI «Ingest once, use anywhere» ... FME INTERLIS Checker GDWH Services.EXE GDAL FGDB FGDB TIF XML ILI ITF XTF](https://reader031.fdocuments.net/reader031/viewer/2022011921/602c67e7c5d9f6029526a4eb/html5/thumbnails/1.jpg)
Federal Department of Defence, Civil Protection and Sport
Federal Office of Topography swisstopo
Dr. Patrick Gamma
Federal Office of Topography swisstopo
Esri European User Conference
Salzburg, 15 October 2015
The Geodata Warehouse of the
Swiss Federal SDI
![Page 2: The Geodata Warehouse of the Swiss Federal SDIConceptual View of the FSDI «Ingest once, use anywhere» ... FME INTERLIS Checker GDWH Services.EXE GDAL FGDB FGDB TIF XML ILI ITF XTF](https://reader031.fdocuments.net/reader031/viewer/2022011921/602c67e7c5d9f6029526a4eb/html5/thumbnails/2.jpg)
2Federal Office of Topography swisstopo Esri European User Conference, Salzburg
15 October 2015
Agenda
• Introduction
• Legal Basis, Terms & Definitions
• Life Cycle of Geodata
• Architecture of the Geodata Warehouse
• Technology
• Next Steps & Conclusions
![Page 3: The Geodata Warehouse of the Swiss Federal SDIConceptual View of the FSDI «Ingest once, use anywhere» ... FME INTERLIS Checker GDWH Services.EXE GDAL FGDB FGDB TIF XML ILI ITF XTF](https://reader031.fdocuments.net/reader031/viewer/2022011921/602c67e7c5d9f6029526a4eb/html5/thumbnails/3.jpg)
3Federal Office of Topography swisstopo Esri European User Conference, Salzburg
15 October 2015
What is the value of your Geodata?
Imagine that your organisation has lost an important dataset…
• Could you recreate it at all?
• Possibly with some information loss?
• Price for recollecting / reconstructing?
• What is the impact of not having the data?
• Good strategies (besides classic IT approach):
Use the data
Open data
• Our strategy: the above, plus a Geodata Warehouse, a «living archive»
![Page 4: The Geodata Warehouse of the Swiss Federal SDIConceptual View of the FSDI «Ingest once, use anywhere» ... FME INTERLIS Checker GDWH Services.EXE GDAL FGDB FGDB TIF XML ILI ITF XTF](https://reader031.fdocuments.net/reader031/viewer/2022011921/602c67e7c5d9f6029526a4eb/html5/thumbnails/4.jpg)
4Federal Office of Topography swisstopo Esri European User Conference, Salzburg
15 October 2015
Could you archive a geodataset from
1995 without any loss?
Imagine that your organisation should file a
geodataset, that was used or created 20
ago…
• Could you still read it?
• Is the expertise available to convert it into
the required formats?
• Would you be able to provide the necessary
information, e.g. Documentation, Metadata,
Data Model, Representation Model?
![Page 5: The Geodata Warehouse of the Swiss Federal SDIConceptual View of the FSDI «Ingest once, use anywhere» ... FME INTERLIS Checker GDWH Services.EXE GDAL FGDB FGDB TIF XML ILI ITF XTF](https://reader031.fdocuments.net/reader031/viewer/2022011921/602c67e7c5d9f6029526a4eb/html5/thumbnails/5.jpg)
5Federal Office of Topography swisstopo Esri European User Conference, Salzburg
15 October 2015
Just some thoughts about data loss...
• Data loss is one of the greatest risks for a NMA on the enterprise level, caused by e.g.
Technical problems, or even on purpose
Deleting the wrong data
• Some losses could only be detected after years, at the time when someone tries to access the data
• One can not keep all data limited storage costs
• Even if you keep all data, you probably won’t have the software to read it in the future
• Decide about which datasets should be kept and for how long, before being sent to the archives
• You should provide the documentation about a dataset while the know-how (people!) is still available
• Future effort for preparing & sending the data to the archives is minimal (ROI)
![Page 6: The Geodata Warehouse of the Swiss Federal SDIConceptual View of the FSDI «Ingest once, use anywhere» ... FME INTERLIS Checker GDWH Services.EXE GDAL FGDB FGDB TIF XML ILI ITF XTF](https://reader031.fdocuments.net/reader031/viewer/2022011921/602c67e7c5d9f6029526a4eb/html5/thumbnails/6.jpg)
6Federal Office of Topography swisstopo Esri European User Conference, Salzburg
15 October 2015
Legal Basis, Terms and Definitions
• The Federal Act on Geoinformation (GeoIA, 2008)
and the Ordinance on Geoinformation (GeoIO,
2008) essentially define 2 states / conditions in
which a geodata set can be:
• Long-term availability
The authorities responsible for the collection,
updating and management of official geodata
guarantee the long-term availability of this geodata.
• Archived
The Federal Archives conserves the geodata
«in use»GIS Community
«not in use»No GIS
![Page 7: The Geodata Warehouse of the Swiss Federal SDIConceptual View of the FSDI «Ingest once, use anywhere» ... FME INTERLIS Checker GDWH Services.EXE GDAL FGDB FGDB TIF XML ILI ITF XTF](https://reader031.fdocuments.net/reader031/viewer/2022011921/602c67e7c5d9f6029526a4eb/html5/thumbnails/7.jpg)
7Federal Office of Topography swisstopo Esri European User Conference, Salzburg
15 October 2015
The «Life Cycle» of Geodata (1)
![Page 8: The Geodata Warehouse of the Swiss Federal SDIConceptual View of the FSDI «Ingest once, use anywhere» ... FME INTERLIS Checker GDWH Services.EXE GDAL FGDB FGDB TIF XML ILI ITF XTF](https://reader031.fdocuments.net/reader031/viewer/2022011921/602c67e7c5d9f6029526a4eb/html5/thumbnails/8.jpg)
8Federal Office of Topography swisstopo Esri European User Conference, Salzburg
15 October 2015
The «Life Cycle» of Geodata (2)
![Page 9: The Geodata Warehouse of the Swiss Federal SDIConceptual View of the FSDI «Ingest once, use anywhere» ... FME INTERLIS Checker GDWH Services.EXE GDAL FGDB FGDB TIF XML ILI ITF XTF](https://reader031.fdocuments.net/reader031/viewer/2022011921/602c67e7c5d9f6029526a4eb/html5/thumbnails/9.jpg)
9Bundesamt für Landestopografie swisstopo
KOGIS
Kolloquium swisstopo
6. März 2015
The Aim of Long-term Availability…
… is to conserve official geodata for a limited period…
… in such a way that their quantity and quality are maintained…
…and they are available for continuous active use.
• Online availability should extend not just to the data fonds that are current at a given time but also to defined older snapshots (in the sense of time series).
From: Project Ellipse: Concept for the archiving of official geodata under federal legislation; Concept report. Federal Office of Topography swisstopo, Swiss Federal Archives, Version 1.3, 14.09.2013, p. 38
![Page 10: The Geodata Warehouse of the Swiss Federal SDIConceptual View of the FSDI «Ingest once, use anywhere» ... FME INTERLIS Checker GDWH Services.EXE GDAL FGDB FGDB TIF XML ILI ITF XTF](https://reader031.fdocuments.net/reader031/viewer/2022011921/602c67e7c5d9f6029526a4eb/html5/thumbnails/10.jpg)
10Federal Office of Topography swisstopo Esri European User Conference, Salzburg
15 October 2015
Long-term Availability vs. Archiving
of Geodata
«Frozen pizza from home delivery»«Hot pizza from pizza service»
Criterion Long-term availability Archiving
Preservation period Limited in time Unlimited in time
GIS functionality Yes No
Formats Current formats, ready for use Archivable formats
PrincipleKeep the data accessible, both
current and older releases
Keep permanently and
safely
Frequency of use Frequently Seldom
![Page 11: The Geodata Warehouse of the Swiss Federal SDIConceptual View of the FSDI «Ingest once, use anywhere» ... FME INTERLIS Checker GDWH Services.EXE GDAL FGDB FGDB TIF XML ILI ITF XTF](https://reader031.fdocuments.net/reader031/viewer/2022011921/602c67e7c5d9f6029526a4eb/html5/thumbnails/11.jpg)
11Federal Office of Topography swisstopo Esri European User Conference, Salzburg
15 October 2015
Long-Term Availability
Conceptual View of the FSDI
«Ingest once, use anywhere»
![Page 12: The Geodata Warehouse of the Swiss Federal SDIConceptual View of the FSDI «Ingest once, use anywhere» ... FME INTERLIS Checker GDWH Services.EXE GDAL FGDB FGDB TIF XML ILI ITF XTF](https://reader031.fdocuments.net/reader031/viewer/2022011921/602c67e7c5d9f6029526a4eb/html5/thumbnails/12.jpg)
12Federal Office of Topography swisstopo Esri European User Conference, Salzburg
15 October 2015
Requirements for the GDWH
• Ensure the long-term availability of official
geodata
• Import, preserve and export the geodata for
use within the FSDI
• Reduce or limit redundancy
• Minimise software manufacturer dependencies
• For swisstopo and for other federal agencies
(replace swisstopo legacy application, make it
useable for others)
• Fit the system in the overall FSDI framework
and the corresponding processes
![Page 13: The Geodata Warehouse of the Swiss Federal SDIConceptual View of the FSDI «Ingest once, use anywhere» ... FME INTERLIS Checker GDWH Services.EXE GDAL FGDB FGDB TIF XML ILI ITF XTF](https://reader031.fdocuments.net/reader031/viewer/2022011921/602c67e7c5d9f6029526a4eb/html5/thumbnails/13.jpg)
13Federal Office of Topography swisstopo Esri European User Conference, Salzburg
15 October 2015
OAIS Open Archival Information System
Reference Model ISO 14721:2012
Submission
Information
Package
Archival
Information
Package
Dissemination
Information
Package
![Page 14: The Geodata Warehouse of the Swiss Federal SDIConceptual View of the FSDI «Ingest once, use anywhere» ... FME INTERLIS Checker GDWH Services.EXE GDAL FGDB FGDB TIF XML ILI ITF XTF](https://reader031.fdocuments.net/reader031/viewer/2022011921/602c67e7c5d9f6029526a4eb/html5/thumbnails/14.jpg)
14Federal Office of Topography swisstopo Esri European User Conference, Salzburg
15 October 2015
Geodata Warehouse FSDI
System ArchitectureB
usi
ne
ssLo
gic
Tie
rD
ata
Tie
r
Filesystem Layer
Standardised Directory Structure
Standardised Data Formats
Consistency Checks
Accompanying documents
Aggregation Layer
DB-Access LayerTransformation
FunctionsConsistency
Checks
Services
GDWH Data Dictionary
Service
Geodata Delivery
Geodata Validation &
Ingest
Geodata Transformation
GenerateTransformAggregate
«Product independent»Original-/Master DataArchivable
Current GIS technology«Product dependent»Replacable/exchangeableGenerated from
Filesystem Layer
Interfaces to FSDILocal Catalogue
![Page 15: The Geodata Warehouse of the Swiss Federal SDIConceptual View of the FSDI «Ingest once, use anywhere» ... FME INTERLIS Checker GDWH Services.EXE GDAL FGDB FGDB TIF XML ILI ITF XTF](https://reader031.fdocuments.net/reader031/viewer/2022011921/602c67e7c5d9f6029526a4eb/html5/thumbnails/15.jpg)
15Federal Office of Topography swisstopo Esri European User Conference, Salzburg
15 October 2015
Filesystem & Aggregation Layer Demo
![Page 16: The Geodata Warehouse of the Swiss Federal SDIConceptual View of the FSDI «Ingest once, use anywhere» ... FME INTERLIS Checker GDWH Services.EXE GDAL FGDB FGDB TIF XML ILI ITF XTF](https://reader031.fdocuments.net/reader031/viewer/2022011921/602c67e7c5d9f6029526a4eb/html5/thumbnails/16.jpg)
16Federal Office of Topography swisstopo Esri European User Conference, Salzburg
15 October 2015
Bu
sin
es
s
Lo
gic
Tie
rD
ata
Tie
r
Filesystem Layer
Original Data2015
Aggregation Layer
Services
Software2015
Geodata Warehouse FSDIMigration Strategy for Software and Data
FilesystemLayer
Original Data2020
Convert
2015 2020
Migrate
Software2015
GIS Data2015
Aggregation Layer
Software2020
GIS Data2020
Services
Software2020
No change
Gene-
rate
Migrate
«Product independent»Original-/Master DataArchivableUntouched
Current GIS technology«Product dependent»Replacable/exchangeableGenerated from Filesystem
Layer…or converted from
Aggregation layer
Interfaces to FSDILocal CatalogueNew/adapted functionality
![Page 17: The Geodata Warehouse of the Swiss Federal SDIConceptual View of the FSDI «Ingest once, use anywhere» ... FME INTERLIS Checker GDWH Services.EXE GDAL FGDB FGDB TIF XML ILI ITF XTF](https://reader031.fdocuments.net/reader031/viewer/2022011921/602c67e7c5d9f6029526a4eb/html5/thumbnails/17.jpg)
17Federal Office of Topography swisstopo Esri European User Conference, Salzburg
15 October 2015
Geodata Warehouse FSDIMigrating over Time
2015 2020 2025 2030
2020 2025 2030
2020 2025 2030
2020 2025 2030
2020 2025 2030
![Page 18: The Geodata Warehouse of the Swiss Federal SDIConceptual View of the FSDI «Ingest once, use anywhere» ... FME INTERLIS Checker GDWH Services.EXE GDAL FGDB FGDB TIF XML ILI ITF XTF](https://reader031.fdocuments.net/reader031/viewer/2022011921/602c67e7c5d9f6029526a4eb/html5/thumbnails/18.jpg)
18Federal Office of Topography swisstopo Esri European User Conference, Salzburg
15 October 2015
Consistency Checks / Validation for
Raster and Vector Data (1)
![Page 19: The Geodata Warehouse of the Swiss Federal SDIConceptual View of the FSDI «Ingest once, use anywhere» ... FME INTERLIS Checker GDWH Services.EXE GDAL FGDB FGDB TIF XML ILI ITF XTF](https://reader031.fdocuments.net/reader031/viewer/2022011921/602c67e7c5d9f6029526a4eb/html5/thumbnails/19.jpg)
19Federal Office of Topography swisstopo Esri European User Conference, Salzburg
15 October 2015
Consistency Checks / Validation for
Raster and Vector Data (2)
![Page 20: The Geodata Warehouse of the Swiss Federal SDIConceptual View of the FSDI «Ingest once, use anywhere» ... FME INTERLIS Checker GDWH Services.EXE GDAL FGDB FGDB TIF XML ILI ITF XTF](https://reader031.fdocuments.net/reader031/viewer/2022011921/602c67e7c5d9f6029526a4eb/html5/thumbnails/20.jpg)
20Federal Office of Topography swisstopo Esri European User Conference, Salzburg
15 October 2015
Archivable Data Formats
Fundamental Factors supporting LTA
in the Geodata Warehouse FSDI
Consistency Checks
System Architecture
Checksums File DatesStructuresMapping…Daily, Weekly,
Monthly
StandardisedOpenDocumented
BackupMirroringAccess Control Inhouse Servers
Dual Data TierHuman Readable StructuresService oriented
Classic IT Measures
Long-term Availability
in the Geodata Warehouse
![Page 21: The Geodata Warehouse of the Swiss Federal SDIConceptual View of the FSDI «Ingest once, use anywhere» ... FME INTERLIS Checker GDWH Services.EXE GDAL FGDB FGDB TIF XML ILI ITF XTF](https://reader031.fdocuments.net/reader031/viewer/2022011921/602c67e7c5d9f6029526a4eb/html5/thumbnails/21.jpg)
21Federal Office of Topography swisstopo Esri European User Conference, Salzburg
15 October 2015
Geodata Warehouse FSDI
Tools and TechnologiesB
usi
ne
ssLo
gic
Tie
rD
ata
Tie
r
Filesystem Layer
Standardised Directory Structure
Standardised Data Formats,
e.g. model based
Consistency Checks
Accompanying documents
Aggregation Layer
DB-Access LayerTransformation
functionsConsistency
Checks
Services
GDWH Data Dictionary
Service
Geodata Delivery
Geodata Validation &
Ingest
Geodata Transformation
RepresentTransform
TIFFTXT, CSVPDF/AILI, ITFXML
ArcGIS DesktopFME DesktopGDAL LibraryREFRAME LibraryINTERLIS ToolsLAS ToolsESRI FGDB
ArcGIS ServerFME Server
C# .NetPythonJavaScript
![Page 22: The Geodata Warehouse of the Swiss Federal SDIConceptual View of the FSDI «Ingest once, use anywhere» ... FME INTERLIS Checker GDWH Services.EXE GDAL FGDB FGDB TIF XML ILI ITF XTF](https://reader031.fdocuments.net/reader031/viewer/2022011921/602c67e7c5d9f6029526a4eb/html5/thumbnails/22.jpg)
22Federal Office of Topography swisstopo Esri European User Conference, Salzburg
15 October 2015
Shop
ArcGIS Web Services
Geodata Warehouse FSDI
Service oriented and modular Elements (1)
ShopExport
Manager
Web UI
FilesystemLayer
Aggregation Layer
Geodatasets ◄►Products
ArcMap
GDWH Core
.EXE FME
GDAL
Cockpit
Win Cmd
Web UI
GDWH Web Services
Web UI
Fire
wall
Order Manager
Web UI
Cockpit
Win Cmd
Web UI
Web UIFME
Desktop
![Page 23: The Geodata Warehouse of the Swiss Federal SDIConceptual View of the FSDI «Ingest once, use anywhere» ... FME INTERLIS Checker GDWH Services.EXE GDAL FGDB FGDB TIF XML ILI ITF XTF](https://reader031.fdocuments.net/reader031/viewer/2022011921/602c67e7c5d9f6029526a4eb/html5/thumbnails/23.jpg)
23Federal Office of Topography swisstopo Esri European User Conference, Salzburg
15 October 2015
VectorRaster
BusinessLogic Tier
Data Tier
Services
Aggregation
Layer
Filesystem
Layer
ArcGIS Server
Geodata Warehouse FSDI
Service oriented and modular Elements (2)
Export Manager
GDWH Core
FME Server
INTERLIS Checker
GDWH Services
.EXE GDAL
FGDBFGDB
TIF XML ILI ITF
XTFCSV TXT XML
Web Server
LAS ToolsLASlib
LAS
LAS
• Additional software components to support specific data types can
be added if necessary, e.g. LAS or 3D
![Page 24: The Geodata Warehouse of the Swiss Federal SDIConceptual View of the FSDI «Ingest once, use anywhere» ... FME INTERLIS Checker GDWH Services.EXE GDAL FGDB FGDB TIF XML ILI ITF XTF](https://reader031.fdocuments.net/reader031/viewer/2022011921/602c67e7c5d9f6029526a4eb/html5/thumbnails/24.jpg)
24Federal Office of Topography swisstopo Esri European User Conference, Salzburg
15 October 2015
Next Steps and Conclusions
• The new Geodata Warehouse enables the
Swiss Federal Spatial Data Infrastructure to
provide an environment to preserve geodata
(sustainable / long-term availability), both for
swisstopo itself and other federal agencies
• Swisstopo is currently adapting the remai-
ning systems of the FSDI to achieve this
goal, including the processes
• A geodataset becomes more interesting with
increasing age no need to archive
![Page 25: The Geodata Warehouse of the Swiss Federal SDIConceptual View of the FSDI «Ingest once, use anywhere» ... FME INTERLIS Checker GDWH Services.EXE GDAL FGDB FGDB TIF XML ILI ITF XTF](https://reader031.fdocuments.net/reader031/viewer/2022011921/602c67e7c5d9f6029526a4eb/html5/thumbnails/25.jpg)
25Federal Office of Topography swisstopo Esri European User Conference, Salzburg
15 October 2015
The End