FME World Tour 2017: les possibilités de FME Desktop et FME Server
FME Extensive Usage Inside the Mapping Production System of Natural Resources Canada
-
Upload
safe-software -
Category
Technology
-
view
4.740 -
download
2
description
Transcript of FME Extensive Usage Inside the Mapping Production System of Natural Resources Canada
1
FME within NRCan
Marie-Eve Martin Jean-Marc Prévost
2
Agenda
! Context ! System architecture ! Programming standards, Quality assurance,
testing methods, etc. ! Validations processes and dependencies ! FME developments at NRCan ! Safe Software collaboration
3
Agenda
! Context ! Who are we? ! Where do we come from? ! Where are we now? ! Where are we going?
! System architecture ! Programming standards, Quality assurance, testing methods, etc. ! Validations processes and dependencies ! FME developments at NRCan ! Safe Software collaboration
4
Natural Resources Canada Mission and Outcomes
5
NRC
an
Earth Science Sector
Geoma4cs Canada
CCMEO / CCCOT
Energy Sector
Canadian Forest Service
Direc4on & Coordina4on
Minerals & Metal Sector
Science & Policy Integra4on
Energy Technology & Programs Sector
Corporate Management & Services Sector
NRCan structure overview
CCMEO Canada Centre for Mapping and Earth Observa5on
CCCOT Centre Canadien de Cartographie et d’ Observa5on de la Terre
CCMEO total number of employees: 319 227 employees for Mapping 92 employees for Earth Observa5on
Sherbrooke
OGawa
6
Agenda
! Context ! Who are we? ! Where do we come from? ! Where are we now? ! Where are we going?
! System architecture ! Programming standards, Quality assurance, testing methods, etc. ! Validations processes and dependencies ! FME developments at NRCan ! Safe Software collaboration
7
FME history at CCMEO
NTDB specs
vectoriza4on File management
CanVec, CanTopo, GeoBase, etc. era Database management
FGP 2015 2010 2005 2000 1995 1990 1985
Transla4ons -‐ Workbench
-‐ Data manipula4on & valida4on
Custom transformers
-‐ Programming standards
-‐ Tes4ng methodology
≈700
8
Agenda
! Context ! Who are we? ! Where do we come from? ! Where are we now? ! Where are we going?
! System architecture ! Programming standards, Quality assurance, testing methods, etc. ! Validations processes and dependencies ! FME developments at NRCan ! Safe Software collaboration
9
Distribu?on Management
System architecture at CCMEO
Aquisi?on
Cond
i4on
ning
Partners
Data acquisition
Others
Data Access QA
10
What do we produce ?
! Main products : ! Canvec ! Toporama ! CDED ! 200 collections of various nature
! Statistics for last year : ! 8 500 000 downloads ! 3 000 mail and phone requests
11
Where to find our products ?
! Data availability : ! Geogratis : geogratis.gc.ca ! Geobase : www.geobase.ca ! WMS services
! Licence : Open Government Licence
12
Data extraction tool http://geogratis.gc.ca/site/eng/extraction
13
Agenda
! Context ! Who are we? ! Where do we come from? ! Where are we now? ! Where are we going?
! System architecture ! Programming standards, Quality assurance, testing methods, etc. ! Validations processes and dependencies ! FME developments at NRCan ! Safe Software collaboration
14
15
Agenda
! Context ! System architecture ! Programming standards, Quality assurance,
testing methods, etc. ! Validations processes and dependencies ! FME developments at NRCan ! Safe Software collaboration
16
System architecture
FME, Python, .EXE, SQL, etc.
Windows servers Produc4on: 64 processes (32bits) Distribu4on: 56 processes (64bits)
Oracle 11g, PostgreSQL 9.2 ArcSDE 10.1, PostGIS 2.0
400 TB
Instruc?on files
17
Agenda
! Context ! System architecture ! Programming standards, Quality assurance,
testing methods, etc. ! Validations processes and dependencies ! FME developments at NRCan ! Safe Software collaboration
18
Worbenches – Last saved build
Programs s4ll in opera4on !
19
Taking control
! 2007 problematic: ! Mix of FME versions on computers ! Everyone wanted the lasted build ! Increasing number of workbenches ! Workbenches were transferred to support team
! Solution : ! FME upgrades management ! Programming standards
20
FME upgrades management
! Upgrade rules: ! Everyone use one common build ! Upgrade only to official builds ! At least one upgrade per year
(idealy no more than two)
21
Programming standards
22
Programming standards
! Documentation (.docx and annotations) ! Parameter naming standards ! Known bad practices review ! Unit tests ! Peer review
23
Programming standards Examples of bad practices
! Missing Rounders ! Missing Terminators
24
Programming standards Unit tests
! Each workbench is managed in a source code control environment
! Each workbench is saved with the bat file, sources and expected results files necessary to test it
! Batch testing of all workbenches before every upgrade.
25
Programming standards Unit tests
! Fail rate is used to plan upgrades
Source Destination Fail rate
2013 SP1 2013 SP4 2%
2013 SP4 2014 SP1 10%
26
Programming standards Peer review
! FME is easy to learn without taking classes ! FME always offer various way of doing
something
! Peer review allows: ! Improved shared knowledge ! Programming uniformity ! Identifying repetitive patterns
27
Agenda
! Context ! System architecture ! Programming standards, Quality assurance,
testing methods, etc. ! Validations processes and dependencies ! FME developments at NRCan ! Safe Software collaboration
28
Validation processes
! At CCMEO, FME is used for : ! Feature translation ! Feature creation ! Feature validation
Catalog
29
Distribu?on Management
Validation processes
Aquisi?on
Cond
i4on
ning
Partners
Data acquisition
Others
Data Access QA
30
Validation Catalog
! ISO-19110 compliant
XML HTML Database
Users
31
Catalog based validations
! Spatial relations validation ! Domain attribute validation ! Proximity validation ! Minimal dimension validation ! Segmentation validation ! Data clipping validation
32
Catalog based validations Example
Spatial relations validation ! Based on Egenhofer-Clementini DE-9IM masks
33
Catalog based validations Examples
! Spatial relations
! Attribute value
34
Catalog validation within FME
ISO 19110 XML
List friendly format
Generic reader
Generic writer
Valida4on Error
forma4ng Valida4on rules
extrac4on Valida4on
Error forma4ng
Valida4on rules extrac4on
One custo
m transfor
mer
per valida
5on conc
ept!
35
Catalog validation within FME
! Real program example:
36
Agenda
! Context ! System architecture ! Programming standards, Quality assurance,
testing methods, etc. ! Validations processes and dependencies ! FME developments at NRCan ! Safe Software collaboration
37
Utility customs - Examples
! TerminatorRecorder ! Record all the errors to a .ffs file before causing the workbench to fail
! TerminatorMinMax ! Cause the workbench to fail if
the number of feature is not as expected
38
! Matcher_CCMEO ! Match features if
start/end points are within tolerance
! SnapToNeatline ! Snap features within tolerance without
collapsing
Utility customs - Examples
39
Utility customs - Examples
! SchemaMapper_CCMEO ! Similar to SchemaMapper but : ! Allow mapping with a variable number of
conditions ! Conditions may be ranges of values ! Conditions may use any operator (>, <, != , etc) ! Conditions are validated and produce report if
badly formatted
TYPE=[11:13];PRECISION<=20;NAME=Lake
40
Validation customs Minimal dimension
Voronoi
Side lines
Centerline
Area Width Length
41
Validation customs Proximity
What are we looking for?
Valida4on Process
Op4misa4on Process
42
Agenda
! Context ! System architecture ! Programming standards, Quality assurance,
testing methods, etc. ! Validations processes and dependencies ! FME developments at NRCan ! Safe Software collaboration
43
Safe Software collaboration
! Motivations for collaboration ! Simplifies our workbenches ! Faster processing ! Maintenance under Safe’s responsibility
44
Safe Software collaboration Generalization
Hydro network
AngularityCalculator
SherbendGeneralizer
StreamOrderCalculator
StreamPriorityCalculator
NetworkFlowOrientor
BoundingBoxAccumulator
45
Safe Software collaboration
VoronoiDiagrammer
47
Amalgamator
48
ConvexityFilter
49
Generalizer Wang The Wang algorithm will itera5vely combine, eliminate and exaggerate bends un5l the input line feature has no bend that is smaller than the given tolerance value.
McMaster / McMaster Weighted Distance The McMaster algorithm calculates a new loca5on for each point by first taking the average value of the x and y coordinates of the point and a number of neighboring points. It then slides the averaged point towards the original point according to a specified displacement value. The overall effect is that each point will be pulled towards its neighboring points.
NURBfit (B-‐Spline) The NURBfit algorithm will fit lines using B-‐Spline curves of given polynomial degree. The resul5ng lines will follow these curves with given segment length. The higher the degree, the smoother the line. An example of usage is smoothing contour lines in order to remove sparks and simulate the work of a cartographic cra[sman.
Inflec?on Points The Inflec5on algorithm will calculate the loca5on of the inflec5on points along a line and return the list of these points. Inflec5on points are measures of the sinuosity of a line.
50
SherbendGeneralizer SherbendGeneralizer
51
NetworkFlowOrientor NetworkFlowOrientor
52
StreamPriorityCalculator
StreamPriorityCalculator
53
Densifier
54
Chopper
55
Chopper use case scenario
chopper
dissolve
chopper chopper
dissolve
dissolve
Data extraction
Group by metadata No group by