Smallworld Core Spatial Technology™ and FME: Configuration Tips and Tricks

Smallworld Core Smallworld Core Spatial Technology™ Spatial Technology™

and FME: and FME: Configuration Tips Configuration Tips

and Tricksand TricksPresented byAlfred Sawatzky

2007 Smallworld Users Conference

Learning ObjectivesLearning Objectives

Easily install the FME Application for Smallworld Core Spatial Technology™

Understand the unique-to-Smallworld Core Spatial Technology™ coordinate system configurations for FME

Find out how to automate data extracts from Smallworld Core Spatial Technology™ via FME using batch files

Safe Software has workshops to show you how to use their tools. This presentation will show you how to implement the Smallworld Core Spatial Technology™/FME interface based on real world lessons learned.

What is FME?What is FME?

Feature Manipulation Engine sold by Safe Software Allows users or processes to…

“Move data from multiple sources, reformat and/or cleanse it, and load it into other database(s), a data mart, a data warehouse, or other operational systems for analysis”

Core Capabilities Translate Transform Federate Distribute

TranslateTranslate

Translate data from/to over 190 CAD, GIS, vector, raster and database formats

Smallworld Core Spatial

Technology™

OGC

MapInfo

Autodesk

Intergraph

ESRI

Google

GeoRSS

TransformTransform

Data restructuring both in terms of attribution and geometry is key!

FederateFederate

Let users access data where it is and share as federated (unified) data view – no need to copy to a data store or put into a unified schema

DistributeDistribute

Distribute data to users in the structure and format users desire – and via the web technology they prefer

FME and Smallworld Core Spatial FME and Smallworld Core Spatial Technology™ : ExamplesTechnology™ : Examples

Visualization: Using FME as a Spatial Business Intelligence Visualization tool FME provides a nice visual tool (Workbench) to allow GIS data analysts to process spatial

data without knowing Smallworld Core Spatial Technology™/Smallworld Magik™. Examples:

• http://sworldwatch.blogspot.com/2007/01/fme-as-spatial-intelligence-tool-for.html• http://sworldwatch.blogspot.com/2007/04/using-fme-to-report-lines-lengths-per.html

Validation: Using FME to validate Smallworld Core Spatial Technology™ data Migration: Migrating from legacy system into Smallworld Core Spatial

Technology™ Dissemination: Distributing Smallworld Core Spatial Technology™ based

data into other formats.

How do FME and Smallworld Core Spatial How do FME and Smallworld Core Spatial Technology™ Interface?Technology™ Interface?

Need a FME installation and license from Safe Software. for Core Spatial Technology™ 4.x need a minimum of FME

2003 ICE and a maximum of FME 2007

How do FME and Smallworld Core Spatial How do FME and Smallworld Core Spatial Technology™ Interface?Technology™ Interface?

Need some Smallworld Magik™ code that interfaces Smallworld Core Spatial Technology™ with FME. Licensed depending on which vendor you purchase from. GE Energy

• 4.1 Translation Suite or FME 4.0. Earlier versions of FME on Smallworld Core Spatial Technology™ (for Core Spatial Technology™ 3.x) also available but they do not work with FME Workbench.

Spatial Business Solutions• FME Plugin• http://www.spatialbiz.com/pages/fme_plugin_spec.htm

Installing FMEInstalling FME

This presentation will focus on installing the interface provided by GE Energy in the Core Spatial Technology™ 4.1 SWAF environment.

Differences between 4.0 and 4.1 will be noted. Download and install FME 2007 from http://www.safe.com.

Be sure to specify the “Smallworld Core Spatial Technology™ Edition” when downloading the installer and requesting licenses.

There are a number of licensing options available to you.

Installing FME-Smallworld Core Spatial Installing FME-Smallworld Core Spatial Technology™ InterfaceTechnology™ Interface

(sworldswaf.dll)(sworldswaf.dll) Be sure you are using the correct version of sworldswaf.dll

in your FME_HOME directory (eg., c:\program files\fme) Smallworld Core Spatial Technology™ 4.0 FME Interface requires dll

version 1.0.1.0 (this is the one that currently ships with FME 2006 GB) Smallworld Core Spatial Technology™ 4.1 FME Interface requires dll

version 1.0.0.11 (this is the one that currently ships with FME 2007) Contact your GE Energy Help Desk for the correct DLL. See

http://sworldwatch.blogspot.com/2007/07/fme-41-translator-dll-problem.html for more details until GE Energy publishes a TSB for this issue.


The Smallworld Core Spatial Technology™ on FME interface is delivered as a Smallworld Magik™ product.

Typically install the product (fme410) in a folder next to the sw410 product folder (eg., c:\sw410\fme410)

You will need to make sure that the fme410\product.def file is known to your image (eg., add the full path to the SW_PRODUCTS_PATH environment variable)


Modify your image build module or some other suitable module to “require” these FME modules: fme_plugin fme_import_plugin fme_export_plugin


(SWAF FME Application)(SWAF FME Application) The FME 4.1 documentation instructs you to load the

swaf_fme_application module.sw_module_manager.load_module(:swaf_fme_application)

While this is useful for a quick demo it is not very helpful for integrating into your existing applications.

Customers say: “How do we integrate the FME functionality into our existing applications?”


How do I integrate the FME interface into my existing application(s)? module.def

• Modify the module.def for your application(s) to “require” fme_plugin fme_import_plugin fme_export_plugin


How do I integrate the FME interface into my existing application(s)? config.xml

• Modify <application path>\resources\base\data\config.xml and add the following plugin definition to the list of plugins.

<plugin name="fme" class_name="fme_plugin">

<properties>

<property name="port_number" value="30000"/>

</properties>

</plugin>


How do I integrate the FME interface into my existing application(s)? explorer_model_config.xml

• Modify a “suitable” (see next slide) resources\base\data\explorer_model_config.xml to include the following two entries:

<plugin name="fme_export" class_name="explorer_model_fme_export_plugin" />

<plugin name="fme_import" class_name="explorer_model_fme_import_plugin" />


How do I integrate the FME interface into my existing application(s)? explorer_model_config.xml

• “Suitable” depends on what the model_config_definition_* property settings are for your explorer plugin in the application’s resources\base\data\config.xml file. You can choose to modify the Explorer for only a single application or for many applications.

<plugin name="explorer" class_name="explorer_plugin">

<properties>

<property name="model_config_definition_file_name“ value="explorer_model_config.xml"/>

<property name="model_config_definition_module_name" value="swaf_fme_application"/>

</properties>

</plugin>


Restart your application after you have made all the XML changes.

Start up the Explorer tool. You should see two new FME folders at the top of the left pane.

Pull not PushPull not Push

First set up FME/Smallworld Core Spatial Technology™ interface as a data source to serve up certain tables/records on request.

Clickand

Drag

Hint For large “My Datasets” lists, the Click and Drag does not work as expected. Use Search Restrictions to Help you out.


Start FME tools to request data from the FME/Smallworld Core Spatial Technology™ data source


We have used FME Universal Viewer to show the connection between FME and Smallworld Core Spatial Technology™.

Once the connection is established we can just as easily use FME Workbench or FME Universal Translator to connect to Smallworld Core Spatial Technology™.

FME TICS and what that means for youFME TICS and what that means for you

At application startup, host port is assigned to FME TICS server (default 30000).

A port on a machine can be used by only one service. If your clients are running on Citrix or Terminal Server this will

cause a problem because you have many Smallworld Core Spatial Technology™/FME users using one machine. Only the first user to log in will be bound to a port.

Unless… You are using FME for Smallworld Core Spatial Technology™ 4.1; or You have some kind of customization for FME for Smallworld Core Spatial

Technology™ 4.0 that attempts to find an unused port for the FME TICS server.

• E.g., p4_1rpc.Smallworld Magik™p4_1rpc.magik

FME TICS and what that means for youFME TICS and what that means for you

If you have a patch that allows multiple FME on Smallworld Core Spatial Technology™ sessions to use different ports on Citrix or Terminal Server, you are confronted with another issue.

How can a Smallworld Core Spatial Technology™ user know which port to connect their FME tools to? Currently, the only way to do that is to inspect the command window for the current Smallworld Core Spatial Technology™ image and look for something like…

TICS server startedFME TICS Server Running on 30000

Not very user-friendly, but…

A few simple fixes can make it easierA few simple fixes can make it easier(part 1)(part 1)

_method explorer_model_fme_import_plugin.setup_explorer_elements()#### Sets up the explorer elements for this plugin.##_if .explorer_element _is _unset_then

# START CHANGE# (2007-03-16 Alfred Sawatzky, RPC Inc.) : #_local group_name << _self.message(:group_name)_local server << fme_tics_server.current_server_local group_name << _self.message(:group_name,

server.port_number, system.host_name, server.application_coordinate_system.external_cs_name)

## END CHANGE

# Create the top level group element for this plugin.explorer_element << explorer_group_element.new_with(

:owner, _self, :name, group_name, :image, { :explorer_fme,

:fme_engine }, :column_headings, { _self.message(:name_heading) }, :visible_field_names, { :name }, :column_widths, { 400 } )

_endif

_endmethod$read_message_patch(:explorer_model_fme_import_plugin,_unset,:en_us)$:group_name Import from FME (Port #1 on #2. Coordinate System '#3')$

A few simple fixes can make it easierA few simple fixes can make it easier(part 2)(part 2)

_method explorer_model_fme_export_plugin.setup_explorer_elements()#### Sets up the explorer elements for this plugin.##_if .explorer_element _is _unset_then

# START CHANGE# (2007-03-16 Alfred Sawatzky, RPC Inc.) : #_local group_name << _self.message(:group_name)_local server << fme_tics_server.current_server_local group_name << _self.message(:group_name,

server.port_number, system.host_name, server.application_coordinate_system.external_cs_name)

## END CHANGE

# Create the top level group element for this plugin.explorer_element << explorer_group_element.new_with(

:owner, _self, :name, group_name, :image, { :explorer_fme,

:fme_engine }, :column_headings, { _self.message(:name_heading) }, :visible_field_names, { :name }, :column_widths, { 400 } )

_endif

_endmethod$read_message_patch(:explorer_model_fme_export_plugin,_unset,:en_us)$:group_name Export to FME (Port #1 on #2. Coordinate System '#3')$

Start FME Universal TranslatorStart FME Universal Translator

For this example we will use the FME Universal Translator for a quick translation.

Lessons learned here about host, port and coordinate system are transferable to the FME Workbench and FME Universal Viewer.

Start FME Universal TranslatorStart FME Universal Translator

Coordinate SystemsCoordinate Systems

FME on Smallworld Core Spatial Technology™ interface uses the Application Coordinate System not the dataset, view or trail coordinate systems. This is an important distinction to remember!!

What if I can't find my Smallworld Core Spatial Technology™ Coordinate System in FME! It is not important that the Smallworld Core Spatial Technology™ and

FME coordinate systems have the exact same name. It is important that they have the same coordinate system definition. It is very likely that the application coordinate system that your

Smallworld Core Spatial Technology™ application is using will not be defined in the FME coordinate system gallery.

Comparing FME and Smallworld Core Comparing FME and Smallworld Core Spatial Technology™ Coordinate Spatial Technology™ Coordinate

SystemsSystems Start Smallworld Core Spatial Technology™

Coordinate System Editor


SystemsSystems Show Properties of the selected FME Coordinate

System


SystemsSystems Compare the two definitions to see if they are the

same. For help with FME coordinate system parameters see

http://www.safe.com/support/online-documentation/CoordSys/coordinate_system_help.htm

Defining a New FME Coordinate SystemDefining a New FME Coordinate System

Look at http://www.safe.com/support/online-documentation/CoordSys/coordinate_system_help.htm#adding_a_custom_coordinate_system.htm

Edit %FME_HOME%/reproject/MyCoordSysDefs.fme on your FME client. When you update to a newer version of FME, the

MyCoordSysDefs.fme file will not be overwritten. If you are working with custom mapping files and you set

coordinate system definitions, remember to remove the definitions once they are defined in the MyCoordSysDefs.fme file – the same coordinate system cannot be defined in both places.

Smallworld Core Spatial Smallworld Core Spatial Technology™/FME Coordinate System Technology™/FME Coordinate System

Problem!Problem! In some instances (eg., Cambridge DB), the application

coordinate system is Lat-Long with units = mm. Smallworld Core Spatial Technology™ automatically assigns a

Cartesian unit to an angular coordinate system for the application CS.

FME does not support Cartesian units for its LL projection. The only out-of-the-box way around this problem is to

modify the :units on the Smallworld Core Spatial Technology™ Application definition to be :degrees. But this can have other impacts (e.g., snapping tolerances)


Solution!Solution! It is only a matter of a little Smallworld Magik™ Add a new property to the fme_plugin class

called :coordinate_system_name Modify fme_plugin.application_coordinate_system Add new property to config.xml with a coordinate

system name that you know FME supports.


Solution (Smallworld Magik™)!Solution (Smallworld Magik™)!_pragma(classify_level=advanced)#### The name of the 'application_coordinate_system' to use when## starting the fme_tics_server. If no name is given or the## name is not valid, then the fme_tics_server will use the## coordinate_system of the current application.##fme_plugin.def_property( :coordinate_system_name,

:type, :string, :default_value, "" )

$_pragma(classify_level=restricted)_method fme_plugin.application_coordinate_system

## ## Returns the current application coordinate system##

>> _if ( app << _self.application ) _isnt _unset _then

# START CHANGE # (2007-08-16 Alfred Sawatzky, RPC Inc.) : #

_local db << app.database _local cs << app.database.coordinate_system(_self.coordinate_system_name)

>> cs.default(app.coordinate_system) #>> app.coordinate_system # # END CHANGE

_endif

_endmethod$


Solution (XML)!Solution (XML)! Modify <application path>\resources\base\data\config.xml

and add the new plugin property.

<plugin name="fme" class_name="fme_plugin">

<properties>

<property name="port_number" value="30000"/>

<property name="coordinate_system_name" value="world_longlat_wgs84_degree"/>

</properties>

</plugin>

Match

es th

e

FME

LL

Coordinate

System

Creating Batch ProcessesCreating Batch Processes

Many customers want to create batch processes that interface Smallworld Core Spatial Technology™ with FME.

First create and test a .fmw (or .fme) file to extract data from Smallworld Core Spatial Technology™.

Then write some Smallworld Magik™ code to launch FME (command line) with your .fmw (or .fme) file.

Creating Batch Processes (Smallworld Creating Batch Processes (Smallworld Magik™)Magik™)

_block_global fme_tics_server_local current_server

# start the fme_tics_server if not already running_if (current_server << fme_tics_server.current_server) _is _unset _orif _not current_server.acp_running?_then

fme_tics_server.start(_unset,_unset,:gis)

current_server << fme_tics_server.current_server_endif

# indicate which records to prepare for export_local export_records << rwo_set.new_from(gis_program_manager.cached_dataset(:gis).collections[:min_road])

current_server.set_output_rwo_set(export_records)

# run the command line FME_local command << {"fme","c:\temp\sworld2kml.fmw"}

system.do_command(command)_endblock$


If you created a .fmw file that had “Published Parameters”…


… you could write something like…_block

_global fme_tics_server_local current_server

# start the fme_tics_server if not already running_if (current_server << fme_tics_server.current_server) _is _unset _orif _not current_server.acp_running?_then

fme_tics_server.start(_unset,_unset,:gis)

current_server << fme_tics_server.current_server_endif

_local export_records << rwo_set.new_from(gis_program_manager.cached_dataset(:gis).collections[:min_road])

current_server.set_output_rwo_set(export_records)

_local command << {"fme","c:\temp\sworld2kml.fmw","--DestDataset_KML",'"c:\temp\cambridge_roads.kml"',"--PORT_SWORLDSWAF",current_server.port_number.write_string,"--COORDSYS","LL"}

system.do_command(command)_endblock$


Now that you have the Smallworld Magik™ batch steps, you could further wrap that in a .cmd or .bat file that: Starts Smallworld Core Spatial Technology™ Runs your FME batch script Closes Smallworld Core Spatial Technology™

FME on Smallworld Core Spatial FME on Smallworld Core Spatial Technology™ User ExtensionsTechnology™ User Extensions

The interface between Smallworld Core Spatial Technology™ and FME is very powerful once it has been configured correctly.

I have seen customers create: Batch FME output New GUIs that use the FME Engine under the hood Enhance the Smallworld Core Spatial Technology™ Explorer

“Export to…” functionality to include “Export to FME…”

FME on Smallworld Core Spatial FME on Smallworld Core Spatial Technology™ User ExtensionsTechnology™ User Extensions

AssistanceAssistance

http://www.safe.com/support/online-documentation/index.php

http://www.fmepedia.com an information commons provided by Safe Software Inc. for

the FME User Community. http://groups.google.com/group/fmetalk http://finance.groups.yahoo.com/group/fme/

No more posts allowed as of August 16, 2007 but is still available for you to search archives.

[email protected]

Questions?Questions?

[email protected] http://sworldwatch.blogspot.com

Smallworld Core Spatial Technology™ and FME: Configuration Tips and Tricks

Documents

Transcript of Smallworld Core Spatial Technology™ and FME: Configuration Tips and Tricks