Ultimate Real-Time Monitor Anything, Update Anything
FME Server Notification Service lets you act on events as they happen, and send information as it becomes available. Publications and Subscriptions let you publish to and monitor different systems and data.
Notifications What they are for
What they are not for
Notifications What they are for
ü A brief message, usually to trigger an action.
What they are not for x Transmitting large amounts of
spatial data.
Notifications What they are for
ü A brief message, usually to trigger an action.
ü Triggering an FME Server response to an event that happened outside of FME.
What they are not for x Transmitting large amounts of
spatial data. x Triggering an FME Server
response to a continuous series of messages (many per second).
Notifications What they are for
ü A brief message, usually to trigger an action.
ü Triggering an FME Server response to an event that happened outside of FME.
ü Sending a message about something that happened in FME.
What they are not for x Transmitting large amounts of
spatial data. x Triggering an FME Server
response to a continuous series of messages (many per second).
x Sending more than one message per second about what’s happening in FME.
FME Server Notification Service
See the list of what built-in publications and subscribers can be: http://fme.ly/protocols
Workflows for “Ultimate Real-Time”
Poll it Push it Update it
Poll it “Are we there yet?”
Polling Workflow 1. Change detection
○ GeoRSS reader ○ Caching ○ Timestamps
2. Run the workspace on a schedule
Timestamp
Scheduling (FME Server)
GeoRSS Reader
GeoRSS Reader
• Contains change detection logic
• Option to only read new features
Caching
• Whole datasets or timestamps
• ChangeDetector
• SQL - match on ID
ChangeDetector Example
SQL to find different features: @Value(uuid) = "uuid" AND @Value(aqi) != "aqi"
SQL Example
Timestamps
● Cache the timestamp
● Last updated attribute
● Filter features newer than timestamp
● Add a timestamp attributes to your dataset
Reads last cached time and replaces with current time
Checks against timestamp and filters out old features
Pros and Cons Caching whole datasets
ü Easy to set up and detect changes.
x Need to store entire dataset - processing time?
Caching timestamps
ü No storage needed. x Harder to set up -
variables, datetimes.
Consider: Does my dataset have a timestamp? Can I rely on the timestamp to indicate change?
Story: Real-Time Traffic
✓ Poll Waze feed
✓ Parse XML
✓ Filter
✓ Transform
✓ Validate
✓ Email notifications
Workflows for “Ultimate Real-Time”
Poll it Push it Update it
Push It
Pushing Methods • Database triggers • Webhooks
In FME Server:
• Direct URL • REST API
Database Triggers
● Databases can trigger events.
● Functions with HTTP
requests can push data to FME Server.
Database Triggers
Webhooks
● i.e. Give systems/services a URL that will respond.
● Send the notification to
FME Server. ● Queue the received
notifications and process the requests on a separate thread.
Pros and Cons Database Triggers
ü Pushes right to FME Server topic.
ü Can push lots of changes to a holding table.
ü Perfect for real-time database changes.
x Database permissions.
Webhooks
ü No need to waste effort polling.
ü Truly real time. x Complex to set up. x Not all systems provide
or accept webhooks.
Direct URL Systems can use HTTP to push data to topics or trigger workspaces to run.
Topic POST URL
REST API In-depth interaction
with FME Server
● Send/receive notifications ● Run jobs ● Manage connections ● Manage users ● Licensing ● Backups ● Scheduling ● Etc!
Workflows for “Ultimate Real-Time”
Poll it Push it Update it
To Poll or Push?
Poll vs. Push Considerations ● Simplicity vs. “real-time”ness ● Can the system push to FME Server? ● Are jobs too long for polling? ● API restrictions for polling?
Tip: Use FME Server for internal systems and FME Cloud for external systems.
Update It
Updating Methods
• FME Writers o Bulk drop/load/overwrite o Update option
• SQLExecuter • HTTPCaller
FME Writers: UPDATE operation
INSERT, UPDATE or DELETE in the middle of a workflow.
POST data or changes to systems.
Summary
• Polling workflow: o Change detection + scheduling
• Pushing options: o Database triggers, webhooks o FME Server: URL or REST API
• Updating: o Operations in writer parameters o SQLExecutor o HTTPCaller
Questions?
Database triggers tutorial: fme.ly/dbtriggers
Webhooks etc:
fme.ly/web
REST API documentation: fme.ly/restv3
Top Related