Using Topic-Based Publish/Subscribe for Managing Real Time GPS Streams
-
Upload
felix-serrano -
Category
Documents
-
view
32 -
download
0
description
Transcript of Using Topic-Based Publish/Subscribe for Managing Real Time GPS Streams
Using Topic-Based Publish/Subscribe for Managing Real Time GPS Streams
Marlon Pierce, Galip Aydin, Zhigang QiCommunity Grids Lab
Indiana University
1
California Real Time Network
Network Data Rates Message Format
Time RYO ASCII GML
CRTN GPS Site Positions(9 Stations)
1 second 1.5KB 4.03KB 48.7KB
1 hour 5.31MB 14.18MB 171.31MB
1 day 127.44MB 340.38MB 4.01GB
1 month 3.8GB 9.97GB 123.3GB
1 year 45.8GB 119.67GB 1.41TB
Entire SCIGN Network (250
stations)1year 1.23TB 16.18TB 160TB
Continuous GPS Stations (CGPS) are depicted as triangles while the Real-Time stations are represented as circles. Image is obtained from SOPAC GPS Explorer at http://sopac.ucsd.edu/projects/realtime
How does one manage all the data generated by the 85 stations? How can you get just the data you want?
Note this is fundamentally different from traditional request/response style Web Services.
RDAHMM: GPS Time Series SegmentationSlide Courtesy of Robert Granat, JPL
Complex data with subtle signals is difficult for humans to analyze, leading to gaps in analysis
HMM segmentation provides an automatic way to focus attention on the most interesting parts of the time
GPS displacement (3D) length two years.
Divided automaticallyby HMM into 7 classes.
Features:• Dip due to aquifer
drainage (days 120-250)
• Hector Mine earthquake (day 626)
• Noisy period at end of time series
Use Case - GPS Sensors GPS is used to identify long-term tectonic deformation and static
displacements. SCIGN has 250 Real-Time GPS Stations. SOPAC GPS networks:
8 networks for 80 stations produce 1Hz high resolution data. Socket based real-time binary-RYO format access is available. We developed filters to provide multiple format (RYO, ASCII, GML) real-
time streaming access. OHIO principle (a general principle required by DOD) and chain of filters.
Our Architecture Uses publish/subscribe based NaradaBrokering for managing real-time
GPS streams Utilizes topics for hierarchical organization of the sensors Deploys successive data filters ranging from format translators to data
analysis codes Could potentially be used to run RDAHMM clones to monitor state
changes in the entire GPS network We are partner in a pioneering project to use the real-time GPS data for
the first time in this context.
4
Processing Real-Time GPS Streams
5
Raw Data
70107010
70117011
70127012
RYOPorts
NB Server
ScrippsRTD
Server
ScrippsRTD
Server
Raw Data
A Complete Sensor Message Processing Path, including a data analysis application.
GPS Networks
Application Integration with Real-Time Filters
Station Monitor Filter records real-time positions for 10 minutes and calculates position changes
Graph Plotter Application creates visual representation of the positions.
RDAHMM Filter records real-time positions for 10 minutes and invokes RDAHMM application which determines state changes in the XYZ signal.
Graph Plotter Application creates visual representation of the RDAHMM output.
6
GIS Grid Usage Model – Earthquake Science Supporting geophysical repositories and real-time
sensors is essential To analyze a typical earthquake it is important to access
to precise measurements of the initial earthquakes and aftershocks
To support earthquake forecasting and the time and spatial positions of the forecasts PI can be used with existing data RDAHMM can be used with the real-time data
Earth Science field is moving from a previously data poor field to a data rich world. We will have thousands of sensors spread around the world. (i.e. GPS sensors, InSAR satellites)
7
Recording and Replaying Sensor Streams
Filters can be used to record and replay scenarios, such as Earthquakes in GPS case.
We developed RYO Recorder and RYO Publisher Filters.
The RYO Recorder creates daily archives of the GPS Streams.
RYO Publisher can be used to play daily or certain segments of the records.
We replayed the 2004 Southern California Earthquake using Parkfield GPS network archive
These filters are used in the performance and scalability tests.
8
SensorGrid Performance Tests Two Major Goals: System Stability and
Scalability Ensuring stability of the distributed Filter Services
for continuous operation. Finding the maximum number of publishers
(sensors) and clients that can be supported with a single broker.
Investigate if system scales for larger number of sensors and clients.
9
Test Methodology
The test system consists of a NaradaBrokering server and a three-filter chain for publishing, converting and receiving RYO messages.
We take 4 timings to calculate mean end-to-end delivery times of GPS measurements.
The tests were run at least for 24 hours. GridFarm001-008 servers are used in these tests.
Ttransfer = (T2 – T1) + (T4 – T3)
10
NB Server
1
1 2
3
4
1- System Stability Test The basic system with
three filters and one broker.
The figure shows average results for every 30 minutes.
The average transfer time shows the continuous operation does not degrade the system performance.
11
2 – Multiple Publishers Test
We add more GPS networks by running more publishers. The results show that 1000 publishers can be supported with no
performance loss. This is an operating system limit.12
Topic 1A
Topic 1B
Topic 2
Topic n
3 – Multiple Clients Test
We add more clients by running multiple Simple Filters which subscribe to the same ASCII topic.
The system can support as many as 1000 clients with very low performance decrease.
13
Topic 1A
Topic 1B
1000 Clients
Extending Scalability
The limit of the basic system appears to be 1000 clients or publishers.
This is due to an Operating System restriction of open file descriptors (1024 for Red Hat Linux) which can be increased by changing OS parameters.
To overcome this limit we create NaradaBrokering networks with linking multiple brokers. NB supports scalable linkage of the brokers for building tree like architectures.
We run 2 brokers to support 1500 clients. Number of brokers can be increased indefinitely, so we
can potentially support any number of publishers and subscribers.
14
4 – Multiple Brokers Test NaradaBrokering allows creation of
Broker networks. We create a two-broker network. Messages published to first broker can
be received from the second broker. We take timings on each broker. We connect 750 clients to each broker
and run for 24 hours. We chose 750 clients to stay well below the saturation limit.
The results show that the performance is very good and similar to single broker test.
15
NB Server
1
NB Server
2
Topic 1A
Topic 1B
Topic 1B
NB Server
2
4 – Multiple Brokers Test
16
Real-Time Filters Test Results
The RYO Publisher filter runs at 1Hz and publishes 24-hour archive of the CRTN_01 GPS network, which contains 9 GPS stations.
The single broker configuration can support 1000 clients or publishers (GPS networks - 9000 individual stations).
The system can be scaled up by creating NaradaBrokering broker networks.
Message order was preserved in all tests.
17
Acknowledgement Galip Aydin: Designer and implementer ZhiGang Qi: SensorGrid Performance Tests We thank Prof. Yehuda Bock and his group at SIO for
their help with real-time GPS data streams. Robert Granat, JPL: RDAHMM developer
18