Ocean Data Interoperability Platform - Big Data - Streams & Workflows
-
Upload
adam-leadbetter -
Category
Science
-
view
255 -
download
2
Transcript of Ocean Data Interoperability Platform - Big Data - Streams & Workflows
ODIP II – Data Streams and FlowsSeptember 2015
Adam Leadbetter, Damian Smyth, Rob [email protected]
http://www.confluent.io/blog/apache-kafka-samza-and-the-unix-philosophy-of-distributed-data
20th Century 21st Century
Single machine Distributed processing
One-to-one communication Publish-subscribe pattern
No fault tolerance Replication, auto-recovery
Fixed schema, encoding Schema management, evolvable encoding
20th Century 21st CenturyByte stream Messages
In-memory Durable
Blocking / backpressure Buffering
One-to-one Multi-subscriber
https://github.com/peterataylor/om-json
Further Work
• Look at the architectures for reprocessing historic data
• Incorporate Sensor Web Enablement, OM-JSON
• Deploy on vessels / mobile, remote platforms
• Investigation of Apache NiFi