I’mBrian! · 14 Confiden)al"Informaon"of"Talksum,"Inc." Network"Monitoring"&"Op)mizaon"...
Transcript of I’mBrian! · 14 Confiden)al"Informaon"of"Talksum,"Inc." Network"Monitoring"&"Op)mizaon"...
![Page 1: I’mBrian! · 14 Confiden)al"Informaon"of"Talksum,"Inc." Network"Monitoring"&"Op)mizaon" Unix"Logs"–RFC3164"UDP/TCP" Ne[low"–UDP"–NG"v.5,"8,"9,"10" Refined"DataStream"](https://reader034.fdocuments.net/reader034/viewer/2022042314/5f01f4937e708231d401dc00/html5/thumbnails/1.jpg)
1 Confiden)al Informa)on of Talksum, Inc.
I’m Brian!
Who Is This Guy?
![Page 2: I’mBrian! · 14 Confiden)al"Informaon"of"Talksum,"Inc." Network"Monitoring"&"Op)mizaon" Unix"Logs"–RFC3164"UDP/TCP" Ne[low"–UDP"–NG"v.5,"8,"9,"10" Refined"DataStream"](https://reader034.fdocuments.net/reader034/viewer/2022042314/5f01f4937e708231d401dc00/html5/thumbnails/2.jpg)
2 Confiden)al Informa)on of Talksum, Inc.
• Principal Architect at Talksum
• Open Source Contributor - ZeroMQ
- Rsyslog
Who Is This Guy?
![Page 3: I’mBrian! · 14 Confiden)al"Informaon"of"Talksum,"Inc." Network"Monitoring"&"Op)mizaon" Unix"Logs"–RFC3164"UDP/TCP" Ne[low"–UDP"–NG"v.5,"8,"9,"10" Refined"DataStream"](https://reader034.fdocuments.net/reader034/viewer/2022042314/5f01f4937e708231d401dc00/html5/thumbnails/3.jpg)
3 Confiden)al Informa)on of Talksum, Inc.
Talksum Data Stream Router
Next Age of Data Management
November 2013
![Page 4: I’mBrian! · 14 Confiden)al"Informaon"of"Talksum,"Inc." Network"Monitoring"&"Op)mizaon" Unix"Logs"–RFC3164"UDP/TCP" Ne[low"–UDP"–NG"v.5,"8,"9,"10" Refined"DataStream"](https://reader034.fdocuments.net/reader034/viewer/2022042314/5f01f4937e708231d401dc00/html5/thumbnails/4.jpg)
4 Confiden)al Informa)on of Talksum, Inc.
The Talksum Data Stream Router
Apache Common Logging – Files SNMP -‐ UDP
Unix Logs – RFC3164 UDP/TCP Ne[low – UDP – NG v.5, 8, 9, 10
Pa)ent Records (HL7) XML/ASN.1 Transporta)on (BSM) SAE J2735
I2C, CAN, SNMP, Serial
XML, JSON, File, HTTP REST
Twiger, RSS, CAP (Weather Alerts)
Refined Data Stream
Refined Data Stream
Refined Data Stream
Indexed, Mapped, Reduced Ordered, Sorted Data Streams
Bulk Data Streams (Lightly Ordered and Filtered)
Talksum Data Stream
Router (TDSR)
• Data Normaliza)on
• Parsers • Filters • Metrics and Counts
• Inline ETL/PTL • Asynchronous Outputs
• Protocol Verifica)on
Customer A: Summarized Data
System Logs
Applica=on Data
Sensor and Industrial
Data
3rd Party Data B2B/M2M
Social and Public Data
Customer B: Aggregated Data
Customer C: Dynamic Stream
Applica=on Logs
• SQL Warehouse • Bulk Data Stores • File Storage
• Object Data Stores • Indexed Data Caches • NoSQL Data Warehouses
![Page 5: I’mBrian! · 14 Confiden)al"Informaon"of"Talksum,"Inc." Network"Monitoring"&"Op)mizaon" Unix"Logs"–RFC3164"UDP/TCP" Ne[low"–UDP"–NG"v.5,"8,"9,"10" Refined"DataStream"](https://reader034.fdocuments.net/reader034/viewer/2022042314/5f01f4937e708231d401dc00/html5/thumbnails/5.jpg)
5 Confiden)al Informa)on of Talksum, Inc.
We turn alphabet soup into coherent streams of informa)on so that you can get to work solving problems.
In Other Words…
![Page 6: I’mBrian! · 14 Confiden)al"Informaon"of"Talksum,"Inc." Network"Monitoring"&"Op)mizaon" Unix"Logs"–RFC3164"UDP/TCP" Ne[low"–UDP"–NG"v.5,"8,"9,"10" Refined"DataStream"](https://reader034.fdocuments.net/reader034/viewer/2022042314/5f01f4937e708231d401dc00/html5/thumbnails/6.jpg)
6 Confiden)al Informa)on of Talksum, Inc.
Cross Domain Rou)ng with ZeroMQ
November 2013
![Page 7: I’mBrian! · 14 Confiden)al"Informaon"of"Talksum,"Inc." Network"Monitoring"&"Op)mizaon" Unix"Logs"–RFC3164"UDP/TCP" Ne[low"–UDP"–NG"v.5,"8,"9,"10" Refined"DataStream"](https://reader034.fdocuments.net/reader034/viewer/2022042314/5f01f4937e708231d401dc00/html5/thumbnails/7.jpg)
7 Confiden)al Informa)on of Talksum, Inc.
“A territory over which rule or control is exercised”
“A sphere of ac)vity, concern, or func)on”
-‐ freedic)onary.com
Domains
![Page 8: I’mBrian! · 14 Confiden)al"Informaon"of"Talksum,"Inc." Network"Monitoring"&"Op)mizaon" Unix"Logs"–RFC3164"UDP/TCP" Ne[low"–UDP"–NG"v.5,"8,"9,"10" Refined"DataStream"](https://reader034.fdocuments.net/reader034/viewer/2022042314/5f01f4937e708231d401dc00/html5/thumbnails/8.jpg)
8 Confiden)al Informa)on of Talksum, Inc.
• Energy • Transporta)on
• Biomedical
• Informa)on Systems
Some Domains!
![Page 9: I’mBrian! · 14 Confiden)al"Informaon"of"Talksum,"Inc." Network"Monitoring"&"Op)mizaon" Unix"Logs"–RFC3164"UDP/TCP" Ne[low"–UDP"–NG"v.5,"8,"9,"10" Refined"DataStream"](https://reader034.fdocuments.net/reader034/viewer/2022042314/5f01f4937e708231d401dc00/html5/thumbnails/9.jpg)
9 Confiden)al Informa)on of Talksum, Inc.
• Applica)on Threads • Applica)on Processes
• Systems on a network
• Networks themselves
Some More Domains!
![Page 10: I’mBrian! · 14 Confiden)al"Informaon"of"Talksum,"Inc." Network"Monitoring"&"Op)mizaon" Unix"Logs"–RFC3164"UDP/TCP" Ne[low"–UDP"–NG"v.5,"8,"9,"10" Refined"DataStream"](https://reader034.fdocuments.net/reader034/viewer/2022042314/5f01f4937e708231d401dc00/html5/thumbnails/10.jpg)
10 Confiden)al Informa)on of Talksum, Inc.
ØMQ \zeromq\:
• A network socket library that acts as a concurreny framework
• Carries messages across mul)ple transports
• Enables networking pagerns • Provides Asyncronous I/O • Large and ac)ve open source community
• Bindings for 40+ programming languages
• Works on most opera)ng systems
What is ZeroMQ
![Page 11: I’mBrian! · 14 Confiden)al"Informaon"of"Talksum,"Inc." Network"Monitoring"&"Op)mizaon" Unix"Logs"–RFC3164"UDP/TCP" Ne[low"–UDP"–NG"v.5,"8,"9,"10" Refined"DataStream"](https://reader034.fdocuments.net/reader034/viewer/2022042314/5f01f4937e708231d401dc00/html5/thumbnails/11.jpg)
11 Confiden)al Informa)on of Talksum, Inc.
What is ZeroMQ
WHAT IF I TOLD YOU
WE CAN HAVE LOCK-FREE IN MEMORY MESSAGE PASSING BETWEEN THREADS?
Here there be dragons
![Page 12: I’mBrian! · 14 Confiden)al"Informaon"of"Talksum,"Inc." Network"Monitoring"&"Op)mizaon" Unix"Logs"–RFC3164"UDP/TCP" Ne[low"–UDP"–NG"v.5,"8,"9,"10" Refined"DataStream"](https://reader034.fdocuments.net/reader034/viewer/2022042314/5f01f4937e708231d401dc00/html5/thumbnails/12.jpg)
12 Confiden)al Informa)on of Talksum, Inc.
What Is ZeroMQ To Us
ZeroMQ is a tool which facilitates the construc)on of cross domain communica)on pagerns
![Page 13: I’mBrian! · 14 Confiden)al"Informaon"of"Talksum,"Inc." Network"Monitoring"&"Op)mizaon" Unix"Logs"–RFC3164"UDP/TCP" Ne[low"–UDP"–NG"v.5,"8,"9,"10" Refined"DataStream"](https://reader034.fdocuments.net/reader034/viewer/2022042314/5f01f4937e708231d401dc00/html5/thumbnails/13.jpg)
13 Confiden)al Informa)on of Talksum, Inc.
When we can apply a pagern from one domain to a different domain, we can do more with less work.
Pagerns!
![Page 14: I’mBrian! · 14 Confiden)al"Informaon"of"Talksum,"Inc." Network"Monitoring"&"Op)mizaon" Unix"Logs"–RFC3164"UDP/TCP" Ne[low"–UDP"–NG"v.5,"8,"9,"10" Refined"DataStream"](https://reader034.fdocuments.net/reader034/viewer/2022042314/5f01f4937e708231d401dc00/html5/thumbnails/14.jpg)
14 Confiden)al Informa)on of Talksum, Inc.
Network Monitoring & Op)miza)on
Unix Logs – RFC3164 UDP/TCP Ne[low – UDP – NG v.5, 8, 9, 10
Refined Data Stream
Refined Data Stream
Indexed, Mapped, Reduced Ordered, Sorted Data Streams
Bulk Data Streams (Lightly Ordered and Filtered)
Talksum Data Stream
Router (TDSR)
• Data Normaliza)on
• Parsers • Filters • Metrics and Counts
• Inline ETL/PTL • Asynchronous Outputs
• Protocol Verifica)on
Exis=ng BI Tools
System Logs
NOC Aler=ng
• SQL Warehouse • Bulk Data Stores • File Storage
• Object Data Stores • Indexed Data Caches • NoSQL Data Warehouses
Customer: Large European ISP/Email Communica)ons Provider Use Case: Ingest Ne[low data, parse and aggregate in real )me, monitors and alerts, op)mize network topology Status: Deploying beta appliance
![Page 15: I’mBrian! · 14 Confiden)al"Informaon"of"Talksum,"Inc." Network"Monitoring"&"Op)mizaon" Unix"Logs"–RFC3164"UDP/TCP" Ne[low"–UDP"–NG"v.5,"8,"9,"10" Refined"DataStream"](https://reader034.fdocuments.net/reader034/viewer/2022042314/5f01f4937e708231d401dc00/html5/thumbnails/15.jpg)
15 Confiden)al Informa)on of Talksum, Inc.
Financial
XML, JSON, File, HTTP REST
Twiger, RSS, CAP (Weather Alerts)
Refined Data Stream
Refined Data Stream
Indexed, Mapped, Reduced Ordered, Sorted Data Streams
Bulk Data Streams (Lightly Ordered and Filtered)
Talksum Data Stream
Router (TDSR)
• Data Normaliza)on
• Parsers • Filters • Metrics and Counts
• Inline ETL/PTL • Asynchronous Outputs
• Protocol Verifica)on
Aler=ng & No=fica=on
3rd Party Data Trading Desks
Social and Public Data
Market Dashboard
• SQL Warehouse • Bulk Data Stores • File Storage
• Object Data Stores • Indexed Data Caches • NoSQL Data Warehouses
Customer: Major Financial Stock Exchange Use Case: Ingest unstructured financial market data, parse and filter for quality, aggregate, integrate with exis)ng data warehouse Status: Acquiring data sample for POC
![Page 16: I’mBrian! · 14 Confiden)al"Informaon"of"Talksum,"Inc." Network"Monitoring"&"Op)mizaon" Unix"Logs"–RFC3164"UDP/TCP" Ne[low"–UDP"–NG"v.5,"8,"9,"10" Refined"DataStream"](https://reader034.fdocuments.net/reader034/viewer/2022042314/5f01f4937e708231d401dc00/html5/thumbnails/16.jpg)
16 Confiden)al Informa)on of Talksum, Inc.
Request Reply
![Page 17: I’mBrian! · 14 Confiden)al"Informaon"of"Talksum,"Inc." Network"Monitoring"&"Op)mizaon" Unix"Logs"–RFC3164"UDP/TCP" Ne[low"–UDP"–NG"v.5,"8,"9,"10" Refined"DataStream"](https://reader034.fdocuments.net/reader034/viewer/2022042314/5f01f4937e708231d401dc00/html5/thumbnails/17.jpg)
17 Confiden)al Informa)on of Talksum, Inc.
Round Robin Load Balancing
![Page 18: I’mBrian! · 14 Confiden)al"Informaon"of"Talksum,"Inc." Network"Monitoring"&"Op)mizaon" Unix"Logs"–RFC3164"UDP/TCP" Ne[low"–UDP"–NG"v.5,"8,"9,"10" Refined"DataStream"](https://reader034.fdocuments.net/reader034/viewer/2022042314/5f01f4937e708231d401dc00/html5/thumbnails/18.jpg)
18 Confiden)al Informa)on of Talksum, Inc.
Unicast
![Page 19: I’mBrian! · 14 Confiden)al"Informaon"of"Talksum,"Inc." Network"Monitoring"&"Op)mizaon" Unix"Logs"–RFC3164"UDP/TCP" Ne[low"–UDP"–NG"v.5,"8,"9,"10" Refined"DataStream"](https://reader034.fdocuments.net/reader034/viewer/2022042314/5f01f4937e708231d401dc00/html5/thumbnails/19.jpg)
19 Confiden)al Informa)on of Talksum, Inc.
Mul)cast
![Page 20: I’mBrian! · 14 Confiden)al"Informaon"of"Talksum,"Inc." Network"Monitoring"&"Op)mizaon" Unix"Logs"–RFC3164"UDP/TCP" Ne[low"–UDP"–NG"v.5,"8,"9,"10" Refined"DataStream"](https://reader034.fdocuments.net/reader034/viewer/2022042314/5f01f4937e708231d401dc00/html5/thumbnails/20.jpg)
20 Confiden)al Informa)on of Talksum, Inc.
Fair Queuing
![Page 21: I’mBrian! · 14 Confiden)al"Informaon"of"Talksum,"Inc." Network"Monitoring"&"Op)mizaon" Unix"Logs"–RFC3164"UDP/TCP" Ne[low"–UDP"–NG"v.5,"8,"9,"10" Refined"DataStream"](https://reader034.fdocuments.net/reader034/viewer/2022042314/5f01f4937e708231d401dc00/html5/thumbnails/21.jpg)
21 Confiden)al Informa)on of Talksum, Inc.
Explicit Rou)ng
![Page 22: I’mBrian! · 14 Confiden)al"Informaon"of"Talksum,"Inc." Network"Monitoring"&"Op)mizaon" Unix"Logs"–RFC3164"UDP/TCP" Ne[low"–UDP"–NG"v.5,"8,"9,"10" Refined"DataStream"](https://reader034.fdocuments.net/reader034/viewer/2022042314/5f01f4937e708231d401dc00/html5/thumbnails/22.jpg)
22 Confiden)al Informa)on of Talksum, Inc.
• Within a thread
• Between threads
• Between processes on the same system
• Between processes on different systems
Inter Domain Messaging
![Page 23: I’mBrian! · 14 Confiden)al"Informaon"of"Talksum,"Inc." Network"Monitoring"&"Op)mizaon" Unix"Logs"–RFC3164"UDP/TCP" Ne[low"–UDP"–NG"v.5,"8,"9,"10" Refined"DataStream"](https://reader034.fdocuments.net/reader034/viewer/2022042314/5f01f4937e708231d401dc00/html5/thumbnails/23.jpg)
23 Confiden)al Informa)on of Talksum, Inc.
Why Talksum Contributes to Zeromq
• We are all facing a lot of problems. • We believe in the value of empirical evidence. • Solu)ons emerge from all of us working together across our various domain boundaries.
• Par)cipate!
![Page 24: I’mBrian! · 14 Confiden)al"Informaon"of"Talksum,"Inc." Network"Monitoring"&"Op)mizaon" Unix"Logs"–RFC3164"UDP/TCP" Ne[low"–UDP"–NG"v.5,"8,"9,"10" Refined"DataStream"](https://reader034.fdocuments.net/reader034/viewer/2022042314/5f01f4937e708231d401dc00/html5/thumbnails/24.jpg)
24 Confiden)al Informa)on of Talksum, Inc.
• hgp://www.zeromq.org/ -‐ Main ZeroMQ Web Site
• hgp://zguide.zeromq.org/ -‐ ZeroMQ User Guide
• hgp://github.com/zeromq -‐ ZeroMQ Git Repository
• hgp://api.zeromq.org/ -‐ ZeroMQ Reference Manual
• hgp://zeromq.org/whitepapers:architecture – Architecture Whitepaper
• hgp://czmq.zeromq.org/ -‐ ZeroMQ C Binding
• hgp://zeromq.github.io/pyzmq/ -‐ ZeroMQ Python Binding
Resources