Event 5 Property Crimes LD23- Crimes in Progress Building ...
Progress Apama & Event Processingcomplexevents.com/slides/Event_Processing_Symposum... · Progress...
-
Upload
truongkhue -
Category
Documents
-
view
221 -
download
0
Transcript of Progress Apama & Event Processingcomplexevents.com/slides/Event_Processing_Symposum... · Progress...
![Page 1: Progress Apama & Event Processingcomplexevents.com/slides/Event_Processing_Symposum... · Progress Apama & Event Processing ... Visualize Key Business Conditions and Actions in Real](https://reader036.fdocuments.net/reader036/viewer/2022062907/5aaac7c67f8b9a86188e7a09/html5/thumbnails/1.jpg)
Progress Apama & Event Processing
Mark Palmer, Vice President of Event Processing
![Page 2: Progress Apama & Event Processingcomplexevents.com/slides/Event_Processing_Symposum... · Progress Apama & Event Processing ... Visualize Key Business Conditions and Actions in Real](https://reader036.fdocuments.net/reader036/viewer/2022062907/5aaac7c67f8b9a86188e7a09/html5/thumbnails/2.jpg)
2
Agenda (based on Symposium Guidelines)
� Major Characteristics of the Progress Approach
� Usage Scenarios
� Major Trends & Roadmap for EP
� Major Challenges for Community
![Page 3: Progress Apama & Event Processingcomplexevents.com/slides/Event_Processing_Symposum... · Progress Apama & Event Processing ... Visualize Key Business Conditions and Actions in Real](https://reader036.fdocuments.net/reader036/viewer/2022062907/5aaac7c67f8b9a86188e7a09/html5/thumbnails/3.jpg)
3
About Progress Apama
� About Progress Software– $400M+ software company– Based in Bedford, MA– Sonic Software, Actional, Neon, Apama
� Apama + Progress Real Time – Apama founded by Dr. John Bates and Dr. Giles Nelson
in 1999– Combined with Progress data streams management
team
� Progress Apama Event Stream Processing Platform– Event processing engine– Event data streams management– Event visualization– Event adapters– Event language development tools– Vertical solutions
![Page 4: Progress Apama & Event Processingcomplexevents.com/slides/Event_Processing_Symposum... · Progress Apama & Event Processing ... Visualize Key Business Conditions and Actions in Real](https://reader036.fdocuments.net/reader036/viewer/2022062907/5aaac7c67f8b9a86188e7a09/html5/thumbnails/4.jpg)
4
3 Challenges for This Group
1) Characterize Event Processing (We Use ESP)
– Customer / usage orientation; not pure technical
– Define the Event Processing taxonomy & glossary
– Start with Roy’s Model: Simple, Mediated, BPM-Enabled, Complex (?)
2) Define EP’s Relationship to BAM
– Does the “M” stand for “Monitoring” or “Management”?
– Dashboards + Event Rules + Event Data Management = SuperBAM
3) Reconcile Current EP Approaches and Standardize Language
– SQL-based approach
– Language-based approach
– EAI-based approach
![Page 5: Progress Apama & Event Processingcomplexevents.com/slides/Event_Processing_Symposum... · Progress Apama & Event Processing ... Visualize Key Business Conditions and Actions in Real](https://reader036.fdocuments.net/reader036/viewer/2022062907/5aaac7c67f8b9a86188e7a09/html5/thumbnails/5.jpg)
5
Event Stream Processing: “When 3 credit card authorizations for the same card occur in any 5 second
window, deny the request and check for fraud.”
Event Stream Processing: “When 3 credit card authorizations for the same card occur in any 5 second
window, deny the request and check for fraud.”
Event Stream Processing (ESP)A New Computing Physics
time
1 2 3 4 5 6 7 8 9
Static Data Processing: “How many fraudulent credit card transactions
occurred last week?”
Static Data Processing: “How many fraudulent credit card transactions
occurred last week?”
![Page 6: Progress Apama & Event Processingcomplexevents.com/slides/Event_Processing_Symposum... · Progress Apama & Event Processing ... Visualize Key Business Conditions and Actions in Real](https://reader036.fdocuments.net/reader036/viewer/2022062907/5aaac7c67f8b9a86188e7a09/html5/thumbnails/6.jpg)
6
event streamsIBM
Acquires …
Monitor
IBM Trades
Analyze
Act
Buy IBMBuy IBM
Event Processing in Algorithmic TradingMonitor Multiple Streams of Events, Analyze for Patterns and Act in Real Time
IBM Trades
IBM Trades
IBM Trades
NewsNews
Market dataMarket data
![Page 7: Progress Apama & Event Processingcomplexevents.com/slides/Event_Processing_Symposum... · Progress Apama & Event Processing ... Visualize Key Business Conditions and Actions in Real](https://reader036.fdocuments.net/reader036/viewer/2022062907/5aaac7c67f8b9a86188e7a09/html5/thumbnails/7.jpg)
7
ALL WITHINany 2 minute time period
time
real
-tim
e da
ta s
tream
s
Trading Rule
! ! ! !
AND (HPQ’s price moves up by 5%
! ! ! !
WHENMSFT price moves outside 2% of MSFT-15-minute-VWAP
! ! ! !
ORMSFT’s price moves down by 2%
)) !
! ! !
FOLLOWED-BY (S&P moving by 0.5%
THENBUY MSFTSELL HPQ
NASDAQ
NYSE
MSFT 15-MIN-VWAP
S&P500
• complex event sequences
An ESP Algorithmic Trading Rule
• multiple data streams
• real-time constraints
• automated actions• pattern abstraction
• temporal constraints
![Page 8: Progress Apama & Event Processingcomplexevents.com/slides/Event_Processing_Symposum... · Progress Apama & Event Processing ... Visualize Key Business Conditions and Actions in Real](https://reader036.fdocuments.net/reader036/viewer/2022062907/5aaac7c67f8b9a86188e7a09/html5/thumbnails/8.jpg)
8
Agenda
� Major Characteristics of the Progress Approach
� Usage Scenarios
� Major Trends & Roadmap for EP
� Major Challenges for Community
![Page 9: Progress Apama & Event Processingcomplexevents.com/slides/Event_Processing_Symposum... · Progress Apama & Event Processing ... Visualize Key Business Conditions and Actions in Real](https://reader036.fdocuments.net/reader036/viewer/2022062907/5aaac7c67f8b9a86188e7a09/html5/thumbnails/9.jpg)
9
Algorithmic TradingAutomated trading based on market movement
Within any 20 second window, when HP rises by more than 2%, and IBM doesn’t, buy IBM.
![Page 10: Progress Apama & Event Processingcomplexevents.com/slides/Event_Processing_Symposum... · Progress Apama & Event Processing ... Visualize Key Business Conditions and Actions in Real](https://reader036.fdocuments.net/reader036/viewer/2022062907/5aaac7c67f8b9a86188e7a09/html5/thumbnails/10.jpg)
10
Real-Time Risk MitigationCalculate VaR in real-time and adjust real-time action to adjust
“When trading brings peso value-at-risk within 1% of risk level cap, lower offer prices for peso FX trading until risk level returns outside of 3% of today’s cap.”
ESP allows risk mitigation to shift to
front office apps - pre-trade - so errors are
eliminated before they occur
![Page 11: Progress Apama & Event Processingcomplexevents.com/slides/Event_Processing_Symposum... · Progress Apama & Event Processing ... Visualize Key Business Conditions and Actions in Real](https://reader036.fdocuments.net/reader036/viewer/2022062907/5aaac7c67f8b9a86188e7a09/html5/thumbnails/11.jpg)
11
Transportation: Security & Fraud DetectionDetect patterns among events to discover fraudulent activity
When a single ID card is used to gain entry twice in less than 10 seconds alert security
for piggybacking
!
![Page 12: Progress Apama & Event Processingcomplexevents.com/slides/Event_Processing_Symposum... · Progress Apama & Event Processing ... Visualize Key Business Conditions and Actions in Real](https://reader036.fdocuments.net/reader036/viewer/2022062907/5aaac7c67f8b9a86188e7a09/html5/thumbnails/12.jpg)
12
Energy & Telecommunications: Alarm CorrelationReducing False Positive Alarms
When 15 alarms are received within any 5
second window, and more than 10 alarms of the same type repeat in 4 subsequent 5-second windows, alert the operator
!
![Page 13: Progress Apama & Event Processingcomplexevents.com/slides/Event_Processing_Symposum... · Progress Apama & Event Processing ... Visualize Key Business Conditions and Actions in Real](https://reader036.fdocuments.net/reader036/viewer/2022062907/5aaac7c67f8b9a86188e7a09/html5/thumbnails/13.jpg)
13
Energy & Telecommunications: Alarm CorrelationReducing False Positive Alarms
When 15 alarms are received within any 5
second window, but < 5 similar alarms are detected within 30 seconds, then DO NOTHING
![Page 14: Progress Apama & Event Processingcomplexevents.com/slides/Event_Processing_Symposum... · Progress Apama & Event Processing ... Visualize Key Business Conditions and Actions in Real](https://reader036.fdocuments.net/reader036/viewer/2022062907/5aaac7c67f8b9a86188e7a09/html5/thumbnails/14.jpg)
14
Anticipitory Flight OperationsMonitor, analyze air space conflicts and act on operational efficiencies
Act: 1. Suggest plane re-route2. Rebook passengers3. Call in stand-by crews
Monitor: Check vertical & horizontal separation by constantly monitoring flight position event streams
Analyze:1. Analyze alternative flight paths2. Analyze passenger impact (missed connections)3. Analyze crew impact
![Page 15: Progress Apama & Event Processingcomplexevents.com/slides/Event_Processing_Symposum... · Progress Apama & Event Processing ... Visualize Key Business Conditions and Actions in Real](https://reader036.fdocuments.net/reader036/viewer/2022062907/5aaac7c67f8b9a86188e7a09/html5/thumbnails/15.jpg)
15
Real-Time Digital BattlefieldPreventing casualties with real-time visibility
Warn NATO squad commander when any
of his troops come within 1 mile a known mine field zone
Event Stream Processing
!
![Page 16: Progress Apama & Event Processingcomplexevents.com/slides/Event_Processing_Symposum... · Progress Apama & Event Processing ... Visualize Key Business Conditions and Actions in Real](https://reader036.fdocuments.net/reader036/viewer/2022062907/5aaac7c67f8b9a86188e7a09/html5/thumbnails/16.jpg)
16
Emergency ResponseDiscover patterns of events and real-time and take preemptive action
When 20 emergencies occur within any 60 minute window and
response capacity is over 50% within 100 miles, alert adjacent districts of stand-by state
![Page 17: Progress Apama & Event Processingcomplexevents.com/slides/Event_Processing_Symposum... · Progress Apama & Event Processing ... Visualize Key Business Conditions and Actions in Real](https://reader036.fdocuments.net/reader036/viewer/2022062907/5aaac7c67f8b9a86188e7a09/html5/thumbnails/17.jpg)
17
Supply Chain: RFID Data ManagementAutomating supply chain and logistics
When truck arrives, and all expected
pallets are not scanned within 60 minutes, send SMS to the operations manager
?
!?
![Page 18: Progress Apama & Event Processingcomplexevents.com/slides/Event_Processing_Symposum... · Progress Apama & Event Processing ... Visualize Key Business Conditions and Actions in Real](https://reader036.fdocuments.net/reader036/viewer/2022062907/5aaac7c67f8b9a86188e7a09/html5/thumbnails/18.jpg)
18
Health Care: Patient MonitoringActing on patient vital sign data
When a change in medication is followed by a rise in blood pressure within 20% of maximum allowable for this patient within any 10 second window, alert nearest nurse
![Page 19: Progress Apama & Event Processingcomplexevents.com/slides/Event_Processing_Symposum... · Progress Apama & Event Processing ... Visualize Key Business Conditions and Actions in Real](https://reader036.fdocuments.net/reader036/viewer/2022062907/5aaac7c67f8b9a86188e7a09/html5/thumbnails/19.jpg)
19
Agenda
� Major Characteristics of the Progress Approach
� Usage Scenarios
� Major Trends & Roadmap for EP
� Major Challenges for Community
![Page 20: Progress Apama & Event Processingcomplexevents.com/slides/Event_Processing_Symposum... · Progress Apama & Event Processing ... Visualize Key Business Conditions and Actions in Real](https://reader036.fdocuments.net/reader036/viewer/2022062907/5aaac7c67f8b9a86188e7a09/html5/thumbnails/20.jpg)
20
The Elements of Event Stream Processing
Management
ESB, Reuters, ALE
Correlation Engine
Real-Time Dashboards
Event Programming Language (EPL)
EPL Development Tools
Event Data Management
ESB, Email, SMS, Portal
![Page 21: Progress Apama & Event Processingcomplexevents.com/slides/Event_Processing_Symposum... · Progress Apama & Event Processing ... Visualize Key Business Conditions and Actions in Real](https://reader036.fdocuments.net/reader036/viewer/2022062907/5aaac7c67f8b9a86188e7a09/html5/thumbnails/21.jpg)
21
The Elements of Event Stream ProcessingThe EPL and Stream Processing Engines
Management
ESB, Reuters, ALE
Correlation Engine
Real-Time Dashboards
Event Programming Language (EPL)
EPL Development Tools
Event Data Management
ESB, Email, SMS, Portal
![Page 22: Progress Apama & Event Processingcomplexevents.com/slides/Event_Processing_Symposum... · Progress Apama & Event Processing ... Visualize Key Business Conditions and Actions in Real](https://reader036.fdocuments.net/reader036/viewer/2022062907/5aaac7c67f8b9a86188e7a09/html5/thumbnails/22.jpg)
22
ALL WITHINany 2 minute time period
time
real
-tim
e da
ta s
tream
s
Trading Rule
! ! ! !
AND (HPQ’s price moves up by 5%
! ! ! !
WHENMSFT price moves outside 2% of MSFT-15-minute-VWAP
! ! ! !
ORMSFT’s price moves down by 2%
)) !
! ! !
FOLLOWED-BY (S&P moving by 0.5%
THENBUY MSFTSELL HPQ
NASDAQ
NYSE
MSFT 15-MIN-VWAP
S&P500
• complex event sequences
An ESP Algorithmic Trading Rule
• multiple data streams
• real-time constraints
• automated actions• pattern abstraction
• temporal sequencing
![Page 23: Progress Apama & Event Processingcomplexevents.com/slides/Event_Processing_Symposum... · Progress Apama & Event Processing ... Visualize Key Business Conditions and Actions in Real](https://reader036.fdocuments.net/reader036/viewer/2022062907/5aaac7c67f8b9a86188e7a09/html5/thumbnails/23.jpg)
23
Correlation Engine
EventIntelligence
ActionEventsControl Events
Data Events
Data EventsData Events
pattern
matcher
pattern
matchertemporal
sequencer
temporal
sequencer ScenariosScenarios
aa
bb
cc
Event Scenarios
feedback loop
![Page 24: Progress Apama & Event Processingcomplexevents.com/slides/Event_Processing_Symposum... · Progress Apama & Event Processing ... Visualize Key Business Conditions and Actions in Real](https://reader036.fdocuments.net/reader036/viewer/2022062907/5aaac7c67f8b9a86188e7a09/html5/thumbnails/24.jpg)
24
Management
ESB, Reuters, ALE, IAF
Correlation Engine
Real-Time Dashboards
Event Programming Language (EPL)
Event Development Tools
Event Data Management
ESB, Email, SMS, Portal
The Elements of Event Stream ProcessingReal Time Dashboards
![Page 25: Progress Apama & Event Processingcomplexevents.com/slides/Event_Processing_Symposum... · Progress Apama & Event Processing ... Visualize Key Business Conditions and Actions in Real](https://reader036.fdocuments.net/reader036/viewer/2022062907/5aaac7c67f8b9a86188e7a09/html5/thumbnails/25.jpg)
25
Event-Driven, Real-Time DashboardsVisualize Key Business Conditions and Actions in Real Time
� Provides real-time dashboards from business � Operations � IT
� Interactive, real-time graphs, charts, tables, and dials
� Dashboard Studio allows full dashboard customization; not a fixed application layout
![Page 26: Progress Apama & Event Processingcomplexevents.com/slides/Event_Processing_Symposum... · Progress Apama & Event Processing ... Visualize Key Business Conditions and Actions in Real](https://reader036.fdocuments.net/reader036/viewer/2022062907/5aaac7c67f8b9a86188e7a09/html5/thumbnails/26.jpg)
26
Management
ESB, Reuters, ALE, IAF
Correlation Engine
Real-Time Dashboards
Event Programming Language (EPL)
Event Development Tools
Event Data Management
ESB, Email, SMS, Portal
The Elements of Event Stream ProcessingReal Time Dashboards
![Page 27: Progress Apama & Event Processingcomplexevents.com/slides/Event_Processing_Symposum... · Progress Apama & Event Processing ... Visualize Key Business Conditions and Actions in Real](https://reader036.fdocuments.net/reader036/viewer/2022062907/5aaac7c67f8b9a86188e7a09/html5/thumbnails/27.jpg)
27
The Apama ESP Developer Studio Enables Business Analysts to Design Powerful Real-Time Analytics
Intuitive visual user interface designed for business analysts
Express time-based real-time rules with a high level development tool
“SmartBlocks”encapsulate pre-packaged modules made available to non-programmers.
Each scenario, or group of rules, represents a “pattern” which can be adjusted by business users to specify conditions to monitor, analyze and act on.
![Page 28: Progress Apama & Event Processingcomplexevents.com/slides/Event_Processing_Symposum... · Progress Apama & Event Processing ... Visualize Key Business Conditions and Actions in Real](https://reader036.fdocuments.net/reader036/viewer/2022062907/5aaac7c67f8b9a86188e7a09/html5/thumbnails/28.jpg)
28
SmartBlocks – Domain Specific “Abstractions”Analytics Extend the Event Programming Environment
SmartBlocks abstract connectivity, event
rules, and databases
SmartBlock catalogs are
available via Apama’s
Scenario Modeler
e.g., RFID SmartBlocks
e.g., Algorithmic Trading / Risk
![Page 29: Progress Apama & Event Processingcomplexevents.com/slides/Event_Processing_Symposum... · Progress Apama & Event Processing ... Visualize Key Business Conditions and Actions in Real](https://reader036.fdocuments.net/reader036/viewer/2022062907/5aaac7c67f8b9a86188e7a09/html5/thumbnails/29.jpg)
29
Management
ESB, Reuters, ALE
Correlation Engine
Real-Time Dashboards
Event Programming Language (EPL)
EPL Development Tools
Event Data Management
ESB, Email, SMS, Portal
The Elements of Event Stream ProcessingEvent Data Management
![Page 30: Progress Apama & Event Processingcomplexevents.com/slides/Event_Processing_Symposum... · Progress Apama & Event Processing ... Visualize Key Business Conditions and Actions in Real](https://reader036.fdocuments.net/reader036/viewer/2022062907/5aaac7c67f8b9a86188e7a09/html5/thumbnails/30.jpg)
30
Event Storage, Replay and AnalysisDavid: “Data Streams Management”
� Pre-Flight Test Real-Time Algorithms
– Test algorithms against historical conditions before they go live
� Event Pattern Detection
– Purchasing agent has signed 5 POs at 95% of his signing authority in the last day – monitor for PO splitting?
� Genetic Tuning
– Run 10,000 instances of a strategy
– Grow the successful/profitable branches
� Root Cause Analysis
– Investigate: “What Happened?”
– Drill-down from dashboard
![Page 31: Progress Apama & Event Processingcomplexevents.com/slides/Event_Processing_Symposum... · Progress Apama & Event Processing ... Visualize Key Business Conditions and Actions in Real](https://reader036.fdocuments.net/reader036/viewer/2022062907/5aaac7c67f8b9a86188e7a09/html5/thumbnails/31.jpg)
31
Business Intelligence
ApamaReal-Time Event
Processing
Event Data Management ArchitectureStore, Replay, and Analyze the Event Driven World
Historical Event Processing
Capture raw events in a high-performance time-series data cache
Capture derived events –action - created by EPL rules
Event Store
“What If” Analysis: Back Testing “Pre-Flight Tests” event processing strategies The event database can feed static
data warehousing and BI infrastructure
“What Happened” Analysis -Dashboards visualize real-time and stored events
![Page 32: Progress Apama & Event Processingcomplexevents.com/slides/Event_Processing_Symposum... · Progress Apama & Event Processing ... Visualize Key Business Conditions and Actions in Real](https://reader036.fdocuments.net/reader036/viewer/2022062907/5aaac7c67f8b9a86188e7a09/html5/thumbnails/32.jpg)
32
Agenda
� Major Characteristics of the Progress Approach
� Usage Scenarios
� Major Trends & Roadmap for EP
� Major Challenges for Community
![Page 33: Progress Apama & Event Processingcomplexevents.com/slides/Event_Processing_Symposum... · Progress Apama & Event Processing ... Visualize Key Business Conditions and Actions in Real](https://reader036.fdocuments.net/reader036/viewer/2022062907/5aaac7c67f8b9a86188e7a09/html5/thumbnails/33.jpg)
33
Roy Schulte’s Event Processing Taxonomy
� Simple Message-Driven Applications– Sink triggered by 1 event, one stream– No pattern detection, no notion of causality– Benefits are for IT
� Mediated Events: Stateless – One stream split into multiple streams– Goal is message enrichment: filtering, CBR, transformation
� BPM – Enabled Events: Stateful– Message splitting and message combining– Flow of control governed by pre-defined BPM model of BP– Requires MOM + BPM engine
� CEP Applications– Multiple events, multiple streams (AGREE)– Sophisticated pattern detection (AGREE)– Non-IT benefit only via dashboard (DISAGREE)– Main benefit is business insight, not faster & easier software engineering
(DISAGREE)– Complex events are often synthesized from primitive events – genetic info is
often inserted (AGREE)
![Page 34: Progress Apama & Event Processingcomplexevents.com/slides/Event_Processing_Symposum... · Progress Apama & Event Processing ... Visualize Key Business Conditions and Actions in Real](https://reader036.fdocuments.net/reader036/viewer/2022062907/5aaac7c67f8b9a86188e7a09/html5/thumbnails/34.jpg)
34
3 Challenges for This Group
1) Characterize Event Processing (We Use ESP)
– Customer / usage orientation; not pure technical
– Define the Event Processing taxonomy & glossary
– Start with Roy’s Model: Simple, Mediated, BPM-Enabled, Complex (?)
2) Define EP’s Relationship to BAM
– Does the “M” stand for “Monitoring” or “Management”?
– Dashboards + Event Rules + Event Data Management = SuperBAM
3) Reconcile Current EP Approaches and Standardize Language
– SQL-based approach
– Language-based approach
– EAI-based approach
![Page 35: Progress Apama & Event Processingcomplexevents.com/slides/Event_Processing_Symposum... · Progress Apama & Event Processing ... Visualize Key Business Conditions and Actions in Real](https://reader036.fdocuments.net/reader036/viewer/2022062907/5aaac7c67f8b9a86188e7a09/html5/thumbnails/35.jpg)
35
Questions?