Transcript of Complex Event Processing: Power your middleware with StreamInsight Mahesh Patel (Microsoft) Amit...
- Slide 1
- Complex Event Processing: Power your middleware with
StreamInsight Mahesh Patel (Microsoft) Amit Bansal
(PeoplewareIndia.com)
- Slide 2
- Agenda The Value of Timely Analytics The challenges / Scenarios
Introduction to Complex Event Processing Introducing Microsoft
StreamInsight Brief introduction to BizTalk Server Understand the
key scenarios that StreamInsight can unlock for BizTalk Map out the
architectural, integration requirements for BizTalk +
StreamInsight.
- Slide 3
- The value of timely analytics
- Slide 4
- Current Products for analytics
- Slide 5
- Slide 6
- Introduction to CEP Complex Event Processing (CEP) is an
architecture style based on the principles of Event Driven
Architecture (EDA). Continuously processing a high-volume stream of
events from various event sources Events are logically grouped in
sequences (typically known as streams) based on a time defined
criteria such as an interval. Event streams are processed through a
series of queries
- Slide 7
- Capabilities of a CEP Application Event Driven Continuous
Queries Time Sensitive Low Latency High Volume
- Slide 8
- The value of timely analytics
- Slide 9
- Scenarios Stream Data Store & Archive Event Processing
Engine Data Stream Asset Specs & Parameters Power, Utilities:
Energy consumption Outages Smart grids 100,000 events/sec Visual
trend-line and KPI monitoring Batch & product management
Automated anomaly detection Real-time customer segmentation
Algorithmic trading Proactive condition-based maintenance Visual
trend-line and KPI monitoring Batch & product management
Automated anomaly detection Real-time customer segmentation
Algorithmic trading Proactive condition-based maintenance Web
Analytics: Click-stream data Online customer behavior Page layout
100,000 events /sec Manufacturing: Sensor on plant floor React
through device controllers Aggregated data 10,000 events/sec
Threshold queries Event correlation from multiple sources Pattern
queries Threshold queries Event correlation from multiple sources
Pattern queries Lookup Asset Instrumentation for Data Acquisition,
Subscriptions to Data Feeds Financial Services: Stock & news
feeds Algorithmic trading Patterns over time Super-low latency
100,000 events /sec Data Stream
- Slide 10
- Introducing StreamInsight
- Slide 11
- DEMO CEP using StreamInsight
- Slide 12
- Traditional messaging applications: High signal every message
is business relevant Rich data schemas sets of related information
Workflow centric and transactional Emerging data sources and
trends: Machine born data is growing at a very rapid rate Not all
of this data is business relevant (low signal/noise ratio) How do
we: Identify and extract business relevant events from streaming
data? Take action on these business insights (line of business,
human workflow, etc) Messages and Events LOB Large Messages Low
Volume High Volume Small Events Web Services B22B SWIFT RFID Stock
Feeds Electrical Grids Operational Data Health Care CRM
- Slide 13
- Send Port MessageBox Orchestration Receive Port Message Maps
Receive Pipeline Adapter (HTTP) Message LOB Host Send Port Adapter
(File) Map Send Pipeline Component Basic BizTalk Architecture
Component Host Management DBTracking DB Message Agent
- Slide 14
- Scenarios Good BizTalk + StreamInsight scenarios: Streaming
data/event sources high noise to signal ratio Need to analyze and
extract business relevant events Integrate and transact with line
of business applications.
- Slide 15
- Scenarios Manufacturing (Oil and Gas) Monitor plant floor
processes Identify business relevant events Transact with LOB apps
Utilities Automated metering handle flood of data from utility
meters Identify business relevant events Transact with LOB apps The
data feeds change, the question changes the core scenario
doesnt!
- Slide 16
- OSIsoft PI & StreamInsight OSIsoft PI - Industry leading
process data management and historian. Integrating and leveraging
StreamInsight OSIsoft shipping StreamInsight adapter for PI. Stream
data in and out of PI Extract powerful business insights from PI
data Integrate with LOB applications using BizTalk Server
- Slide 17
- OPC & StreamInsight (with MatrikonOPC) OPC Industry
standard for process automation and data exchange MatrikonOPC
Industry leader in OPC interfaces Developed StreamInsight adapters
for their OPC implementation Extract powerful business insights
from OPC data Integrate with LOB applications using BizTalk
Server
- Slide 18
- Automated Metering Contoso Utilities is rolling out smart
electrical meters to their customers. Meters report power usage
(wattage, voltage) at a point in time. Data is consolidated into a
back end application used for billing and forecasting. The volume
of data has forced Contoso to only read values from meters every
hour They want to: Monitor the data more closely, compressing the
point stream to an interval stream before loading into the back end
application. i.e. if the data does not change by more than 2% over
a 10 minute window, keep the average value. Capture spikes and
threshold events, route to a process control application for
further inspection.
- Slide 19
- Automated Metering Collect high volume streaming data from
smart meters Data consists of timestamp, source ID and payload
(power in Watts) Sample operations on the streaming data:
Consolidate stream of points events into 5 minute chunks; record
average, count and standard deviation. Detect anomalies (threshold
10%), collect last 5 minutes worth of data for that meter and all
meters fed from that substation Need to link meter ID with
reference data (location, substation) Results must be integrated
into line of business application
- Slide 20
- Automated Metering 1 1 2 2 3 3 4 5 4 5
- Slide 21
- Architecture Message Integration WCF Adapters BizTalk Host
OrchestrationOrchestration Business Activity Monitoring Business
Rules Engine Message Box StreamInsight XmlPayload Output Adapter
QueriesQueries Input Adapters 11 22 33 11 StreamInsight extracts
insights from real time data feeds, feeds aggregate information to
BizTalk. 22 A strongly typed XML schema is inferred from the
StreamInsight query 33 Batched and correlated query results are
delivered to a BizTalk receive port.
- Slide 22
- StreamInsight is an in-memory event processing engine. Do not
deploy StreamInsight instance on the same hardware as your BizTalk
host instances. Leverage StreamInsight-local MSMQ for message
transfer. Deployment Message Integration
- Slide 23
- Summary BizTalk Server and StreamInsight together can unlock
powerful scenarios: High volume data processing Message integration
Leverage StreamInsight to: Consolidation and analyze incoming data
streams convert event sets into messages Deliver to BizTalk receive
port Leverage BizTalk Server to: Integrate these insights into line
of Business applications Other potential alignment points: Business
Activity Monitoring (BAM). Tap into BAM event events stream for
rich analytics and event detections Business Rules Engine (BRE).
Leverage BRE to define event pattern rules
- Slide 24
- 2011 Microsoft Corporation. All rights reserved. Microsoft,
Windows, Windows Vista and other product names are or may be
registered trademarks and/or trademarks in the U.S. and/or other
countries. The information herein is for informational purposes
only and represents the current view of Microsoft Corporation as of
the date of this presentation. Because Microsoft must respond to
changing market conditions, it should not be interpreted to be a
commitment on the part of Microsoft, and Microsoft cannot guarantee
the accuracy of any information provided after the date of this
presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR
STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.