SQL Server 2008 R2 StreamInsight
description
Transcript of SQL Server 2008 R2 StreamInsight
![Page 1: SQL Server 2008 R2 StreamInsight](https://reader036.fdocuments.net/reader036/viewer/2022081422/5681650c550346895dd78287/html5/thumbnails/1.jpg)
SQL Server 2008 R2 StreamInsight
Complex Event ProcessingEvent Stream Processing
![Page 2: SQL Server 2008 R2 StreamInsight](https://reader036.fdocuments.net/reader036/viewer/2022081422/5681650c550346895dd78287/html5/thumbnails/2.jpg)
Who Am I?
• SQL Server MVP• SQL Server Consultant• Joint author on Wrox Professional SSIS book• www.SQLDTS.com and www.SQLIS.com• Specialise in Moving Data• @allanSQLIS (twitter)
![Page 3: SQL Server 2008 R2 StreamInsight](https://reader036.fdocuments.net/reader036/viewer/2022081422/5681650c550346895dd78287/html5/thumbnails/3.jpg)
Agenda
• Use Cases & Challenges• Formulating Declarative Queries• Windows in Time• Event Flow Debugging• Demos
![Page 4: SQL Server 2008 R2 StreamInsight](https://reader036.fdocuments.net/reader036/viewer/2022081422/5681650c550346895dd78287/html5/thumbnails/4.jpg)
Complex Event Processing (CEP) is the continuous and incremental processing of event streams from multiple sources based on declarative query and pattern specifications with near-zero latency.
What is CEP?
![Page 5: SQL Server 2008 R2 StreamInsight](https://reader036.fdocuments.net/reader036/viewer/2022081422/5681650c550346895dd78287/html5/thumbnails/5.jpg)
Complex Event Processing (CEP) is the continuous and incremental processing of event streams from multiple sources based on declarative query and pattern specifications with near-zero latency.
Database Applications Event-driven Applications
Query Paradigm
Ad-hoc queries or requests
Continuous standing queries
request
response
Eventoutput streaminput
stream
What is CEP?
![Page 6: SQL Server 2008 R2 StreamInsight](https://reader036.fdocuments.net/reader036/viewer/2022081422/5681650c550346895dd78287/html5/thumbnails/6.jpg)
Complex Event Processing (CEP) is the continuous and incremental processing of event streams from multiple sources based on declarative query and pattern specifications with near-zero latency.
Database Applications Event-driven Applications
Query Paradigm
Ad-hoc queries or requests
Continuous standing queries
Latency Seconds, hours, days Milliseconds or less
request
response
Eventoutput streaminput
stream
What is CEP?
![Page 7: SQL Server 2008 R2 StreamInsight](https://reader036.fdocuments.net/reader036/viewer/2022081422/5681650c550346895dd78287/html5/thumbnails/7.jpg)
Complex Event Processing (CEP) is the continuous and incremental processing of event streams from multiple sources based on declarative query and pattern specifications with near-zero latency.
Database Applications Event-driven Applications
Query Paradigm
Ad-hoc queries or requests
Continuous standing queries
Latency Seconds, hours, days Milliseconds or less
Data Rate Hundreds of events/sec > Tens of thousands of events/sec
request
response
Eventoutput streaminput
stream
What is CEP?
![Page 8: SQL Server 2008 R2 StreamInsight](https://reader036.fdocuments.net/reader036/viewer/2022081422/5681650c550346895dd78287/html5/thumbnails/8.jpg)
Who might need CEP
• Fraud Detection• Real-Time Trade Risking• Algorithmic Trading/Betting• Meter throughputs– Oil, Gas, Water, Electricity– Use to drive alarms, alerts etc
![Page 9: SQL Server 2008 R2 StreamInsight](https://reader036.fdocuments.net/reader036/viewer/2022081422/5681650c550346895dd78287/html5/thumbnails/9.jpg)
Standing Queries
Query Logic
Event sources Event targets
`
Devices, Sensors
Web servers
Event stores & Databases
Stock ticker, news feeds Event stores & Databases
Pagers &Monitoring devices
KPI Dashboards, SharePoint UI
Trading stations
StreamInsight Application at Runtime
.NETC#
LINQStreamInsight Application Development
InputAdapters
OutputAdapters
StreamInsight Engine
Query Logic
Query Logic
StreamInsight Platform
1
2 34
![Page 10: SQL Server 2008 R2 StreamInsight](https://reader036.fdocuments.net/reader036/viewer/2022081422/5681650c550346895dd78287/html5/thumbnails/10.jpg)
Query Expressiveness
• Selection of events (filter)• Calculations on the payload (project)• Correlation of streams (join)• Stream partitioning (group and apply)• Aggregation (sum, count, …) over event
windows• Ranking over event windows (topK)
![Page 11: SQL Server 2008 R2 StreamInsight](https://reader036.fdocuments.net/reader036/viewer/2022081422/5681650c550346895dd78287/html5/thumbnails/11.jpg)
Query Expressiveness
var result = from e in inputStream select new { id = e.id, W = (double)e.intW / 10 };
• Projection
![Page 12: SQL Server 2008 R2 StreamInsight](https://reader036.fdocuments.net/reader036/viewer/2022081422/5681650c550346895dd78287/html5/thumbnails/12.jpg)
Query Expressiveness
var result = from e in inputStream where e.id > 3 select new { id = e.id, W = (double)e.intW / 10 };
• Projection• Filter
![Page 13: SQL Server 2008 R2 StreamInsight](https://reader036.fdocuments.net/reader036/viewer/2022081422/5681650c550346895dd78287/html5/thumbnails/13.jpg)
Query Expressiveness
var result = from eLeft in inputStream1 join eRight in inputStream2 on eLeft.id equals eRight.id select new { id = eLeft.id, diff = eLeft.W - eRight.w };
• Projection• Filter• Correlation (Join)
![Page 14: SQL Server 2008 R2 StreamInsight](https://reader036.fdocuments.net/reader036/viewer/2022081422/5681650c550346895dd78287/html5/thumbnails/14.jpg)
Query Expressiveness
var result = from win in inputStream.TumblingWindow( TimeSpan.FromSeconds(10)) select new { avg = win.Avg(e => e.W) };
• Projection• Filter• Correlation (Join)• Aggregation over windows
![Page 15: SQL Server 2008 R2 StreamInsight](https://reader036.fdocuments.net/reader036/viewer/2022081422/5681650c550346895dd78287/html5/thumbnails/15.jpg)
• Projection• Filter• Correlation (Join)• Aggregation over windows• Group and Aggregate
Query Expressiveness
var result = from e in inputStream group e by e.id into eachGroup from win in eachGroup.TumblingWindow( TimeSpan.FromSeconds(10)) select new { eachGroup.Key, avg = win.Avg(e => e.W) };
![Page 16: SQL Server 2008 R2 StreamInsight](https://reader036.fdocuments.net/reader036/viewer/2022081422/5681650c550346895dd78287/html5/thumbnails/16.jpg)
Windowing
• Hopping– Tumbling
• Snapshot• Count
![Page 17: SQL Server 2008 R2 StreamInsight](https://reader036.fdocuments.net/reader036/viewer/2022081422/5681650c550346895dd78287/html5/thumbnails/17.jpg)
Time Windows
Time
Hopping Window
![Page 18: SQL Server 2008 R2 StreamInsight](https://reader036.fdocuments.net/reader036/viewer/2022081422/5681650c550346895dd78287/html5/thumbnails/18.jpg)
Time Windows
Time
Tumbling Window
![Page 19: SQL Server 2008 R2 StreamInsight](https://reader036.fdocuments.net/reader036/viewer/2022081422/5681650c550346895dd78287/html5/thumbnails/19.jpg)
Snapshot Windows
![Page 20: SQL Server 2008 R2 StreamInsight](https://reader036.fdocuments.net/reader036/viewer/2022081422/5681650c550346895dd78287/html5/thumbnails/20.jpg)
Snapshot Windows
![Page 21: SQL Server 2008 R2 StreamInsight](https://reader036.fdocuments.net/reader036/viewer/2022081422/5681650c550346895dd78287/html5/thumbnails/21.jpg)
Debugger
• Has its own debugger• You cannot live without it (believe me)• Set breakpoints• Filter
• A wonderful tool
![Page 22: SQL Server 2008 R2 StreamInsight](https://reader036.fdocuments.net/reader036/viewer/2022081422/5681650c550346895dd78287/html5/thumbnails/22.jpg)
DEMOS
•Tour•Trace Reader Live•Trace File Reader•Query Demo (Joining)•Debugger Tool (How it will save you)