Advanced Microsoft SQL Server 2008 R2 StreamInsightBeysim Sezgin (Architect)Roman Schindlauer (Program Manager)Microsoft Corporation

PDC09-SVR08

Agenda> Use Cases & Challenges> Formulating Declarative Queries> Windows in Time> Implementing Adapters> Architecture> Extensibility> Event Flow Debugging

Complex Event Processing (CEP) is the continuous and incremental processing of event streams from multiple sources based on declarative query and pattern specifications with near-zero latency.

What is CEP?

Complex Event Processing (CEP) is the continuous and incremental processing of event streams from multiple sources based on declarative query and pattern specifications with near-zero latency.

Database Applications

Event-driven Applications

Query Paradigm

Ad-hoc queries or requests

Continuous standing queries

request

response

Event output

streaminput stream

What is CEP?

Complex Event Processing (CEP) is the continuous and incremental processing of event streams from multiple sources based on declarative query and pattern specifications with near-zero latency.

Database Applications

Event-driven Applications

Query Paradigm

Ad-hoc queries or requests

Continuous standing queries

Latency Seconds, hours, days Milliseconds or less

request

response

Event output

streaminput stream

What is CEP?

Complex Event Processing (CEP) is the continuous and incremental processing of event streams from multiple sources based on declarative query and pattern specifications with near-zero latency.

Database Applications

Event-driven Applications

Query Paradigm

Ad-hoc queries or requests

Continuous standing queries

Latency Seconds, hours, days Milliseconds or lessData Rate Hundreds of

events/sec> Tens of thousands of events/sec

request

response

Event output

streaminput stream

What is CEP?

Standing Queries

Query Logic

Event sources Event targets

`

Devices, Sensors

Web servers

Event stores & Databases

Stock ticker, news feeds

Event stores & Databases

Pagers &Monitoring devices

KPI Dashboards, SharePoint UI

Trading stations

StreamInsight Application at Runtime

.NETC#

LINQStreamInsight Application Development

InputAdapter

s

OutputAdapter

sStreamInsight

Engine

Query Logic

Query Logic

StreamInsight Platform

12 3

4

Use Case: NBC Sunday Night Football

Telemetry Receiver

StreamInsight

Listener Adapter

GeoTag and group by region

Count total eventsCount session

startsCount active

sessions

SQL Adapter

PerfCounter Adapter

1

2

3

4

Use Case: Data Center

QueryCentral

time seriesarchive

Power Consumption

Process Information

Visualize

Complex Aggregations

/Correlations

Power MeterInput

Adapter

ETWInput

Adapter

Query

Query1 23

ChallengesHow do I …

> detect interesting patterns?> reason about temporal semantics?> correlate data?> aggregate data?> avoid writing custom imperative code?> create a runtime environment for

continuous and event-driven processing?

As a developer, I need a platform!

Query Expressiveness> Selection of events (filter)> Calculations on the payload

(project)> Correlation of streams (join)> Stream partitioning (group and

apply)> Aggregation (sum, count, …) over

event windows> Ranking over event windows

(topK)

Query Expressiveness

var result = from e in inputStream select new { id = e.id, W = (double)e.intW / 10 };

> Projection

Query Expressiveness

var result = from e in inputStream where e.id > 3 select new { id = e.id, W = (double)e.intW / 10 };

> Projection> Filter

Query Expressiveness

var result = from eLeft in inputStream1 join eRight in inputStream2 on eLeft.id equals eRight.id select new { id = eLeft.id, diff = eLeft.W - eRight.w };

> Projection> Filter> Correlation (Join)

Query Expressiveness

var result = from win in inputStream.TumblingWindow( TimeSpan.FromSeconds(10)) select new { avg = win.Avg(e => e.W) };

> Projection> Filter> Correlation (Join)> Aggregation over windows

> Projection> Filter> Correlation (Join)> Aggregation over windows> Group and Aggregate

Query Expressiveness

var result = from e in inputStream group e by e.id into eachGroup from win in eachGroup.TumblingWindow( TimeSpan.FromSeconds(10)) select new { eachGroup.Key, avg = win.Avg(e => e.W) };

Time Windows

Time

Tumbling Window

Time Windows

Time

Hopping Window

demo LINQ Queries

> Import data into StreamInsight platform

> Adapter characteristics:> Pull vs Push> Out-of-order vs. In-Order Events

> Adapter API> Observable model

> wraps adapter framework

Adapters

Adapter Factory

public class MyFactory : ITypedInputAdapterFactory<MyConfig>{ public InputAdapterBase Create<Payload>(MyConfig conf, EventShape ev) { return new MyAdapter(conf); }

public void Dispose() { ... }}

Adapter API

public class MyAdapter : TypedPointInputAdapter<MyPayload>{ public MyAdapter(MyConfig conf) { ... } public override void Start() { ... } public override void Resume() { ... }}

Stream Liveliness

??

Time

.Net Development

User App

Engine

Development Steps

LINQ

StreamInsight IL

Adapter Runtime query

1

Adapter

3

2

4

CompilationParser Algebrizer

Compiler

Type SystemMetadata

Expression Service

Runtime

Stream Manager

EventsStreams

Execution Operators Scheduler

Event Manager

Stream OS Resource Governor

Synch. Primitives

Buffer Manager CLR Hosting

StreamInsight Architecture

Query Compilation> Multiple Syntactic surfaces

over a single algebra, e.g., LINQ, CQL, etc.

> Compile time type checking and type safe code generation for minimal runtime impact.

> Leverage CLR > Code generator, JIT

support > Type System> Tools and Libraries (LINQ

Expressions, IDE, etc.)> Multiple Deployment

Environment with different form factors, e.g. SmartPhone, PDA, PCs.

CompilationParser Algebrizer

Compiler

Type SystemMetadata

Expression Service

Runtime – Events, Streams, Scheduler> JIT code generation for field

references, expression evaluation

> Event manager is implemented as a combination of managed and native code in order to minimize overhead and ensure predictable performance.

Runtime

Stream Manager

EventsStreams

Execution Operators Scheduler

Event Manager

> Cooperative Scheduler (less context switch, better SLA)> Operators are affinitized to a Scheduler. Periodic checks

and migration for load balancing> DataFlow Architecture: Reduced coupling and pipeline

parallelism

Runtime – Execution Operators> Efficient implementation of

operators that perform incremental evaluation as each event is processed.

> GroupAndApply Operator:> Enables parallelism for both

scale-up (multi-core) and scale-out(cluster).

> Groups are dynamically instantiated and torn down based upon the data. Large numbers of groups can be simultaneously active.

Machine2Machine1

~ MAP

ApplyApplyApply

Union X,Y,Z

ABC

Machine1Group A,B,C

BBBAA CC

XX ZZYYY

XYZ

~ REDUCE

Extensibility

> Need for domain-specific extensions> Integrate with functionality available in existing libraries

> User-defined operators, functions, aggregates

> Code written in .NET, deployed as .NET assembly

> Window-based semantics for UDOs, UDAs> Receive a set of events> Produce a value / set of events

Supportability> Diagnostic interface

> Retrieve statistical information at runtime

> Part of the object model API> Manage through Powershell> Debugger tool

> Retrieve live diagnostics> Record and replay queries> Analyze event processing along

operators

Debugger Tool

demo

Conclusion> CEP Platform & API> Event-triggered, fast Computation> API for Adapters, Queries, Applications> Declarative LINQ> Flexible Adapter API> Extensible> Supportability

Please Evaluate!> Sign in to microsoftpdc.com> Follow the evaluation link for the

session> PDC will donate $1 for each person

who completes an evaluation to the local Boys and Girls Club

© 2009 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.