Has the ETL run yet?
-
Upload
jonas-harrington -
Category
Documents
-
view
215 -
download
0
Transcript of Has the ETL run yet?
![Page 1: Has the ETL run yet?](https://reader035.fdocuments.net/reader035/viewer/2022062517/56649f275503460f94c3f6d6/html5/thumbnails/1.jpg)
Real-Time Business Intelligence with Microsoft SQL Server 2008 R2 Jesus RodriguezChief Architect, TellagoMicrosoft Architect AdvisorMicrosoft MVPOracle SOA ACE
SESSION CODE: BIE403
![Page 2: Has the ETL run yet?](https://reader035.fdocuments.net/reader035/viewer/2022062517/56649f275503460f94c3f6d6/html5/thumbnails/2.jpg)
About me…
Chief Architect, Tellago, IncMicrosoft advisorMicrosoft MVPOracle SOA ACESpeaker, authorhttp://weblogs.asp.net/gsusx
![Page 3: Has the ETL run yet?](https://reader035.fdocuments.net/reader035/viewer/2022062517/56649f275503460f94c3f6d6/html5/thumbnails/3.jpg)
Agenda
The challenges of real time business intelligenceIntroducing Microsoft StreamInsightReal Time BI patterns Demo, Demo, Demo, Demo, Demo
![Page 4: Has the ETL run yet?](https://reader035.fdocuments.net/reader035/viewer/2022062517/56649f275503460f94c3f6d6/html5/thumbnails/4.jpg)
Real time business intelligence?
![Page 5: Has the ETL run yet?](https://reader035.fdocuments.net/reader035/viewer/2022062517/56649f275503460f94c3f6d6/html5/thumbnails/5.jpg)
Real-time Business Intelligence systems are event driven, and use Event Stream Processing techniques to enable events to be analysed without being first transformed and stored in a database.
These in- memory techniques have the advantage that high rates of events can be monitored, and since data does not have to be written into databases data latency can be reduced to milliseconds.
![Page 6: Has the ETL run yet?](https://reader035.fdocuments.net/reader035/viewer/2022062517/56649f275503460f94c3f6d6/html5/thumbnails/6.jpg)
Capabilities?
![Page 7: Has the ETL run yet?](https://reader035.fdocuments.net/reader035/viewer/2022062517/56649f275503460f94c3f6d6/html5/thumbnails/7.jpg)
Event Driven
![Page 8: Has the ETL run yet?](https://reader035.fdocuments.net/reader035/viewer/2022062517/56649f275503460f94c3f6d6/html5/thumbnails/8.jpg)
Continuous queries
![Page 9: Has the ETL run yet?](https://reader035.fdocuments.net/reader035/viewer/2022062517/56649f275503460f94c3f6d6/html5/thumbnails/9.jpg)
Time Sensitive
![Page 10: Has the ETL run yet?](https://reader035.fdocuments.net/reader035/viewer/2022062517/56649f275503460f94c3f6d6/html5/thumbnails/10.jpg)
Low Latency
![Page 11: Has the ETL run yet?](https://reader035.fdocuments.net/reader035/viewer/2022062517/56649f275503460f94c3f6d6/html5/thumbnails/11.jpg)
High Volume
![Page 12: Has the ETL run yet?](https://reader035.fdocuments.net/reader035/viewer/2022062517/56649f275503460f94c3f6d6/html5/thumbnails/12.jpg)
Event processing is the foundation of real time BI…
![Page 13: Has the ETL run yet?](https://reader035.fdocuments.net/reader035/viewer/2022062517/56649f275503460f94c3f6d6/html5/thumbnails/13.jpg)
Scenarios?
![Page 14: Has the ETL run yet?](https://reader035.fdocuments.net/reader035/viewer/2022062517/56649f275503460f94c3f6d6/html5/thumbnails/14.jpg)
Financial Sector
![Page 15: Has the ETL run yet?](https://reader035.fdocuments.net/reader035/viewer/2022062517/56649f275503460f94c3f6d6/html5/thumbnails/15.jpg)
Call Centers
![Page 16: Has the ETL run yet?](https://reader035.fdocuments.net/reader035/viewer/2022062517/56649f275503460f94c3f6d6/html5/thumbnails/16.jpg)
RFID Systems
![Page 17: Has the ETL run yet?](https://reader035.fdocuments.net/reader035/viewer/2022062517/56649f275503460f94c3f6d6/html5/thumbnails/17.jpg)
Web Analytics
![Page 18: Has the ETL run yet?](https://reader035.fdocuments.net/reader035/viewer/2022062517/56649f275503460f94c3f6d6/html5/thumbnails/18.jpg)
What are the challenges of traditional (batch) BI in real time scenarios?
![Page 19: Has the ETL run yet?](https://reader035.fdocuments.net/reader035/viewer/2022062517/56649f275503460f94c3f6d6/html5/thumbnails/19.jpg)
Query Overloading
![Page 20: Has the ETL run yet?](https://reader035.fdocuments.net/reader035/viewer/2022062517/56649f275503460f94c3f6d6/html5/thumbnails/20.jpg)
Agility
![Page 21: Has the ETL run yet?](https://reader035.fdocuments.net/reader035/viewer/2022062517/56649f275503460f94c3f6d6/html5/thumbnails/21.jpg)
ETL Dependencies
Has the ETL run yet?
![Page 22: Has the ETL run yet?](https://reader035.fdocuments.net/reader035/viewer/2022062517/56649f275503460f94c3f6d6/html5/thumbnails/22.jpg)
Report Abuse
![Page 23: Has the ETL run yet?](https://reader035.fdocuments.net/reader035/viewer/2022062517/56649f275503460f94c3f6d6/html5/thumbnails/23.jpg)
Ok I got it…..How do I implement something like that on the Microsoft
platform?
![Page 24: Has the ETL run yet?](https://reader035.fdocuments.net/reader035/viewer/2022062517/56649f275503460f94c3f6d6/html5/thumbnails/24.jpg)
Real time event processing was the missing piece….
![Page 25: Has the ETL run yet?](https://reader035.fdocuments.net/reader035/viewer/2022062517/56649f275503460f94c3f6d6/html5/thumbnails/25.jpg)
Introducing Microsoft StreamInsight
![Page 26: Has the ETL run yet?](https://reader035.fdocuments.net/reader035/viewer/2022062517/56649f275503460f94c3f6d6/html5/thumbnails/26.jpg)
Microsoft StreamInsight is a platform that you use to develop and deploy complex event processing (CEP) applications. Its high-throughput stream processing
architecture and the Microsoft .NET Framework-based development platform enable you to quickly implement robust and highly efficient event processing applications.
![Page 27: Has the ETL run yet?](https://reader035.fdocuments.net/reader035/viewer/2022062517/56649f275503460f94c3f6d6/html5/thumbnails/27.jpg)
Microsoft StreamInsight
27
CEP EngineO
utput Adapters
Input Adapters
Event
Standing Queries
Event sources Event targets
`
Devices, Sensors
Web servers
Event stores & Databases
Stock tickers & News feeds
Event
Event
Event
Event
Event
Event
C_ID C_NAME C_ZIP
Event stores & Databases
Pagers & Monitoring devices
KPI Dashboards, SharePoint UI
Trading stations
Event
Event
CEP Application at Runtime
Static reference data
IDE
.NETC#
LINQCEP Application Development
![Page 28: Has the ETL run yet?](https://reader035.fdocuments.net/reader035/viewer/2022062517/56649f275503460f94c3f6d6/html5/thumbnails/28.jpg)
Developing Your First StreamInsight Application
DEMO
![Page 29: Has the ETL run yet?](https://reader035.fdocuments.net/reader035/viewer/2022062517/56649f275503460f94c3f6d6/html5/thumbnails/29.jpg)
Events
Events expose different temporal characteristicsPoint in time eventsInterval events with fixed durationInterval events with initially unknown duration
Rich payloads capture all properties of an event
t1 t4t3t2 t5Time
Payl
oad/
val
ue
ab
c de
![Page 30: Has the ETL run yet?](https://reader035.fdocuments.net/reader035/viewer/2022062517/56649f275503460f94c3f6d6/html5/thumbnails/30.jpg)
Event Types
Events in Microsoft’s CEP platform use the .NET type systemEvents are structured and can have multiple fieldsFields are typed using the .NET framework types CEP engine provisioned timestamp fields capture all the different temporal event characteristicsEvent sources populate time stamp fields
Timestamps/Metadata
LongpumpID
StringType
StringLocation
Doubleflow
Doublepressure
… … … … … …
![Page 31: Has the ETL run yet?](https://reader035.fdocuments.net/reader035/viewer/2022062517/56649f275503460f94c3f6d6/html5/thumbnails/31.jpg)
Event Streams & AdaptersA stream is a possibly infinite sequence of events
Insertions of new eventsChanges to event durations
Stream characteristics:Event/data arrival patterns
Steady rate with end-of-stream indication Intermittent, random, or in bursts
Out of order events: Order of arrival of events does not match the order of their application timestamps
AdaptersReceive/get events from the data sourceEnqueue events for processing in the engine
31
![Page 32: Has the ETL run yet?](https://reader035.fdocuments.net/reader035/viewer/2022062517/56649f275503460f94c3f6d6/html5/thumbnails/32.jpg)
LINQ Query Examples
LINQ Example – GROUP&APPLY, WINDOW:
from e3 in MyStream3group e3 by e3.i into SubStreamfrom win in SubStream.HoppingWindow( FiveMinutes,ThreeSeconds)select new { i = SubStream.Key, a = win.Avg(e => e.f) };
LINQ Example – JOIN, PROJECT, FILTER:
from e1 in MyStream1join e2 in MyStream2 on e1.ID equals e2.IDwhere e1.f2 == “foo”select new { e1.f1, e2.f4 };
Join
FilterProject
Grouping
Window
Project &Aggregate
![Page 33: Has the ETL run yet?](https://reader035.fdocuments.net/reader035/viewer/2022062517/56649f275503460f94c3f6d6/html5/thumbnails/33.jpg)
Time Windows
Time
Tumbling Window
![Page 34: Has the ETL run yet?](https://reader035.fdocuments.net/reader035/viewer/2022062517/56649f275503460f94c3f6d6/html5/thumbnails/34.jpg)
Time Windows
Time
Hopping Window
![Page 35: Has the ETL run yet?](https://reader035.fdocuments.net/reader035/viewer/2022062517/56649f275503460f94c3f6d6/html5/thumbnails/35.jpg)
StreamInsight Hosting Models
CEP in-process server (explicit model)
Application
CEP explicit logic
CEP Server
Application
In-process server modeHosted server mode
![Page 36: Has the ETL run yet?](https://reader035.fdocuments.net/reader035/viewer/2022062517/56649f275503460f94c3f6d6/html5/thumbnails/36.jpg)
Real Time BI Patterns with Microsoft StreamInsight
![Page 37: Has the ETL run yet?](https://reader035.fdocuments.net/reader035/viewer/2022062517/56649f275503460f94c3f6d6/html5/thumbnails/37.jpg)
Real Time Distributed Event Processing
UI applicationStreamInsight Server
QueryStart Query
Events
UI application
Start Query
EventsOutput Adapter
Input Adapter
The event processing logic leaves on the StreamInsight ServerClients are able to start and stop queriesMultiple clients subscribe to the same even types
![Page 38: Has the ETL run yet?](https://reader035.fdocuments.net/reader035/viewer/2022062517/56649f275503460f94c3f6d6/html5/thumbnails/38.jpg)
Real Time Distributed Event Processing
DEMO
![Page 39: Has the ETL run yet?](https://reader035.fdocuments.net/reader035/viewer/2022062517/56649f275503460f94c3f6d6/html5/thumbnails/39.jpg)
Aggregations and Windowing
UI application StreamInsight Server
QueryStart Query
Events
Output Adapter
Input Adapter
The aggregation formulas are calculated in the StreamInsight ServerAggregations are processed using different time windowsThe logic can be extended with custom aggregates
![Page 40: Has the ETL run yet?](https://reader035.fdocuments.net/reader035/viewer/2022062517/56649f275503460f94c3f6d6/html5/thumbnails/40.jpg)
Real Time Event Windows and Aggregations
DEMO
![Page 41: Has the ETL run yet?](https://reader035.fdocuments.net/reader035/viewer/2022062517/56649f275503460f94c3f6d6/html5/thumbnails/41.jpg)
Real Time Event Replay
UI application StreamInsight Server
QueryReplay
Events
Output Adapter
Input Adapter
The StreamInsight server tracks the events processed by the different input adaptersThe client is able to replay events based on specific time windowsThe StreamInsight server uses a replay adapter to reprocess specific events
Event tracking
![Page 42: Has the ETL run yet?](https://reader035.fdocuments.net/reader035/viewer/2022062517/56649f275503460f94c3f6d6/html5/thumbnails/42.jpg)
Real Time Event Replay
DEMO
![Page 43: Has the ETL run yet?](https://reader035.fdocuments.net/reader035/viewer/2022062517/56649f275503460f94c3f6d6/html5/thumbnails/43.jpg)
Self Service BI + Real Time Event Processing
PowerPivot UI StreamInsight Server
Query
Output Adapter
Input Adapter
The StreamInsight server tracks the events processed by the different input adaptersThe PowerPivot UI polls the data into its own databaseThe user can use the data to calculate specific aggregations
Event tracking
PowerPivot UI
![Page 44: Has the ETL run yet?](https://reader035.fdocuments.net/reader035/viewer/2022062517/56649f275503460f94c3f6d6/html5/thumbnails/44.jpg)
Self-Service Event Analysis with StreamInsight and PowerPivot
DEMO
![Page 45: Has the ETL run yet?](https://reader035.fdocuments.net/reader035/viewer/2022062517/56649f275503460f94c3f6d6/html5/thumbnails/45.jpg)
Conclusions
Event processing is the foundation of real time business intelligenceMicrosoft StreamInsight enables complex event processing capabilities on the Microsoft platformTake into account real time business intelligence patterns such as time windows aggregations, replays, event subscriptions, etcTechnologies like Microsoft PowerPivot can complement the capabilities of StreamInsight with self-service data mechanisms
![Page 46: Has the ETL run yet?](https://reader035.fdocuments.net/reader035/viewer/2022062517/56649f275503460f94c3f6d6/html5/thumbnails/46.jpg)
Resources
www.microsoft.com/teched
Sessions On-Demand & Community Microsoft Certification & Training Resources
Resources for IT Professionals Resources for Developers
www.microsoft.com/learning
http://microsoft.com/technet http://microsoft.com/msdn
Learning
![Page 47: Has the ETL run yet?](https://reader035.fdocuments.net/reader035/viewer/2022062517/56649f275503460f94c3f6d6/html5/thumbnails/47.jpg)
Complete an evaluation on CommNet and enter to win!
![Page 48: Has the ETL run yet?](https://reader035.fdocuments.net/reader035/viewer/2022062517/56649f275503460f94c3f6d6/html5/thumbnails/48.jpg)
Sign up for Tech·Ed 2011 and save $500 starting June 8 – June 31st
http://northamerica.msteched.com/registration
You can also register at the
North America 2011 kiosk located at registrationJoin us in Atlanta next year
![Page 49: Has the ETL run yet?](https://reader035.fdocuments.net/reader035/viewer/2022062517/56649f275503460f94c3f6d6/html5/thumbnails/49.jpg)
© 2010 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to
be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
![Page 50: Has the ETL run yet?](https://reader035.fdocuments.net/reader035/viewer/2022062517/56649f275503460f94c3f6d6/html5/thumbnails/50.jpg)