2011-07-27 - Builders Café Webinar Series - PI for StreamInsight - Practical Uses Cases and...

Copyright 2010 OSIsoft, LLC

BUILDERS' CAF WEBINAR SERIES

PI FOR STREAMINSIGHT PRACTICAL USE CASES AND

IMPLEMENTATIONS

Not hearing anything right now? - Make sure Internet Audio is turned on in LiveMeeting - If not working, try to disconnect and reconnect - Otherwise, conference call +1 (800) 593-7197 (toll: +1 (210) 234-0067) - passcode: 2214913


2

PRESENTERS

Glenn Moffett, Product Management

Craig Bauhaus, Program Director of Real Time Data Management Solutions, Logica

J Sawyer, Sr Architect of Real Time Data Management Solutions, Logica - blog http://www.devbiker.net/

Ahmad Fattahi, OSIsoft vCampus Team


3

VCAMPUS-EXCLUSIVE WEBINARS

The Builders' Caf Webinar Series

Submit your ideas here:


4

OSISOFT VCAMPUS LIVE! 2011

Annual in-depth technology event for

Developers, System Integrators and Architects, System Administrators

Week of November 28th

Palace Hotel, San Francisco

Stay tuned!


INTRODUCTION TO PI FOR STREAMINSIGHT


7

PI FOR STREAMINSIGHT

Input and Output adapters for Microsoft StreamInsight

Analytics using Microsoft StreamInsight

Microsoft StreamInsight

PI Data Archive

Queries

Other Output Adapter(s)

Other Input Adapter(s)

PI Input Adapter

PI Output Adapter


LOGICA OVERVIEW


9

WE ARE LOGICA

Leading business and technology service company Employ 39,000 people worldwide

Operate in 40 countries

HQ Reading, UK

Revenue ~USD 6 billion in 2010

Provide business consulting, systems integration and outsourcing, serving Oil & Gas

Utilities

Transport & Logistics

Create value by successfully integrating people, business and technology Leverage deep domain knowledge with IT for superior service at best cost

Committed to long-term collaboration with clients, applying insight to create innovative answers to clients business needs

Strive for close, long-lasting relationships with select clients


TODAYS WORKING ENVIRONMENT


11

TODAYS WORKING ENVIRONMENT

Simultaneous, complex operations How do we maintain visibility better manage risk?

New technologies in unfamiliar areas Example: microseismic while fracturing

Critical applications and systems Cannot wait for problems to occur cannot afford downtime

Equipment and process availability Balancing maintenance, turnaround timing and downtime costs

Sensors everywhere Topside, downhole, Smart Grid

Massive data volumes Overwhelming

Analytics Many require lots of custom work on the part of the user

Can we do better than acquire, store and analyze after the fact?


12

WHAT IS COMPLEX EVENT PROCESSING?

Complex Event Processing (CEP) is the continuous and incremental processing of event streams from multiple sources based on declarative query and pattern specifications with near-zero latency

request

response

Event

output stream

input stream


13

REAL-TIME BI IS ALL ABOUT REDUCING TIME TO ACTION

What is it?

RT-BI is the ability to sense complex events hidden in massive real-time data streams, and being able to analyze and react to those events in near real-time with value adding activities.

How is this different from BI as we know it?

Traditional Business intelligence starts with surveillance, data sampling and capture, post-event analysis, and then notification and actions taken.

While each of these steps has some latency, real-time business intelligence technologies are designed to reduce all three latencies to as close to zero as possible.


14

Standing Queries

Query Logic

Event Sources Event targets

`

Web servers

Event stores & Databases

Stock ticker, news feeds Event stores & Databases

Pagers & Monitoring devices

KPI Dashboards, SharePoint UI

Trading stations

Input Adapters

Output Adapters StreamInsight Engine

Query Logic

Query Logic

MICROSOFT STREAMINSIGHT PLATFORM

PI for StreamInsight - Adapters

StreamInsight Application

Development


15

LOGICA CAPABILITIES WITH STREAMINSIGHT

North America-based development and implementation team

Fluent in databases, historians, GUI creation and StreamInsight technology

Developed API framework for StreamInsight technology

Developed Industry Adapters

WPC, OPC, WITSML, PRODML, others etc.

Logica API Framework

StreamInsight Engine

WITSML OPC-DA PRODML

WCF Duplex

File Trace

SQL RCDR

OSIsoft PI

TCP/UDP Sockets


16

DATA QUALITY DATA CLEANSING WITH PI SYSTEM ADAPTER

Historian


17

STREAM LIVELINESS

Processing done by application time not event arrival time per system clock. Current Time Increments (CTIs) events that set application time.

Adapters raise CTIs

? ?

Time

Out of order event

Regular event

CTI


18

PAYLOAD

.NET struct or class

Fixed structure once declared

Must use public fields or properties

Private fields, properties, and methods are ignored

Considerations:

Only simple scalar types supported

With StreamInsight 1.2, types may be nested

Empty payload structures are not permitted

Fields may not be adorned with custom attributes

Nullability is inferred from type


19

SAMPLE PAYLOAD DEFINITION

public class TollPayload

{

public int TollBoothID { get; set; }

public int LaneID { get; set; }

public int VehicleType { get; set; }

public String TagID { get; set; }

}


20

EVENT MODELS

Interval (payload valid for a period)

StartTime and EndTime

Values are valid for the defined interval

Point (payload valid at an instant)

StartTime

Event occurrence at a single point in time

EndTime is inferred by server as StartTime + 1 cronon

Edge (payload changing over time)

StartTime

End Time initially unknown (set to DateTime.MaxValue)

Important to know event occurring prior to completion


21

LINQ-BASED QUERIES

Read from an input stream

Outgoing events sent to an output stream

Projection

Selects required fields from a stream for new stream

May create a new type or refer to an existing type

May apply calculations to the data

Filtering

Joins and Unions

Evaluated based on overlapping time intervals


22

EXAMPLE PI QUERY // Process data input Stream CepStream dataStream = CepStream.Create( "PI Input Main Stream", //Name of the stream typeof(SnapshotInputFactory), //Specify the PI Input Adapter snapshotInputConfig, //Configuration for the input adapter EventShape.Point, //Desired Event Shape AdvanceTimeSettings.IncreasingStartTime //How application time is advanced ); // Join, Filter, Project, var alarmStream = from data in dataStream join limit in limitStream on data.Path.Substring(0, 5) equals limit.Path.Substring(0, 5) where !data.IsEdited select new PIEventBasic() { Id = data.Id, Value = (data.Status == 0 && limit.Status == 0) ? (data.Value > limit.Value ? 1 : -1) : 0, Path = data.Path.Replace("AlarmTest.Input", "AlarmTest.Output"), Status = data.Status, }; //Create a query from the stream. Only queries run. var query = alarmStream.ToQuery( cepApplication, //Reference to the StreamInsight application "Alarm Sample", //Unique name for the query "Sample alarm query with join, filter and project", //Description typeof(SnapshotOutputFactory), //Specify the PI Output Adapter outputConfig, //Configuration for the output adapter EventShape.Point, //Event shape StreamEventOrder.FullyOrdered);

// Start the query to begin processing. query.Start();


23

EVENT WINDOWS

Allows computation against events within an interval

The following operations are permitted on windows: Aggregation

TopK

User-defined operations

Windows are time-based or count-based

Do not change the underlying events

Can have input and output policies

4 types of windows Hopping

Tumbling

Snapshot

Count


24

EVENT WINDOWS

//Calculate rolling aggregate. Aggregate 1 hour of data, recalculate every minute. var aggregateStream = from d in dataStream group d by new { d.Id, d.Path } into idGroup

from i in idGroup.HoppingWindow( TimeSpan.FromHours(1), //1 hour aggregate TimeSpan.FromMinutes(1), //Calculated every minute

DateTime.Now.Date) //Alignment starts at midnight. select new {

Id = idGroup.Key.Id, Path = idGroup.Key.Path, Average = i.Avg(e => e.Value),

Sum = i.Sum(e => e.Value), Min = i.Min(e => e.Value), Max = i.Max(e => e.Value)

}; // Get the last known value, if no more than 2 days old.

var lastKnownValue = from e in dataStream .AlterEventDuration( e => TimeSpan.FromDays( 2 ) ) // Values from the past 2 days.

.ClipEventDuration( dataStream, ( e1, e2 ) => (e1.Id == e2.Id) && (e1.Path == e2.Path)) //Clip events based in identifier select e;


25

DETECTING MISSING EVENTS

//Create a query with all items in the past 30 seconds. var currentItemList = from i in dataStream .AlterEventDuration(e => TimeSpan.FromSeconds(30)) .ClipEventDuration(dataStream, (e, i) => (e.Id == i.Id && e.Path == i.Path)) select new { Id = i.Id, Path = i.Path }; //Do a left anti-semi-join to find the missing values. var missingItems = from il in lastKnownValue //Use our last known values stream where (from cil in currentItemList where il.Id == cil.Id && il.Path == cil.Path select cil).IsEmpty() //Finds where there are no readings for the past 30 seconds select new { Message = "[" + il.Id + "] Sensor appears to be offline. Last value:" + il.Value.ToString("#,##0.00"), Id = il.Id, Path = il.Path, LastValue = il.Value };


26

LIVE DEMO


27

PI FOR STREAMINSIGHT

Version 1 is released! (May this year)

Support for

reading and writing PI Points

Snapshot input data

Included in PI Server 2010


28

MICROSOFT STREAMINSIGHT V1.2 RELEASED!

Resiliency against planned and unplanned downtime through check pointing.

Better support for predictive modeling and pattern matching scenarios through new user-defined stream operators.

Improved administrator experience through performance counters and event logging

Improved development experience through support for nested event type structures and additional LINQ expressions.

A number of other improvementslike n-way joins, nested queries, and support for query macrosenhance the developer experience.


29

MORE INFORMATION

OSIsoft.com - introduction and access to datasheet

OSIsoft User Conference 2011 Developing Analytics Over Streaming Data with Microsoft StreamInsight & PI for StreamInsight

OSIsoft Virtual Campus Live 2010 - Complex Event Processing (CEP) with PI for StreamInsight

OSIsoft Technical Support website overview, system and licensing requirements (release notes and user guide available after login)

PI for StreamInsight available on vCampus!


30

Q & A

Users on Conference Call: Please do not put the call on hold as we would probably hear music! Users on Internet Audio: Type your questions in LiveMeeting


31

OSISOFT VCAMPUS LIVE! 2011

Annual in-depth technology event for

Developers, System Integrators and Architects, System Administrators

Week of November 28th

Palace Hotel, San Francisco

Stay tuned!


32

NEXT VCAMPUS WEBINARS


BUILDERS' CAF WEBINAR SERIES

PI FOR STREAMINSIGHT PRACTICAL USE CASES AND

IMPLEMENTATIONS

THANK YOU FOR JOINING US

2011-07-27 - Builders Café Webinar Series - PI for StreamInsight - Practical Uses Cases and...

Documents

Transcript of 2011-07-27 - Builders Café Webinar Series - PI for StreamInsight - Practical Uses Cases and...