CON308

Building an Enterprise-Wide Instrumentation Solution Using the

Microsoft BizTalk BAM Infrastructure

Ulrich Roxburgh,Solution Architect,ulrich.roxburgh@services2.co.nz

Services2 Ltd.

Raise awareness of the existence of the Business Activity Monitoring (BAM) technology

Detail BAM capabilities

Demonstrate how it can be used to provide a “enterprise-wide” instrumentation layer

Session Objectives

The Problem Space

“The Business,” Support Staff, Developers, Stakeholders…

Needs to see what is going onand in a simple

understandable model

Instrumenting the Enterprise

BAM

SLA Monitoring

BAM

SLA: 3 Minutes

SLA: 5 Hours

End-to-End Instrumentation

BAM

ASP.NETBizTalk

...

Thick Client

WF ASMXWCF

.NET

BAM Overview

Business Activity Monitoring?

Key component of BizTalk ServerHigh-performance and scalable tracking framework Real-time visibility into business processesAutomatic creation of business monitoring infrastructure (no coding required)

Tables, Views, Archival, OLAP Cube

BAM Event API “Free” once you’ve purchased a BizTalk Server license (BizTalk Server 2006)

BAM API can be installed on non-BizTalk servers if BizTalk is used as part of your solution

Maps KPIs to Orchestration

Data Of Interest

Identifies KPIs

Business Activity MonitoringInformation Worker, Stakeholder..

Needs real-time visibility in a familiar user interface

Analyst, Stakeholder..

Analyzes information and defines what data will be presented and how

Developer

Knows how toget the data

ObservationModel

BAM Infrastructure

Itinerary

Customer Name Business Data (Text)

Customer Age Business Data (Integer)

PO Submitted Business Milestone

PO Processed Business Milestone

A Few Quick Concepts

BAM ActivityCollection of data items

Implemented as a SQL Table

Operations View

Total PO Processing Time Aggregation (Duration)

Supplier Web Service Time Aggregation (Duration)

Shipping Processing Time Time Dimension (Total Shipping Processing Time)

Purchase Order

Customer Name

Customer Status

Total PO Price

PO Submitted

PO Processed

A Few Quick Concepts...

BAM Activity ViewConsolidated “view” of activity data itemsImplemented as a SQL View/OLAP CubeMeasures, Dimensions (Progress, Data, Time, Range)

Business View

POs Processed Aggregation (Count)

Shipping Address Data Dimension (Going To)

BAM Observation Model

Defined using Excel via an add-inActivities and Activity Views created through a wizardData is simulated through PivotTables to help visualize Activity ViewsObservation Model then created by Exporting BAM XML or using the Worksheet directlyExcel can be used for Aggregation views

RTA Aggregations for real-time monitoring Deploy the Excel Spreadsheet for automatic connection

BAM Observation Model

demo

BAM Database Infrastructure

Provisioned automatically by bm.exeActivities are represented as SQL TablesViews are represented by SQL Views and optionally OLAP Cube

Some features require OLAP Cube processing unless RTA is used

Highly performantActive and Complete SplitPartitioningArchiving

Specific indexes can be supplied at creation time

Useful for tuning YOUR queries against the BAM databaseNo need to tune otherwise

BAM Database Infrastructure

demo

Tracking Profile Editor (TPE)

BizTalk Server onlyGraphically define the source for BAM data

Drag Shapes and Message Data to the Activity definition created via Excel

No code required, instant deploymentInstrumentation points are not fixed Can retrofit your existing systems with easeMuch improved in BizTalk Server 2006

New Event SourcesMessaging Context, Pipelines…

TPE

demo

BAM Portal

Available “in-the-box” with BizTalk Server 2006Activity Searches

Enable you to search and view any activity data with full reference semantics

Aggregation ViewsAny Activity Views defined visible via the portalPivotTable and Chart Views

Not extensible apart from basic skinningBut may be enough for your scenario

SQL Notification Services integration

Tracking Portal

Providing a view “inside the black box”Not just about “support” rolesDifferent variations can be used for all interested parties

Support Staff

What went wrong with PO: 1234 ?

Business User

How many POs are we receiving a day?

0

10

20

30

40

50

60

70

80

90

1st Qtr 2nd Qtr 3rd Qtr 4th Qtr

EastWestNorth

Tracking Portal

demo

BAM Internals

Activity Creation

A unique ID (ActivityID) is supplied at creation time

Examples: BTS.InterchangeID or System.Guid.NewGuid().ToString();

Activities can be long runningThey are “begun” and “ended” explicitlyAllows Activities to be added to over time

Whilst they are in the “begun” state they are held in the “xx_Active” tableWhen completed a trigger fires and they are moved to the “xx_Completed” table

Hotel Activity

Flight Activity

Hire Car Activity

Travel Insurance Activity

Itinerary Activity

Activity References

Any Activity can link to one or more ActivitiesMost Observation Models “model” interrelated items

Orders, Line Items, Credit Card Authorisation..

Stored in thebam_<ActivityName>_<Status>Relationships table

Custom References

“Link” some data to an ActivityUseful for attaching data which doesn’t naturally fit into Activity Data ItemsMessage Bodies are the most common approach

Can only be utilised via the BAM APIAddReference method (longReferenceData)Stored in the

bam_<ActivityName>_<Status>Relationships table

Continuation

Enables multiple “sources” to contribute to Activity Data

The downstream source may not have access to the same ActivityIDContinuation allows you to supply a “new” ActivityID that the downstream system has access to

Must also be used for any Asynchronous EventStream usage even if the same ActivityID is used

Itinerary Message Containing

CustomerReference CustomerRef I …….. I FlightREference

BAM Activity

Wiley Flight Services Booking

Web Service

Wiley Travel Services BIZTalk Solution

ActivityID: CustomerReferncce

ActivityID: BookingID

Booking ID

BAM without BizTalk!

BAM API

Four Event StreamsDirectEventStreamBufferedEventStreamOrchestrationEventStreamMessagingEventStream

BeginActivity, EndActivity, UpdateActivity, EnableContinuation, AddReferenceUntyped to enable flexibilityUnlike TPE your instrumentation points are hardcoded

GenerateTypedBamApi

BAM API code requires heavy usage of string literals

Brittle, time-consuming, prone to errors…GenerateTypedBamApi enables you to generate a typed API against your Activity DefinitionUtilises a XSLT transform to generate C# codehttp://www.codeplex.com/GenerateTypedBamApi

string ActivityID = System.Guid.NewGuid().ToString();

es.BeginActivity(“Itinerary”,ActivityID);

es.UpdateActivity(“Itinerary”,ActivityID, ”Name”,”Darren”);

es.EndActivity(“Itinerary”,ActivityID);

Instrumentation in Code

demo

BAM Interceptors

Interceptors (Shipped in BizTalk Server 2006 R2)

Windows Workflow (WF)SharePoint not possible at this time

Windows Communication Foundation (WCF)Custom

Instrument key points of your application

BAM Interceptors

demo

Why Would You Not Use TPE?

It’s graphical and easy?Modelling Continuation can be confusingLooping constructs are awkwardCan’t really test it independentlyYou often end up with a mixture of TPE and BAM API which can confuse mattersOnly available for BizTalk parts of your solution

It’s often a personal choice, butCode is easier to understand Straightforward to test directly against the APISource Code DiffHard if you don’t use the GenerateTypedBAMAPI approach

Reporting Services UI

Available “in-the-box” with Microsoft SQL Server 2000 and 2005“Reports” are built using Microsoft Visual Studio or the new Report Builder Windows Forms applicationQuick and straightforward “Portal” developmentReports can also be rendered to PDF, Microsoft Office Excel, etc.

Sample SQL QueriesSELECT ActivityID,Customer$Name,Received,Processed,Address,County,Town,Post$Code,Telephone$Number,Total$Itinerary$PriceFROM bam_Itinerary_CompletedInstances

SELECT f.*FROM bam_Flight_AllInstances f,bam_Itinerary_AllRelationships rWHERE r.ActivityID = @ItineraryActivityIDAND r.ReferenceName = ‘Flight’AND r.ReferenceData = f.ActivityIDAND r.ReferenceType = ‘Activity’

SELECT f.*FROM bam_Hotel_AllInstances f, bam_Itinerary_AllRelationships rWHERE r.ActivityID = @ItineraryActivityIDAND r.ReferenceName = ‘Hotel’AND r.ReferenceData = f.ActivityIDAND r.ReferenceType = ‘Activity’

Tracking Portals

demo

Summary

The Tracking Portal concept can be incredibly important to the supportability of your solutionBAM has a place in virtually all BizTalk solutions

Huge benefit to the overall solutionRemember that you can retro-fit your existing systems with ease

BAM can be used on non-BizTalk serversInstrument your entire solution end-to-end

WCF and Windows Forms BAM Interceptors coming soon

Problem Space, Tracking Portal Demonstration

Real-World Usage Scenarios

BAM Overview

BAM Internals

Getting data into the BAM tables

Tracking Portals

Summary

ResourcesTechnical Communities, Webcasts, Blogs, Chats & User Groupshttp://www.microsoft.com/communities/default.mspx

Microsoft Developer Network (MSDN) & TechNet http://microsoft.com/msdn http://microsoft.com/technet

Trial Software and Virtual Labshttp://www.microsoft.com/technet/downloads/trials/default.mspx

Microsoft Learning and Certificationhttp://www.microsoft.com/learning/default.mspx

BizTalk Server TrainingBizTalk 2006 R2 Jumpstart Training (2 Days)

 The course will explore new functionality such as WF and WCF integration with BizTalk, but assumes some existing knowledge in these areas. The course also explores the new EDI and RFID technologies.Prerequisites:

 Experience with Visual Studio 2005, C# and .NET development.1+ year of experience with BizTalk Server 2006 is essential.

AUCKLAND – Dates TBC - register your interest by emailing nzbiztk@microsoft.com

Designing and Developing BizTalk 2006 and 2006 R2 Solutions (3-day workshop)

BizTalk Architecture – Standards - BizTalk Tools - Development and Web ServicesDebugging and monitoring - Business Rules Engine - Single Sign-On - Deployment and

administrationAdvanced techniques (time permitting) - Hands-on Labs (real-life, end-to-end scenario)Cost $1500 per personLocation Microsoft or on client PremisesDate – Organised as required Email nzbiztk@microsoft.com to arrange your training

WW BizTalk R2 Launch Event in Sydney - Sept 14th   September 14 - Sydney, Australia - Sydney Convention & Exhibition Center

9:00 - 9:30 Registration

9:30 - 10:10 Keynote - Empowering Dynamic IT (Don F)

10:10 - 10:30 Break

10:30 - 12:00 BizTalk Server 2006 R2 - Extending the Connected Enterprise

12:00 - 13:00 Lunch

Industry Solutions(Health)- Content Owner – Renee Cathcart

Industry Solutions(Commercial)- Content Owner – Jane Mackarell & David Cryer

Industry Solutions (FSI)- Content Owner – Angela Larkin (BPM

session only)

13:00 - 14:00

Health Information Networks – ie focus on the Health Connect Engine

Supply Chain Management – Hub & Spoke ie How to Integrate disparate systems within Large ERP

systems?Business Process Management in

Organisations (FSI slant)

14:00 - 14:15 Break

14:15 - 15:15 Supply Chain Management – Business Process

Automation & Monitoring (eg Bryn/Datanet & RFID, Dynamics)

CCF (telco & FSI – Services presentation)

15:15 - 15:30 Break

15:30 - 16:30 Supply Chain Management – B-B ie ecommerce &

trading grids (eg Tony from FormFill presenting value chain integration in utilities)

ASB

Email nzbiztk@microsoft.com for more information  

Q&A

© 2007 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.

The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after

the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.