BizTalk Technical Document

Technical DocumentBizTalk Technical DocumentPrepared forNALCOThursday, 1 December 2011Version 1.0Prepared byMPS PartnersContributorsAshish Talati

Prepared for NALCO

The information contained in this document represents the current view of MPS Partners, LLC. on the issues discussed as of the date of publication. Because MPS Partners must respond to changing market conditions, it should not be interpreted to be a commitment on the part of MPS Partners, and MPS Partners cannot guarantee the accuracy of any information presented after the date of publication.MPS PARTNERS MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS DOCUMENT.Complying with all applicable copyright laws is the responsibility of the user. Without limiting the rights under copyright, no part of this document may be reproduced, stored in or introduced into a retrieval system, or transmitted in any form or by any means (electronic, mechanical, photocopying, recording, or otherwise), or for any purpose, without the express written permission of MPS Partners, LLC.. MPS Partners may have patents, patent applications, trademarks, copyrights, or other intellectual property rights covering subject matter in this document. Except as expressly provided in any written license agreement from MPS Partners, the furnishing of this document does not give you any license to these patents, trademarks, copyrights, or other intellectual property. The descriptions of other companies products in this document, if any, are provided only as a convenience to you. Any such references should not be considered an endorsement or support by MPS Partners. MPS Partners cannot guarantee their accuracy, and the products may change over time. Also, the descriptions are intended as brief highlights to aid understanding, rather than as thorough coverage. For authoritative descriptions of these products, please consult their respective manufacturers. 2010 MPS Partners, LLC.. All rights reserved. Any use or distribution of these materials without express authorization of MPS Partners is strictly prohibited.MPS Partners and Windows are either registered trademarks of MPS Partners, LLC. in the United States and/or other countries.The names of actual companies and products mentioned herein may be the trademarks of their respective owners.Page iiBizTalk Technical Document

Revision and Signoff SheetChange RecordDateAuthorVersionChange reference

10/18/2011Ashish Talati1.0Created

11/30/2011Ashish Talati1.1Completed Cooling Controller

ReviewersNameVersion approvedPositionDate

Table of Contents1Environment11.1Overview11.2Installation Summary12Implementation22.1Overview22.2TFS32.3Cooling Controller Process32.4Schemas Cooling Controller42.4.1Cooling Schema42.4.1Cooling Property Schema62.4.1Cooling Alarm Schema72.4.1Cooling Apache Schema72.4.1Record Schema82.5Scenario Components102.5.1Cooling Data Receipt102.5.2Helper Components132.5.3Cooling Processor142.5.4Testability163Monitoring and Exception Handling193.1Tracking & Message Monitoring193.1.1BizTalk19

Page ivBizTalk Technical Document

EnvironmentOverviewThe NALCO BizTalk proof of concept did not plan for performance testing as a success criteria, so a highly available environment was not created. Two environments were created for the POC: Development Workstation: Used by developers to create and test the solution. Developer Integration Server: Solution was deployed to this server for testing purposes. This environment is typically were multiple developers code is integrated and tested before moving on to a QA process.The development workstation had the following software installed: Windows 7 IIS 7 SQL Server 2008 R2 Visual Studio 2010 BizTalk Server 2010 The development integration server had the following software installed: Windows 2008 R2 Enterprise IIS 7 SQL Server 2008 R2 Visual Studio 2010 BizTalk Server 2010

Installation SummaryBecause each of the environments were installed with single server envionrments, the installations were very basic and followed the installation guides from Microsoft. The following guides were used:Development Workstation: Installing BizTalk Server 2010 on Windows 7 and Windows Vista.docxDevelopment Integration Server:Installing BizTalk Server 2010 on Windows Server 2008 R2 and 2008.docxImplementation OverviewThe following is the high level process when a Boiler or Cooling controller Data file ( either in email format or an XML Format ) is received: File Receipt: File is Archived Data is parsed and Converted to Schema Optionally if file is compressed, uncompress it Component adds context information into message Header Mapper component: transforms file to canonical Schema Validator Validation & Enrichment Business validation Business Process Cooling Data Process Cooling Alarms Process Boiler Data, Event, Alarms Process Web for All Data and Configuration Process Monitoring Monitoring Orchestration Performance

This document will now describe each component of the scenario in a little more detail to show how this was implemented using BizTalk Server.

TFS

All source code are stored in TFS @ $/3DT Web/WPS Biztalk ProjectsA separate BizTalk application and solution are created for each Parser. Every solution is following same naming convention based on type of artifacts for BizTalk.The default naming convention is that the name of a folder containing a BizTalk Project is identical to projects output DLL name ( without extension ). For example, if BizTalk project resides in Cooling.Controller.Orchestrations folder, then that project must output a DLL named Cooling.Controller.Orchestrations.dllTypical example of project naming structure isOrchestration = Project.OrchestraionsPipelines = Project.PipelinesPipeline Components = Project.PipelineComponentsHelper Components = Project.ComponentsSchemas = Project.Schema ( or Project.Generated.Schemas )Maps/Transformaton = Project.TransformUnit Tests = Project.Tests

Cooling Controller Process

High Level Process Cooling Alarms, Cooling Data, Cooling Apache email messages are picked up by BizTalk Receive Location Receive Pipeline component Parses email Header and store that information into context properties Parses email body and using BizTalk flat file disassembler component will convert Flat file into XML ( for apache and data ). For alarm parser XML file is created with appropriate email body as received Filters out SPAM message ( by looking into email header information as well as content in email body ) Invoke Archive Process to archive data based on business requirement Few Business processes are done in pipeline component ( Message logging, saving email Subject Line, inserting blacklisted alarms ) Pipeline component will drop either apache, alarm or data XML message to BizTalk MessageBox Alarms Data Alarms Orchestration/Workflow will pickup the message and perform business processes ( Parse Alarms - .net SP Call ) and saving debug data ( Debug2 ) For Noflow, Interlock or NDM alarms, it will call Alarms recurring Workflow Email Data Workflow will find out ControllerId based on controller header data and insert data into ParameterValuesN or ParameterValuesT for each section data received for controller Apache Data Workflow will execute business process by calling parseApache stored procedure

Schemas Cooling ControllerBefore we define the scenarios itself, it makes sense to look at each of the message types being used in the scenario. Cooling SchemaThe Cooling Data schema is a flat file schema created using BizTalk Flat File Schema wizard feature. Each section within the file are created as a separate schema and all those schemas are then imported into main CoolingFF Schema.

Since not all the groups can come in data file, Choice Group option with max Occurs of 0 is set for all sections.

The following diagram is a representation of this schema:

The schema was modified to relax the way input data was received. This was done to allow looser checking of data in the pipeline ( for e.g. not always all data section are received in input message ) 0. Cooling Property SchemaThe Cooling property schema was created to store the most common data field in message context. When message is received in BizTalk, it contains the actual data as well as the message context. The message context is same concept as Message Header, SOAP Header or WCF Header as in a container to store various properties that can be used throught BizTalk Process. There are two types of message context properties. Property fields are context properties used by BizTalk Messaging enging mainly for message routing and some evaluations within BizTalk Orchestrations. One limitation is they can be upto 255 characters. It provides centralized mechanism for pulling key pieces of information that is then easily accessible to BizTalk Server components that are handling the message as it passes through BizTalk Server.

Email header related data are used throughout the process to subscribe and/or route message along the process.

The way to identify if its property schema is by looking into its Properties windows with a Schema Type as Property

0. Cooling Alarm SchemaThe Cooling Alarm schema will transform Cooling Alarm data from email body in Flat file to message schema

0. Cooling Apache SchemaThe Cooling Apache schema will transform Cooling Apache data from email body in Flat file to message schema

0. Record SchemaThe data that are received for Cooling controller has multiple sections and each section has multiple rows. For every single rows that is received in data, we need to update Parameter Table for all parameters for that row. To improve the performance instead of calling stored procedure to update parameter values for every single column in a row, the whole row is send to SQL server as Composite Operation. Following is the sample of DATA2 Section received with multiple rows with specific parameter values for that timeframe. [DATA2]|02/23/2011 04:15104.2249.841151.340.00-532.001.060.01107.91-999.000.000.000.000.00-14.5111.1694.420.00110.000.000.000.00128000128012801280128000|02/23/2011 04:30105.9950.801149.050.00-532.001.060.01109.49-999.000.000.000.000.00-14.4411.1795.270.00110.000.000.000.00128000128012811280128000|02/23/2011 04:45105.5450.841181.720.00-532.001.000.01109.51-999.000.000.000.000.00-14.2811.1495.340.00110.000.000.000.00128000128012821280128000|02/23/2011 05:00105.8451.241190.010.00-532.001.000.01110.25-999.000.000.000.000.00-14.2811.1495.440.00110.000.000.000.00128000128012821280128000|[END_DATA2]|To process and perform Composite operation on a single row, for every single section a new Schema is created for simpler mapping process. Looping and parsing single record from a source data is done within BizTalk Orchestration using xPath.

0. Scenario Components1. Cooling Data ReceiptThe purpose of the cooling receipt process is to archive the message that is received ( with an ability to disable at runtime ) and bring the cooling data into BizTalk and transform it to a format that can be used to process the message. This process is represented by the BizTalk receive pipeline as shown below:Cooling Receive Pipeline Component

The following sections describe each portion of the process.0. File ArchiveThe file archive component was implemented as a BizTalk custom pipeline component. BizTalk tracing is an alternate way to achieve that as well, but since we already have an existing process to archive the data to a file the same approach is taken here. This gives us an exact replica of the file before any processing commences. Below is a screenshot of the configuration of that pipeline component: 0. Disassemble Component BizTalk Disassembler pipeline Component is implement to Single Component to process Cooling water Data, Alarms and Apache Data Parse the incoming email message ( as *.eml ) and create a strongly typed .Net Object Determine validity of message and filter out Spam Business Processing like verifying blacklisted alarms, Verify PI data archiving

Promote Context Properties

Archive Data

Disassemble Flat file email Body data and create XML representation using BizTalk Standard Flat file disassembler.

1. Helper ComponentsHelper Components are .Net Class Library that are used throughout the Business Processing. They are used from within Pipeline Components or from within Orchestrations. EmailObject : A strongly typed .net Object to represent email Object ParseEmailObject : A .net Class to parse email message ( *.eml ) and create a email Object identifying business specific data like email subject, soldTo, username, message type CoolingDynamicTransform : A .net class ( serializable ) used from within BizTalk Orchestration to perform Dynamic Transformation using XSLT.

1. Cooling Processor2. Composite UpdateThe out of box WCF-SQL adapter enables adapter clients to perform composite operations on the SQL Server database. A composite operation can include: Insert, Update, and Delete operations. A Select operation is not supported as part of a composite operation. Stored procedures executed as operations.A single composite operation can have any number of these operations, in any order. Composite operation pattern is implemented to improve performance and send multiple calls to Stored procedure as a single composite operation.The following orchestration represents this process: 2. Data Transformation and TranslationBizTalk mapper is used to create maps, which are used for translating and transforming xml messages. Depending on the type and complexity of transformation, following different transformation features in within BizTalk Mapper are used Using Inline XSLInline XSLT is used in the map below to transform many to one relationship

Using BizTalk FunctoidsUsage of Inbuilt BizTalk Functoids to perform and check if source element is null or empty and based on the conditional apply the mapping to destination schema

Using XSLBizTalk mapper by default uses XSL to perform transformation and translation. If out of the box functoid provided by BizTalk doesnt satisfy business requirement, they can be extended with using custom XSL. 1. TestabilityAll BizTalk Artifacts had unit Testing enabled and depending on type of BizTalk Artifact Object mocking was performed if needed.3. Schema TestingBizTalk Projects were enable to allow Schema Testing

Once the project is enabled, testing can be performed using TestTools provided by Microsoft BizTalk Assemblies.

3. Maps TestingMaps can be tested as Schema, alternatively BizUnit ( open source ) was used to perform BizTalk Map testing as well as validating the output data via XPath

3. PipelineComponent TestingPipelinetesting unit tests are written to use Winterdom Tools

1. Monitoring and Exception Handling1. Tracking & Message Monitoring0. BizTalkBizTalk has its own tracking and debugging capabilities built into the administration tool. We can use these abilities to get information about the processes being run as well as view message contents and processes. Please see screenshots below:

We can also use the administration tool to debug the orchestration processes or view a process that occurred in the past.

Page 2BizTalk Technical Document