Brian LoesgenCharles YoungJan Eliasen

Scott Colestock

Anush Kumar

Jon Flanders

Microsoft*BizTalk'Server 2010

UNLEASHED

800 East 96th Street, Indianapolis, Indiana 46240 USA

Table of Contents

Foreword xx"

Part I The Basics

1 What Is BizTalk Server? 3

A Brief History of Application Integration 3

BizTalk Server Capabilities 7

Adaptation 7

Mediation 8

Exception Handling 8

Orchestration and Choreography 9

Performance and Scalability 9

Security 10

Insight 10

Electronic Data Interchange 11

RFID Event Handling 11

What Is a "Typical" BizTalk Solution? 11

BizTalk Server, WCF, and WF 12

Summary 14

2 Schemas 15

BizTalk Schemas 16

XML Schema Definition 16

Properties 17

Internal Schemas 18

XML Schemas 20

Existing XSDs 20

Generating an XSD 21

Creating an XSD 21

Flat File Schemas 36

Add Existing Schemas 38

Creating by Hand 38

Flat File Schema Wizard 47

EDI Schemas 60

Messages That Are Not XML and Not Flat File 60

Pass-Through Pipeline 60

Custom Disassembler 61

Custom Editor Extensions 61

Third-Party Components 61

Property Promotion 61

Distinguished Fields 63

Promoted Properties 65

Property Demotion 66

When to Use What 67

Versioning of Schemas 69

No Long-Running Transactions and a Short

Downtime Acceptable 69

Long-Running Transactions or a Short Downtime

Is Unacceptable 70

Testing 71

Validate Schemas 71

Validate Instances 72

Generate Instances 74

Unit Testing of Schemas 75

Testing Using Pipeline Tools 80

Schemas for Scenario Used in This Book 81

FineFoods.Common.Schemas 81

FineFoods.CreditCheck.Schemas 82

FineFoods.Customers.C1701 82

FineFoods.Customers.C1702 83

FineFoods.Customers.Schemas 84

FineFoods.Inventory.Schemas 84

FlneFoods.Orders.Schemas 84

FineFoods.PurchaseOrders.Schemas 87

Summary 88

3 Maps 89

The Mapper 90

Layout of Mapper 90

Initial Considerations 92

Creating a Simple Map 94

Functoids 108

String Functoids Ill

Mathematical Functoids 112

Logical Functoids 113

Date/Time Functoids 115

Conversion Functoids 116

Scientific Functoids 116

Cumulative Functoids 117

Database Functoids 118

Advanced Functoids 120

Third-Party Functoids 122

Advanced Maps 123

Mapping Optional Fields 123

Looping Functoid 123

Index Functoid 125

Database Lookup 127

Scripting Functoid 129

Functoid Combination 131

Combination of Functoids for If-Then-Else 131

Create Separated List 132

Table Looping Functoid 132

Conditional Creation of Output Nodes 135

Custom XSLT 136

Cross Referencing 136

Building Custom Functoids 140

Initial Setup 141

Normal Functoid 146

Cumulative Functoid 151

Developing Advanced Functoids 155

Deployment of Custom Functoids 157

Debugging 161

Testing of Maps 163

Validating Maps 164

Testing Maps 164

Debugging a Map 167

Unit Testing 168

Summary 172

4 Orchestrations 173

Orchestration Designer 174

Defining Orchestrations 177

Building Orchestrations 178

Messages 182

Variables 186

Shapes 188

Delivery Notification and Handling Retries 217

Calling Pipelines 218

Web Services 221

Dehydration and Rehydration 228

Correlations 229

Convoys 234

Parallel Convoys 234

Sequential Convoys 235

Zombies 236

Transactions 237

Atomic 238

Long Running 240

Compensating Code 241

Persistence Points 246

Exception Handling 247

Debugging 250

Send Out Messages 250

Debug and Trace 250

Breakpoints in Orchestration Debugger 250

Summary 255

5 Pipelines 257

Stages in Pipelines 258

Stages in a Receive Pipeline 259

Stages in a Send Pipeline 261

Built-in Pipelines 262

Receive Pipelines 262

Send Pipelines 263

Built-in Pipeline Components 263

XML Components 264

Flat Files 268

Encoding, Encrypting, and Signing 272

BizTalk Framework 275

Validator and Party Resolution 280

Custom Pipelines 283

Using the Built-in Pipeline Templates 283

Creating Custom Pipeline Templates 284

Custom Pipeline Components 287

Resources, Attributes, and Constructors 288

Interfaces 292

Message and Context Interfaces 305

Miscellaneous Functionality 309

Streaming 314

Properties 317

Really Fast Pipeline Component Implementation 323

Deployment 324

Debugging 327

Pipeline Component Wizard 329

Testing 330

Pipeline.exe 330

Unit Testing 331

Summary 334

6 Adapters 337

BizTalk Adapters 337

Native Adapters 338

Line-of-Business Adapters 339

BizTalk Adapter Pack 339

Host Adapters 339

Third-Party and Custom Adapters 339

Additional Microsoft Adapters 340

The Role of WCF Adapters 340

Adapter Characteristics 340

Direction 341

Push and Pull 341

Message Interchange Pattern 341

Hosting 342

Configuration 342

Batches 343

Transactions 344

Message Context 344

Metadata Harvesting 344

Registering Adapters .345

Creating Adapter Handlers 346

Port-Level Configuration 349

Configuring Receive Locations 350

Configuring Send Ports 352

Adapter Properties 355

Deploying Bindings 355

Native Adapters 357

File Adapter 357

Robust Interchange 357

Polling Locked Files 358

File Renaming 359

Reliable Messaging Issues 359

Path and File Names 359

Security 360

Additional Send Handler Issues 360

FTP Adapter 360FTP Issues 361

Handling Duplicate Messages 362

Staging Files in Temporary Folders 362

Raw FTP Commands 363

Secure Messaging 363

HTTP Adapter 364

Using HTTP Receive Handlers 364

Using HTTP Send Handlers 366

Additional Configuration 366

MQ Series Adapter 367

Using MQ Series Receive Handlers 368

Using MQ Series Send Handlers 369

Managing Queues 369

Configuring MQSAgent 370

MSMQ Adapter 370

Using MSMQ Receive Handlers 371

Using MSMQ Send Handlers 372

Authenticating and Securing Messages 374

POP3 Adapter 375

Using POP3 Receive Handlers 376

Handling Encrypted Messages 377

SMTP Adapter 377

Using SMTP Send Handlers 378

Windows SharePoint Services Adapter 379

Using WSS Receive Handlers 380

Using WSS Send Handlers 381

Mapping SharePoint Columns 383

SOAP Adapter 383

WCF Adapters 384

Windows Communication Foundation 385

Comparing WCF to BizTalk Server 386

The Role of BizTalk Native WCF Adapters 388

Hosting Native WCF Adapters 389

The WCF Service Publishing Wizard 389

Publishing Orchestrations 392

Publishing Schemas 392

WCF Send Handlers 394

Importing MEX Endpoints 395

Importing Metadata Files 396

Dynamic Ports 397

Configuring WCF Adapters 397

Addresses and Identity 398

Bindings 399

Behavior 400

Security and Credentials 401

Message Handling 402

Using the SQL Server LoB Adapter 404

WCF LoB Framework and SDK 404

SQL Server Adapter 404

Polling and Notification 405

Performing Operations via a Send Handler 407

Additional Adapter Capabilities 408

Metadata Harvesting 409

Summary 412

Part II Advanced Topics

7 BizTalk 2010 and WCF: Extensibility 415

WCF Extensibility 416

The WCF Channel Stack 416

ABCs Reviewed 417

ServiceContract in BizTalk 418

WCF Behaviors 420

Example of WCF Extensibility in BizTalk 420

Summary 429

8 BizTalk and Windows Azure 431

Extending the Reach of BizTalk Applications 431

The AppFabric SDK 432

Receiving Messages 433

Sending Messages 434

Static Send Port 435

Dynamic Send Port 436

ESB Off-Ramp 436

Using InfoPath as a Client 438

Summary 439

9 Business Activity Monitoring with BizTalk BAM 441

BAM and Metrics 441

What Is BizTalk BAM? 442

Using BizTalk BAM 444

End-to-End, High-Level Walkthrough of the BAM Process 444

Real-Time Versus Scheduled Aggregations 446

Defining Activities and Views 447

Progress Dimension 450

Data Dimension 450

Numeric Range Dimension 450

Time Dimension 450

Using the Tracking Profile Editor 452

Using the BAM APIs 453

DirectEventStream (DES) 453

BufferedEventStream (BES) 453

OrchestrationEventStream (OES) 454

IPipelineContext Interface 454

Creating a Higher-Level API Specifically for Service Metrics 454

Working with the WCF and WF Interceptors 457

Using Notifications 460

Rapid Prototyping 460

REST and BAM 461

Managing BAM 461

BAM Database Considerations 461

Deployment and Management 461

Security 462

Scripting Deployment 462

Summary 465

10 The Business Rules Framework 467

The Importance of Rules 468

Processes and Policies 468

Business Policies 469

Policy Externalization 469

Policy Scenarios 471

Business Versus Executable Rules 472

Business Rule Management 473

BRMS and the BRF 475

Example Scenario: Order Processing 476

Incomplete and Implicit Business Rules 478

Indirect Policy Mapping 478

Technical Policy 479

Data Models 479

Programmatic Bindings 479

Priority 479

Traceability 479

Refactoring 480

Testing, Publishing, and Deployment 480

Managing Change 481

Real-World Rule-Processing 482

Using Vocabularies 483

What About Performance? 484

Inference and Reasoning 485

The Business Rules Framework 487

Introducing the BRF 487

Rule Storage and Administration 488

Rule Deployment 489

Rule Modeling 495

Rule Execution 497

Components and Tools 499

Microsoft Business Rule Language 499

Business Rules Database 499

Pub-Sub Adapter 504

Rule Store Components 505

SqlRuleStore 505

OleDbRuleStore 505

FileRuleStore 506

Rule Set Deployment Driver 507

Business Rules Language Converter 507

Business Rules Engine 507

Policy Class 507

Policy Tester Class 509

BizTalk Server 2010 Rule Engine Extensions 511

Rule Definition and Deployment 511

The Rule Composer 512

Loading Rule Stores 513

Using the Policy Explorer 516

Using the Facts Explorer 520

Composing Rule Conditions 525

Creating Rule Actions 530

Rule Engine Component Configuration 532

Testing Rule Sets 534

Vocabularies 538

Strategies for Vocabulary Versioning 543

Publishing and Deployment 545

The Rules Engine Deployment Wizard 546

Using Rules with BizTalk Server 547

ESB Toolkit 547

RFID Server 548

Using Rules in Custom Code 548

Policy Management in the Administration Console 548

The Call Rules Orchestration Shape 551

Policy-Driven Features of the ESB Toolkit 556

The RFID Server BRE Event Handler 558

Summary 561

11 Rule-Based Programming 563

The Roots of Confusion 563

Declarativity 564

Set-Based Programming 565

Recursive Processing 565

Blended Paradigms 566

Limits of Expressivity 567

Understanding the Rule Engine 568

Understanding Production Systems 568

Understanding Short-Circuiting 571

Using OR Connectives 573

Understanding Implicit Conditions 576

Common Rule Patterns 577

Implementing Quantification 577

Handling Negation-as-Failure 581

Using Strong Negation 583

Designing Rule Sets as State Machines 584

Exploiting Situated Reasoning 587

Rule Engine Mechanisms 589

Understanding Working Memory 589

The Match-Resolve-Act Cycle 590

Introducing the Rete Algorithm 593

Managing Conflict Resolution 594

Forward- and Backward-Chaining 595

Working with Facts 597

Using Typed Fact Classes 597

Handling XML Documents 598

Setting XPath Properties in the Rule Composer 599

XML Type Specifiers 600

Handling XML Namespaces 602

Reading and Writing XML Data 602

Managing Optional XML Nodes 603

Handling ADO.NET DataTable and DataRow Objects 606

Handling Data Connections 607

Handling .NET Types 609

Invoking Static Type Members 613

Optimizing Rule Sets bLb

Controlling Side Effects 615

Optimizing the Rete Network 617

Programming with the Rule API 618

Using the Policy Class 618

Handling Long-Term Facts 623

Implementing Compensation Handlers 624

Using the RuleEngine Class 627

Implementing Custom Rule Store Components 628

Managing Deployment Programmatically 630

Creating Rules Programmatically 633

Summary 637

ESB with BizTalk Server 639

What Is an ESB? 639

Introducing the Enterprise Service Bus 639

What Problems Does an ESB Solve? 640

What Are the Components of an ESB? 641

Dynamic Routing 643

Dynamic Transformation 644

Message Validation 644

Message-Oriented Middleware 645

Is BizTalk a Fully Functional ESB? 645

What Is the ESB Toolkit? 645

History of the ESB Toolkit 646

What Is in the ESB Toolkit? 646

What's the Difference Between Native BizTalk Server and BizTalk

Server with the ESB Toolkit? 646

The Magic Behind an ESB 647

The ESB Toolkit Stack 649

Itineraries 650

Specifying Itineraries 651

The Itinerary Lifecycle 652

Dynamic Resolution: The Resolvers 653

Adapter Providers 655

Service Composition 656

Messaging-Only Implementations 657

Unified Exception Management 658

Exposing Core Services 660

Distributed ESBs 660

REST and BizTalk ESB 661

A Stylistic Comparison 661

Incorporating REST into the BizTalk ESB 662

Management 662

Provisioning and Runtime Governance 662

SLA Enforcement 663

Monitoring 663

Organizational Considerations 664

Ensuring a Smooth Transition 664

Gatekeeper Process 665

Summary 666

Part III Deployment and Administration

13 Administration Console Concepts 669

Introducing the Administration Console 669

BizTalk Group Properties 670

BizTalk Settings Dashboard 672

Group Hub and Query View 678

Applications Node 680

Platform Settings 681

Hosts 681

Host Instances 681

Servers 682

Message Boxes 683

Adapters 684

Summary 685

14 Deployment Concepts 687

The Work to Be Done 687

"Application" as a Formal Concept 689

Where Does It All Begin? (Inside Visual Studio) 691

Folder and Project Structure 691

Namespaces and Assembly Names 692

Applying Strong Names 693

Setting Deployment Properties 694

Fine Foods Solution 696

Deploying from Visual Studio 697

Binding and Starting the Application 698

Edit/Debug Cycle 700

Handling Binding Files During Development 703

Creating and Managing Deployable Packages 704

Other Types of Resources 707

Binding Files as Resources 708

Deployment Scripts as Resources 709

Exporting MSI Files 71Z

Handling MSI Export on a Build Server 713

Deploying MSI Packages to a BizTalk Group 715

Import/Install via Command Line 717

Handling Other Deployables 718

Business Activity Monitoring 718

Rule Vocabularies and Policies 719

Handling Upgrade and Versioning Scenarios 719

Summary 720

Part IV RFID

15 BizTalk RFID 723

RFID Overview 724

The BizTalk RFID Framework 725

Installation Notes for BizTalk RFID 727

Device Applications 731

Vendor Extensions and Extensibility 743

Tag Operations 749

Introducing RFID Processes 756

Exception Handling 771

Debugging (Process Hosting Model) 773

Integration and Deployment Considerations 773

Summary 778

16 BizTalk RFID Mobile 779

Mobile RFID Overview 779

The BizTalk RFID Mobile Framework 780

Installation Notes 781

Device Applications 782

Running Your First Mobile Application 787

Barcode Support 791

BizTalk RFID Mobile Connector Architecture

(Store and Forward) 792

Remote Device Management 796

Summary 798

Closing notes 799

Index 803