Introduction to Web Services, SOA, and ESBs

Introduction toWeb Services, SOA, and ESBs

The Enterprise Integration Perspective

Odysseas Pentakalos, Ph.D.

Chief Technology Officer

SYSNET International, Inc.

[email protected]

Copyright 2006 SYSNET International, Inc. 2

Agenda

� Enterprise Application Integration

� Service Oriented Architecture

� Characteristics, Advantages, Disadvantages

� Web Services


� Phase I, Phase II for SOA 2.0

� Enterprise Service Bus


� Business Processes in a SOA

� BPEL, Current state


Enterprise Application Integration

Definition from Webopedia:EAI is the unrestricted sharing of data and business processes throughout the networked applications or data sources in an organization.

Definition from Whatis.com:EAI is a business computing term for the plans, methods, and tools aimed at modernizing, consolidating, and coordinating the computer applications in an enterprise.


Demand for Enterprise Integration

A number of business drivers are generating growing demand for Enterprise Integration

� Mergers and acquisitions

� Constant need to cut costs

� Constant need to streamline business processes

� Compliance with government regulations

Gartner just released some statistics around the worldwide application integration market…, finding a seven percent jump in spending from 2004 to 2005. Total revenue totaled $8.5 billion in 2005. Gartner says

it looks for a similar growth rate this year in the market.


Requirements

Requirements on an Enterprise Integration solution:

� The solution should not cost much; remember the need for cost-cutting

� Should not require extensive changes to existing applications

� Can handle lots of traffic with high reliability

� Flexible so that management and reconfiguration is easy to do


Enterprise Integration Challenges

� Typical environment consists of a collection of applications

� In-house systems: usually developed on the principle of expediency so extensibility and connectivity options are limited

� Packaged-applications: vendors prefer to keep their system closed and limit integration capabilities


Enterprise Integration Challenges

The architect needs to reconcile incompatibilities and differences at multiple levels:

� Business processes are incompatible

� Data model variations at the syntactic and semantic level

� Differences in the technologies used for each system

� Differences in the timing at which systems operate

� Differences in the interaction patterns: synchronous versus asynchronous

From: “Understanding SOA with Web Services” by Eric Newcomer and Greg Lomow


Enterprise Integration Approaches

Some of the possible ways of accomplishing integration

First taxonomy (from Enterprise Integration Patterns book):

� File Transfer

� Shared Database

� Remote Procedure Invocation

� Messaging

Another taxonomy (EIA JavaWorld article by Abraham Kang):

� Data-level integration

� User interface-level integration

� Application-level integration


Enterprise Integration Scenario

� Loan Broker Scenario; this will be a running scenario throughout the presentation

Customer

Credit Bureau

BrokerBank

Bank

Bank

1

3

2

3

3

From: “Enterprise Integration Patterns” byHohpe and Woolf.

Service Oriented Architecture


Service Oriented Architecture (SOA)

� The definition of services that encapsulate reusable business function

� The use of explicit implementation independent interfaces to define services

� Loosely coupled; services are independent of other services

� Services are registered in a directory service so that they can be located during design or runtime

� Service definition is not dependent on transport protocol used to access it

� Service can be used in a location/platform/technology agnostic approach


SOA Interaction Pattern

� This is the basic interaction pattern in a SOA-based application:

Consumer

Provider Registry1 : Register

2 : Find3 : Bind


Service Granularity

From: “SOA Blueprints Concepts”, The Middleware Company Research Team.


Service Types

Most people think of services as the request-reply type

� Request/Reply – receives information, does something and returns information

� Worker – performs a function or changes state

� Monitor – observes a process and generates events

� Agent – observes a process and takes action based on rules

� Aggregator – combines results from disparate sources

� Process – long running service that implements a business service


SOA Vision

� Multiple applications utilize the services� Multiple service providers make services available� The lifetimes of the services and applications are fairly decoupled

CompositeApplication




Is SOA a new concept?

� It is just a new name for a sought-after approach

� What about DCE, CORBA, RMI, DCOM?� Lack of industry wide standardization prevented complete interoperability and platform independence

� Transport protocols were proprietary or specific to distributed object technology

� Security concerns and firewalls prevented interaction across boundaries

� Bad timing with respect to availability of computational power and other industry dynamics

Web Services


Why Web Services for SOA?

� Earlier middleware technologies were:

� Did not support heterogeneous clients and platforms

� Could not work over the Internet

� Some of the choices were hard to use

� Most of the choices were expensive to buy and maintain

� Web Services address these issues !!


XML: The Data Format

� eXtensible Markup Language (XML)

� An XML document is based on a hierarchy of elements and associated attributes


XML Schema

� XML language for defining structure and constraints of an XML document


XSLT: Transformation Language

� eXtensible Stylesheet Language Transformations


Overview of SOAP

� Definition from SOAP 1.1 Spec

� “SOAP provides a simple and lightweight mechanism for exchanging structured and typed information between peers in a decentralized, distributed environment using XML. ..”

� Simplicity is a key requirement

� Extensibility is another key requirement

� Binding into the network is unspecified

� Supports various message exchange patterns:

� request/respond

� notification

� callback


Structure of a SOAP Message

SOAP envelope

SOAP header

SOAP body

SOAP block

SOAP block

SOAP block

SOAP block


Example SOAP Message

<soap:Envelopexmlns:sn="http://www.sysnetint.com/stockquote/"xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"soap:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/"><soap:Header>

<h:from xmlns:h=http://www.sysnetint.com/Header>[email protected]

</h:from></soap:Header><soap:Body>

<sn:GetStockQuotePrice><sn:symbol>MSFT</sn:symbol>

</sn:GetStockQuotePrice></soap:Body>

</soap:Envelope>


Example Web Service

getQuote(String symbol)

<soap:Envelope xmlns: …><soap:Body>

<m:getQuote xmlns="urn:xmethods-delayed-quotes“><symbol xsi:type="xsd:string">MSFT</symbol>

</m:getQuote></soap:Body>

</soap:Envelope>

HTTP

HTTP

<soap:Envelope xmlns: …><soap:Body>

<n:getQuoteResponse xmlns="urn:xmethods-delayed-quotes"><Result xsi:type="xsd:float">24.67</Result>

</n:getQuoteResponse></soap:Body>

</soap:Envelope>


WSDL: Introduction

� Is an XML document itself with a given schema

� Describes the interface to a Web Service

� Separates the abstract definition of a service from the implementation

� Describes the what, the how, and the where of the Web Service

� Similar to predecessors such as IDL of CORBA and DCOM


WSDL Describes Services

� Web Services Description Language (WSDL)

WSDL

Platform Tools

Client Stubs

Server Stubs

Client Impl.

Server Impl.

Requests


WSDL: The description language

� Standard for describing services and their bindings.� <definitions>: is the container and defines name and namespaces for WSDL document.

� <types>: defines any custom data types that are passed in/out of the service.

� <message>: models the data that is exchanged within method calls

� <portType>: operations supported through an endpoint of a web service

� <binding>: associates a portType with a protocol

� <service>: associates a binding with a specific URL


WSDL: Sample document

� Demo of sample from W3C


UDDI History

� Originally defined for the purpose of providing a public business and service registry

� First few versions came out through UDDI.org

� UDDI v3.0 is an OASIS standard

� Currently private registries are more popular in support of SOA-based applications


UDDI Basic Concepts

� Defines a number of elements as part of the data model

� businessEntity, businessService, bindingTemplate, tModel

� Data Model takes the form of an XML document

� Spec defines APIs for inquiry, publication, and security

� The UDDI registry itself is exposed as a set of Web Services


UDDI Scenarios

� Design time use

Developer searches through the private registry for existing services that can take part in an orchestrated application

Interest is in the functionality but not the specific instances

� Run time use

Application searches through the private registry for the service but focus is now on implementations of it


EAI using Web Services

UDDI Server

WSDLWSDL

WSDL

Query

Invoke

Query

Invoke


Web Services: Phase 1

� Services were developed to support a specific purpose

� Registration of interfaces is not necessary

� Discovery and utilization of interfaces is done at design time

� Web Services provide platform independence



Loan Broker with Web Services


Web Services-based Integration

� Advantages (Tactical Approach)

� Easy to implement

� Fairly low cost

� Does not require enterprise-wide agreements

� Disadvantages

� Fairly limited in terms of message exchange patterns

� Lower reusability of the services

� Consistency across web services implementations is limited


SOA Vision

� Multiple applications utilize the services� Multiple service providers make services available� The lifetimes of the services and application are fairly decoupled





WS-* Standards

� WS-Addressing – Need a standard approach for naming endpoints across protocols

� WS-ReliableMessaging – Need to support reliable messaging on top or unreliable protocols

� WS-Coordination/WS-Transactions – Need to support transactions across services

� WS-Security – Security is crucial for Web Services

� WS-Policy – Need to be able to specify non-functional and QoS requirements.


WS-Addressing


Why do we need Addressing?

� To find a Web Service you need a URL. Simple right?

� Address the web service with the URL, set the action in the HTTP headers and you are set.

� How about intermediaries?

� How about SOAP over non-HTTP protocol?

� How about dynamically generated web services?


Basic Concepts: Endpoint References

� Endpoint references:

� Dynamic generation and customization of service endpoint descriptions.

� Referencing and description of specific service instances that are created as the result of stateful interactions.

� Flexible and dynamic exchange of endpoint information in tightly coupled environments where communicating parties share a set of common assumptions about specific policies or protocols that are used during the interaction.


Endpoint Reference Definition

� An Endpoint Reference consists of:� Address: is a URI and identifies the address of the endpoint

� ReferenceParameters: XML elements that are necessary in order to successfully interact with the web service

� Metadata: provides an extensible container for metadata that describes the endpoint

<wsa:EndpointReferencexmlns:wsa="http://www.w3.org/2005/03/addressing"><wsa:Address>http://example.com/fabrikam/acct</wsa:Address></wsa:EndpointReference>


Message Information Headers

� Message Information Headers: headers to allow messages to be addressed to an endpoint all within the SOAP message.

� To: The URL of the target service

<wsa:To>http://localhost/MyServiceTest</wsa:To>

� From: The ERP of the message’s sender.

<wsa:From>

<wsa:Address>http://localhost/YourService</wsa:Address>

</wsa:From>


Message Information Headers

� ReplyTo: The ERP to which the response should be sent to.

� FaultTo: The ERP to which the SOAP fault should be sent to.

� MessageID: Is a URI that uniquely identifies the message

� Action: Takes the place of the SOAPAction

� RelatesTo: Element that indicates the MessageID of the caller.


WS-Reliable Messaging


Why do we need Reliable Messaging?

� Network failures may prevent messages from being delivered

� Middleware failures may cause messages from being delivered

� Intermittent failures may cause duplicates

� Routing delays may cause out of order delivery


WS-Reliable Messaging Concepts

� Sequence: reliable messaging is enforced within the context of a sequence between two endpoints

� Acknowledgments: an ack from the server indicates the sequence range of messages received

� Delivery Assurance Policy: endpoints can negotiate the delivery policy


Reliable Messaging Model

ApplicationSource

Initial Sender

ApplicationSource

Ultimate Receiver

Send Delivery

RM SourceTransmit

RM DestinationReceiveAcknowledge


Reliable Messaging ProtocolEndpoint

AEndpoint

BCreate Sequence

Create Sequence Response (Identifier)

Sequence (Identifier, MessageNumber=1)


Sequence (Identifier, MessageNumber=3, LastMessage)

Sequence Ack. (Identifier, Range=1,3)


Sequence Ack. (Identifier, Range=1..3)

Terminate Sequence


Sequence Message

<wsrm:Sequence ...>

<wsu:Identifier> [URI] </wsu:Identifier>

<wsrm:MessageNumber> [unsignedLong] </wsrm:MessageNumber>

<wsrm:LastMessage/>?

<wsu:Expires> [dateTime] </wsu:Expires>?

...

</wsrm:Sequence>


Sequence Example

<wsrm:Sequence>

<wsu:Identifier>

http://fabrikam123.com/abc

</wsu:Identifier>

<wsrm:MessageNumber>10</wsrm:MessageNumber>

<wsrm:LastMessage/>

</wsrm:Sequence>


Sequence Acknowledgement

<wsrm:SequenceAcknowledgement ...>


[ <wsrm:AcknowledgementRange ...

Upper="[unsignedLong]"

Lower="[unsignedLong]"/> +

| <wsrm:Nack>[unsignedLong]</wsrm:Nack> + ]

...

<wsrm:SequenceAcknowledgement>


Sequence Ack Example

<wsrm:SequenceAcknowledgement>

<wsu:Identifier>

http://fabrikam123.com/abc

</wsu:Identifier>

<wsrm:AcknowledgementRange Upper="2" Lower="1"/>



</wsrm:SequenceAcknowledgement>


Create Sequence Message

Request

<wsrm:CreateSequence ...>

...

</wsrm:CreateSequence>

Response

<wsrm:CreateSequenceResponse ...>


...

</wsrm:CreateSequenceResponse>


Sequence Termination

<wsrm:TerminateSequence ...>


...

</wsrm:TerminateSequence>


Transactions


Why do we need Transactions?

� In an application based on an orchestration of Web Services all services involved in an operation must come to an undisputed resolution.

� Typical ACID semantics are useful for some but long-running transactions need different transactional support.


Atomic Transactions

� Atomicity: on completion either all actions or none are completed.

� Consistency: consistency is the system is preserved after processing multiple concurrent transactions

� Isolation: intermediate states of the system are not observable outside the transaction

� Durability: if a transaction commits, the changes are preserved even in after a system failure


Business Transactions

� Long-running transactions: transactions may take hours, days or weeks to complete

� Need to be able to select a subset of the original participants before a commit

� Compensating transactions: operations that are used to restore the state in case of a failure


Standards and scope

� WS-Coordination:

� allows a distributed system to establish a group of participantsaround an activity

� participants can register interest in participating in the outcome

� Coordination protocol can vary but operates upon completion of the activity

� WS-Atomic:

� Handles ACID transactions and defines variations of 2PC

� WS-Business Activity:

� Allows for the definition of nested scopes of operations

� Allows for the definition of compensating transactions


Coordinator


Interactions for Coordination


Why do we need Security?

� Protect messages from someone eavesdropping on the network

� Prevent tampering with the messages

� Currently security for web services is done at the transport layer (SSL).

� This won’t work when the same message has multiple links to traverse

� Need the ability to selectively encrypt parts of the message.


WS-Security Specifications

� Authentication Profiles

� Username/password

� Kerberos

� X.509 Certificates

� Message Integrity

� XML Signature

� XML Key Management

� Message Confidentiality

� XML Encryption


SOA for Enterprise Integration

� They all need to talk to one another despite running on top of different protocols

� They need support for all the WS-* services we talked about

� The data structures between the elements may not be identical

� Need to be able to support various message exchange patterns

Service ServiceService

Service ServiceService

Service


Enterprise Service Bus (ESB)

� Abstract the development of service components from wire protocol and deployment platform from below and the business processes above

� Facilitates the requirements of a highly-scalable, fault-tolerant, message-driven, service-oriented enterprise.

� Utilizes XML and related-technologies for unifying the message format

� Provides transformation and routing capabilities to the bus as a service

� Provides centralized management capabilities of the distributed services components


ESB Architecture

Provides a standards-based bus and adapters for multiple transports and protocols along with common services


Java Business Integration

� High-level architecture of JBI framework (JSR 208)


Business Processes

� Composition of services into a business process

ClientTravel

Reservation

United AirlineReservations

Delta AirlineReservations

Hertz CarReservations

HotelReservations


BPEL

� Business Process Execution Language:

� Language for describing business processes

� Convergence of a number of earlier efforts (WSFL, XLANG)

� Invoke operations in synchronous/asynchronous mode

� Invoke operations sequentially or in parallel

� Define fault and transaction handling

� Compose business processes from business processes


Loan Broker with BPEL

� Broker is defined as a BPEL document

Customer

Credit Bureau

BPELProcess

Bank

Bank

Bank

1

3

2

3

3


Enterprise Integration Design Patterns

Design patterns for Enterprise Integration

� Fill the gap between high-level architecture and implementation

Pattern Classification

� Messaging Systems: basic concepts in integration with messaging

� Messaging Channels: describe channels used for integration

� Message Construction: describe message types and intent

� Message Routing: describes provision of routing and brokering

� Message Transformation: variations of message translator

� Messaging Endpoints: connecting to and from the messaging system

� System Management: management of the integration infrastructure


Sample EI Patterns

� Messaging Channels

� Datatype Channel: all messages on a given channel are of the same type

� Invalid Message Channel:what to do about invalid messages.

� Dead Letter Channel: what to do about undeliverable messages.


Sample EI Patterns

Messaging Routing

� Content-based Router: routes message based on the message’s content

� Recipient List: routes message to dynamic list of recipients

� Splitter/Aggregator: one splits message into sub-messages; other one aggregates back into one


Application of EIP

Loan Broker using Enterprise Integration Patterns

Customer

Credit Agency

Endpoint

Data Enricher

Recipient List

Aggregator Normalizer Translators

Mule: Enterprise Service Bus


Mule EIP Server

� Mule is an open source framework based on the concept of an ESB (mule.codehaus.org)

� What’s with the name?

� Follows an event-driven architecture

� System consists of producers, consumers and a message server

� Configuration of the routing of events between components takes place outside the scope of consumers/producers

� Container follows the SEDA architecture model

� Master’s thesis work of Matt Welsh at UC Berkeley (http://www.eecs.harvard.edu/~mdw/proj/seda/)


Mule Architecture


Conclusion

� SOA has great potential for finally achieving reusability

� Web Services is the platform that will make SOA a reality

� Existing holes to achieve the full vision are currently in development or production

� An Enterprise Service Bus provides the glue to make SOA a reality today

Backup Slides


Mule Configuration

An XML file defines the integration model:

� connector: Configuration of the components that attaches to the various providers

� container-context: Configuration of the container that will provide the components

� endpoint-identifiers: Configuration of endpoints that provide services

� interceptors: code to execute before and/or after a component

� model: Configuration of the components (UMO) that implement the services


Mule-based Loan Broker

Loan Broker

Credit AgencyGateway

Credit Agency

LenderService

BankingGateway

Bank 1

Bank 2

Bank n


Service Definition

� Loan Broker: (REST Web Service) receives requests for loan quotes

� Credit Agency: (EJB Service) provides credit information about the customer

� Lender Service: (Web Service) receives customer loan requests with credit information and forwards requests to appropriate banks

� Banks: (Web Service) provide quotes on customer loan requests based on request and credit information


Endpoint Definition

<endpoint-identifiers><endpoint-identifier name="LoanBrokerRequestsREST"

value="jetty:rest://localhost:8080/loanbroker"/><endpoint-identifier name="LoanBrokerRequests"

value="vm://loan.broker.requests"/><endpoint-identifier name="LoanQuotes" value="jms://esb.loan.quotes"/><endpoint-identifier name="CreditAgencyGateway" value="jms://esb.credit.agency"/><endpoint-identifier name="CreditAgency"

value="ejb://localhost:1099/local/CreditAgency?method=getCreditProfile"/><endpoint-identifier name="LenderGateway" value="jms://esb.lender.service"/><endpoint-identifier name="LenderService" value="vm://lender.service"/><endpoint-identifier name="BankingGateway" value="jms://esb.banks"/><endpoint-identifier name="Bank1" value="axis:http://localhost:10080/mule"/><endpoint-identifier name="Bank2" value="axis:http://localhost:20080/mule"/><endpoint-identifier name="Bank3" value="axis:http://localhost:30080/mule"/><endpoint-identifier name="Bank4" value="axis:http://localhost:40080/mule"/><endpoint-identifier name="Bank5" value="axis:http://localhost:50080/mule"/>

</endpoint-identifiers>

Loan Broker Definition

<model name="loan-broker"><mule-descriptor name="LoanBroker"implementation="org.mule.samples.loanbroker.esb.LoanBroker"><inbound-router><endpoint address="LoanBrokerReqREST“ transformers=“RestReqToCustReq”/><endpoint address="LoanBrokerRequests"/>

</inbound-router><outbound-router><router className="org….outbound.OutboundPassThroughRouter"><endpoint address="CreditAgencyGateway"/>

</router></outbound-router><response-router timeout="1000000"><endpoint address="LoanQuotes"/><router className="org…..routers.BankQuotesResponseAggregator"/>

</response-router></mule-descriptor>

Credit Agency Definition

<mule-descriptor name="CreditAgencyGateway"

implementation="org.mule.components.builder.ReflectionMessageBuilder"><inbound-router>

<endpoint address="CreditAgencyGateway"/></inbound-router><outbound-router>

<router className="org.mule….FilteringOutboundRouter"><endpoint remoteSync="true“

transformers="LoanQuoteRequestToCreditProfileArgs"responseTransformers="CreditProfileXmlToCreditProfile“address="CreditAgency"/>

<endpoint address="LenderGateway"/></router>

</outbound-router></mule-descriptor>


Banking Gateway Definition

<mule-descriptor name="BankingGateway"implementation="org.mule….BridgeComponent"><inbound-router><endpoint address="BankingGateway"/>

</inbound-router>1<outbound-router><router className="org….StaticRecipientList"><reply-to address="LoanQuotes"/>

</router></outbound-router>

</mule-descriptor>

Introduction to Web Services, SOA, and ESBs

Documents

Transcript of Introduction to Web Services, SOA, and ESBs