ARCH-10: Desperately Seeking SOA – The Refactoring Journey for QAD Fred Yeadon QAD Inc.

ARCH-10: Desperately Seeking SOA –

The Refactoring Journey for QAD

Fred YeadonQAD Inc.

QAD Proprietary

ARCH-10: Desperately Seeking SOA -- The Refactor Journey for QAD

2

TopicsThe ProblemQAD Reference ArchitectureThe Refactoring Process

QAD Proprietary


3

QAD GLOBAL FOOTPRINT

NASDAQ: QADI

Founded: 1979

IPO: 1997

FY05 Est. Revenue: $232-237 million

Licensed Sites: 5,200 worldwide

Customer Renewals: 90%+

Offices: 26 countries

Employees: 1,275 worldwide

Headquarters: Santa Barbara, CA

Fiscal Year End: January 31

ABOUT QADQAD COMPANY INFORMATION

A leader in providing ERP and supply chain execution software applications and services to global manufacturers in target vertical markets.

QAD Snapshot

QAD Proprietary


4

QAD Serves Global Manufacturers

AUTOMOTIVE ELECTRONICSCONSUMER PRODUCTS

FOOD & BEVERAGE MEDICALINDUSTRIAL

QAD Proprietary


5

Our Core Issues

Very large legacy base

Growing set of acquired/partner products

Tight UI – business logic coupling

Over 8,000 installed sites in over 26 countries.

Over 12,000 source files in MFG/PRO.

Customers resist upgrades!

Complementary add-ons. Progress and non-Progress. Interoperability challenges!

Difficult to place new skin on product.

Little use of AppServers. Most ‘APIs’ are UI-

dependent!

QAD Proprietary


6

What Changes Within Our Applications?

User interface– Technology– Fashion

Business logic and features– Customer/Market– Government/Regulation– Internationalization/Localization– Customization

Add-on products– Integrations– Acquisitions

Deployment platforms and topologies– Data centers (consolidate or distribute) – Platforms– Scalability

QAD Proprietary


7

Change is Evolutionary But Sporadic …

Refactor!

‘Critical mass’points

QAD Proprietary


8

Historical Solutions – Alternatives To Refactoring

Cloning

Flexibility Simplifies

(locally)ReuseMaintainabilit

y

Overloading

FlexibilitySimplifiesReuseMaintainabili

ty

QAD Proprietary


9

History of Refactoring At QAD …

OBCM

ProjectX

4GL APIs

Entity Framework

ProRefactor Research

QAD Reference Architecture

1999-2001

2001

2004

2002-03

1995-97

QAD Proprietary


10

… And Issues/Pitfalls Encountered

Too limited in applicability.Out of step with PSC direction.Not performant enough.Too difficult to understand.Too far outside limits of 4GL. Insufficient development tooling.

QAD Proprietary


11

Course Corrections Made

Closer dialog with PSCSingle empowered Architecture group

– Executive commitment

Intensive POC and external research– Enterprise Service Bus– Microsoft .NET™– Business Rules Engines– Orchestration/Workflow

QAD Proprietary


12

QAD Architecture Team QAD Architecture Team OrganizationOrganization

Director,Product

Architecture

TechnicalArchitects

ApplicationArchitects

Based Mainly In:• California• Ireland• Australia

Time split between:• Development projects• Planning-POCs-evaluations• Consulting/communication• Standards

QAD Proprietary


13

Topics The Problem QAD Reference Architecture The Refactoring Process

QAD Proprietary


14

What Is Service-Oriented Architecture?

Independent units of application logic Available across a network (web-service) Loosely coupled, collaborating services Expose message-based interfaces (contracts) Callable from any client:

– User Interface (many varieties)– Other services– External applications– Reporting/BI– Process Orchestration– …

QAD Proprietary


15

Evolution of SOA

Hybrid of many maturing practices, standards, and architectures

SOA

DBUI

Legacy applications

Queue

Messaging

Class Interface

O-O and component-based development

XML

SOAP

BPEL

SAMLOpen standards

WSDL

QAD Proprietary


16

Data Access

Business Services

BusinessEntities

BusinessTasks

Structured Unstructured

Workflow

ProcessOrchestrations

Service Interfaces

BusinessRules

Integration

BusinessReports

BusinessIntelligence

Data Warehouse

OLAP

ESB

Develo

pm

en

t & D

ep

loym

en

t Tools

Presentation

Web Local

Controller

Model

Web Service

EDI IV&I

Event Pub

QAD Reference Architecture Overview

QAD Proprietary


17

QAD Reference Architecture: Technology Stack

Business Services

Workflow

Data Access

Develo

pm

en

t & D

ep

loym

en

t Tools

Presentation IntegrationBusinessIntelligence

Sonic ESBSonicMQSonic XML ServerXSLTXQuery

.NET (GUI)HTML (Web)OpenEdge®(ChUI, OpenClient)

OpenEdge 4GLCognos PowerPlay

J2EE (web svc, IV&I)OpenEdge 4GL, AppServer™(event pub, EDI)SOAP, WSDLXSLT

Sonic Orchest ServerBPEL

OpenEdge 4GL, WebSpeed®OpenEdge AppServer, Client NetworkingiLog JRules

OpenEdge RDBMS, DataServer for OracleOracle DB

OpenEdge 4

GL,

WebSpeed

Eclip

se ID

E, p

lug-in

sPro

parse

QAD Proprietary


18

QRA and OERA: Key Differences

Embedded ESB Solution

Business Rules Engine Provided

Separate Workflow Layer

Separate Business Intelligence Layer

Business Reporting Services

Service Call-Outs

Used for inter-product as well as external interoperability.

Use of ILOG JRules.

Sonic Orchestration Server is a user of business services.

Often-neglected portion of ERP business solutions.

Critical and distinct from other types of business services.

Service proxies allow transparent use of external services.

QAD Proprietary


19

Business Services Layer: Responsibilities

Provide application services in support of end-user business processes.– Container for all business logic.

Keep business logic separate from its external ‘users’.– User interfaces– Systems integration frameworks/brokers.– External application products.

Goal: Ensure that business logic can be reused within any business process for any authorized client/user in any supported operating environment.

QAD Proprietary


20

Business Entities: Responsibilities

Implement the business logic that operates on persistent collections of business data.– 80%+ of all application code!

Realize in software the data, relationships and behavior that collectively model useful, intuitive business concepts.– Products and services– Supply and demand– Business and legal documents– Financial records and transaction history– Plans, schedules and forecasts

Goal: Capture all the behavior and properties of the concepts that are central to the business application.

Business Services

BusinessEntities

BusinessTasks

Service Interfaces

BusinessRules

BusinessReports

QAD Proprietary


21

Business Entities and Data Entities

Sales Order

Sales OrderLine

Sales OrderComment

SO LineComment

Sold-ToCustomer

Bill-ToCustomer

CustomerAddress

Ship-ToCustomer

SO Import-Export

Sales OrderTrailer Tax

Line Import-Export

Sales OrderTax Detail

Sales OrderBill

DetailAllocation

SO BillComment

…

…

Sales OrderBusinessEntity

DataEntities

Dataassociations

QAD Proprietary


22

OpenEdge ProDataSets

Dataset

Grand-ChildBuffer

Data-Source

Database

Buffer Buffer

Buffer

Buffer

.

.

.

.

.

.

Buffer

Queries & source buffers

Data-relations &relation-fields

Table, field-mappingsforattach-data-source

ChildBuffer

ChildBuffer

Top-LevelBuffer

Before-imagebuffers

Get-changes dataset

Business Services Layer Data Access Layer

QAD Proprietary


23

Why Use Datasets?

Use OpenEdge high-level FILL method– One 4GL statement traverses an entire dataset.– Replace nested FOR EACH loops with smaller

callback routines.

Remove data relationships from application code– Less code is affected when DB schema changes.

Business logic operates on temp-table buffers, not DB– Support decoupling of business logic and DB access

layers.

QAD Proprietary


24

Topics The Problem QAD Reference Architecture The Refactoring Process

QAD Proprietary


25

A Refactoring Task List

1. Model-definebusinessentities.

3. Wrap legacy inweb-service

façade.

2. Formulatetarget

design-codepatterns.

4. Identifyrefactoringscope and

boundaries. 6. Design methodsrequired for UIs

and APIs.

5. Reorganizelegacycode.

DevelopSOA

Infrastructure.

…

…

A

= top-down

= bottom-up

Design newUI.

QAD Proprietary


26

1. Model, Define Business Entities

Data Model User Interface

BusinessAnalyst

QAD Proprietary


27

Business Entities in MFG/PRO*

Category # Business Entities Identified

Common Codes 26

Products 22

Demand Side 16

Supply Side 14

Manufacturing 13

Inventory & Shipping

21

Service & Support 15

Financials 15

Total 142

* Analysis performed for QXtend Outbound.

QAD Proprietary


28

2. Formulate Target Design-Code Patterns

Purpose: Capture past design experience in a catalog form that can be re-used effectively by other developers.– Disseminate standards in more usable form.

Many external sources …– Design Patterns, by Gamma-Helm-Johnson-Vlissides– Core J2EE Patterns, by Alur-Crupi-Malks– Martin Fowler books– Microsoft Patterns & Practices web site

… but many known patterns are too low-level for OpenEdge 4GL developers!

QAD Proprietary


29

QAD Business Entity Patterns

Inheritance-Related

Extension/Customization-Related

Behavior-Related

Common Field Group Multi-Purpose Table

Extended Data Entity Extended Business Entity Business Entity

Wrapper

Entity Calls Entity Entity Generates History

QAD Proprietary


30

Common Field Group

MFG/PRO Examples:– G/L codes– Multi-currency data– Item-quantity-UM

Data Entity

CommonField Group

fieldfieldfieldfieldfieldfieldfieldfield…

Data Entity

fieldfieldfieldfieldfield…

Data Entity

fieldfieldfieldfieldfield…

fieldfieldfield

QAD Proprietary


31

Business Entity

Calling Program

Business Entity Wrapper

Business Entity Program

New Data Entity Wrapper

DatasetDefinition

DataEntity

Include

DataEntity

Include…

Existing Data Entity Wrappers

run aMethod in …

EntityRegistry

Callbacks MFG/PRO Examples:

– Customizations– Locale-specific

functionality

QAD Proprietary


32

3. Wrap Legacy In Web-Service Façade

New interface, mixed old and new implementation. ‘Scaffolding’ to support incremental refactoring work. Support open interoperability standards. Buy time!

ESB (JMS, Web Services)

<SOAP-ENV:Envelope … > <SOAP-ENV:Body> <diffgr:diffgram … > <salesOrder …> <soNbr>SO001</soNbr> <soShip>DEF</soShip> <salesOrderDetail>

… </salesOrderDetail> </salesOrder> <diffgr:before> <soCust>ABC</soCust> … </diffgr:before> </diffgr:diffgram … > <SOAP-ENV:Body><SOAP-ENV:Envelope>

New/RefactoredComponent

LegacyComponents

3P Apps

QXtend

QAD Proprietary


33

MFG/PRO QDoc Fragment for Sales Order

<salesOrder>

<operation>A</operation>

<soNbr>SO0001</soNbr>

<soCust>001</soCust>

<salesOrderDetail><operation>A</operation>

<sodPart>10-10000</sodPart>

<sodQtyOrd>100</sodQtyOrd>

</salesOrderDetail>

</salesOrder>

QAD Proprietary


34

QAD QXtend Inbound Architecture

MFG

/PR

OM

FG/P

RO

QXtend Inbound (J2EE)QXtend Inbound (J2EE)

Request

Response

Web ServiceWeb Service• Is valid Receiver?• Is valid API?• Is valid version?• Identify MFG/PRO message routing

UIUIAdapter Adapter

APIAPIAdapterAdapter

Logging/AuditingLogging/Auditing

Connection Connection ManagerManager

Transaction

SecuritySecurity

QAD Proprietary


35

4. Identify Refactoring Scope & Boundaries

Create adapter out of legacy code.

OrderProcessing (legacy)

Inventory(legacy)

OrderProcessing

(legacy)

Inventory(refactored)

Inventory adapter

QAD Proprietary


36

Example: MFG/PRO Sales Order Maintenance

One tenth of MFG/PRO 1,250 programs (.i’s and .p’s) 280,000 lines of code 25% contains user interface actions (including

transactional statements) Predominantly version 6-7 Progress (EDITING

blocks, …) 3 overlapping variants!

– On-line SO Maint– EDI SO load– Internal API SO load

QAD Proprietary


37

Pilot Refactoring Efforts Partial refactoring of Sales Order Maintenance

started.– Part of a larger development project

Refactoring of an entire MFG/PRO module still to be launched.– Accounts Receivable– Configured Product Order Entry– Service-Support Call Management– ??

QAD Proprietary


38

5. Reorganize Legacy Code

Some semi-automatable steps:– Consolidate top-level- and sub-programs by business

entity.– Push most code, variables into internal procedures.– Gather related shared references into same procedure.– Remove shared variables, unnecessary global variable

references.– Extract cohesive data structures to separate (persistent)

procedures.– Add ‘getter’ and ‘setter’ methods for external references.

Use tools if possible.– ProRefactor from Joanju Ltd., co-developed with QAD.

Saves developer code ‘hunt-and-peck’ time. Complements top-down modeling/design efforts.

QAD Proprietary


39

6. Design Methods Required for UIs and APIs

Separation of UI is top priority!

Shared Business Logic (refactored)

API Service ProgramUI Service Program

ControllerViews

Integration (QXtend)Model

User Interface (.NET)

Coarse-grainedFine-

grained

Method Method Method Method Method MethodState-less

QAD Proprietary


40

Refactoring Task List (cont)

9. Granularizevalidation &

calculation logic.

7. Convertdatabase totemp-tablereferences.

8. Imposestandardexceptionhandling.

10. Move iterativelogic into

callback routines.

A

= top-down

= bottom-up

QAD Proprietary


41

7. Convert Database To Temp-Table References Convert only database table references associated

with business entity.– Side validation, lookups against other tables left alone

initially.

Define temp-tables like database tables in most cases.– Exception: Normalize any arrays!– Generate full field definitions, not just using LIKE clauses.– Generate temp-table to DB field mappings.

Define business entity dataset.– Including all parent-child relations.

Replace transaction blocks and queries with calls to new ‘save’ and ‘fetch’ methods.– Remove explicit commits and access to entity-related tables.

QAD Proprietary


42

8. Impose Standard Exception Handling

Application and system exceptions are often not handled rigorously– Often surfacing in lower-level routines, then lost/forgotten.– Never explicitly identified as an interface or contract.– Who is responsible for handling each exception?

OpenEdge support is limited– Basic ON ERROR, ON STOP, RETURN ERROR, RETURN-

VALUE, NO-ERROR constructs must be applied consistently by developers.

– Many errors cannot be trapped.– No concept of one routine catching another’s thrown

exception.– Some exceptions should be thrown without raising ERROR

condition, rolling back transactions, or interrupting execution.

QAD Proprietary


43

Sample Exception Handling Code

{ExceptionManager.i super} /* Function prototypes */

do on error undo, leave: /* Equivalent of ‘try’ block */ run throwException(“22”, /* Exception number being thrown */ {&APP-ERROR-RESULT}, /* Severity level */ “orderNum”, /* Field name */ “|”, /* Delimiter for message data */ “ttOrder|” + orderNum /* Substitutable message text */ ). end.

run catchException(“22”, /* Exception number(s) being caught */ ?). /* Target non-fatal severity level */

run catchException(“176,177”, /* Another exception handler */ {&WARNING-RESULT}). /* Reclass errors as non-fatal */

run rethrowExceptions( {&APP-ERROR-RESULT}). /* Minimum severity to rethrow */

QAD Proprietary


44

9. Granularize Validation & Calculation Logic

Move most in-line processing into small methods– Mostly validation, calculations, and defaults.

MFG/PRO sales order examples– validatePriceList– validateTemporaryShipTo– setAddressDefaults– setTaxDefaults– calculateFreight– checkCreditLimit

Some complex common logic not refactored– Example: tax calculation– Write ‘wrapper’ method to hide legacy code.

QAD Proprietary


45

10. Move Iterative Logic Into Callback Routines

for each SalesOrder … <processSalesOrder> for each SalesOrderLine … <processSOLine> . . . end.end.

run processChanges.…

procedure salesOrderRowValidateCallback:…end.procedure salesOrderLineRowValidateCallback:… run super (input dataset dsEntity by-reference ) no-error.end.

procedure processChanges: … end.…procedure traverseBuffers private: … run value(<data entity name> + <Action> + <Event> + “Callback”) in …. run traverseBuffers.end.

EntityLib.p

SalesOrderEntity.p

1

2

3

Without callbacks: With callbacks:

Event!

QAD Proprietary


46

Why Use Callbacks?

Removes much redundant FOR EACH navigation from business logic.– Parent-child data-relations, joins are hard-coded in

dataset definition only.

Supports consistent packaging of business logic.– Sales order header vs ship-to address logic (for

example).– Consistent naming of callback routines.

Supports fine-grained extensions to business logic.– Each callback may be overridden, called as

superprocedure.

QAD Proprietary


47

QAD Present Status

Identifying, documenting design-coding patterns, templates, samples.

Developing SOA infrastructure.– Session state management.– XML serialization-deserialization.– Dispatch calls across ESB, AppServers.– Security

Conducting initial pilot projects using portions of target architecture.– Sonic ESB™.– Business Rules Engine.– Mixture of wrapping and refactoring 4GL code.

Surveying development tools suitable for SOA.– Future PSC Eclipse plug-ins.– Other 3P.

QAD Proprietary


48

Ongoing Challenges!

More complex technology stack. Supporting mix of old and new. Business case for incremental refactoring. Need for tooling to handle added development

complexity.

QAD Proprietary


49

Questions?

ARCH-10: Desperately Seeking SOA – The Refactoring Journey for QAD Fred Yeadon QAD Inc.

Documents

Transcript of ARCH-10: Desperately Seeking SOA – The Refactoring Journey for QAD Fred Yeadon QAD Inc.