A microservice approach for legacy modernisation

Luis [email protected]

uk.linkedin.com/in/lweir@luisw19

soa4u.co.uk/

A MICROSERVICE

APPROACH

FOR

LEGACY

MODERNISATION

Beyond The HorizonJune 2,3 2016

{2}

About me

Luis Weir Oracle Ace Director – Principal Architect

in assisting organisations define and implement solutions and strategies that can helpthem realise the benefits that such technologies have to offer.I am very passionate about technology. I have be the lead authored of two books (OracleSOA Governance 11g Implementation and Oracle API Management 12c Implementation), I am aregular blogger and speaker in major conferences and events. A well-known industry expertespecially when it comes to Oracle middleware technologies I am also an OTN certified SOAblack belt.

I am an Oracle Ace Director, principal architect and a thought leaderspecialised in Oracle Fusion Middleware & Oracle PaaS technologies. Withmore than 15 years of experience implementing IT solutions across theglobe, I have been exposed to a wide wide variety of business problemsmany of which I’ve helped solved by adopting SOA architectural styles suchas traditional SOA, API management and now Microservices. My current focusis

2nd Place1st OTN Cloud HackathonJune, 2016

Cloud Contribution AwardSOA CommunityMarch, 2016

Latest Media:•Oracle Magazine May/June 2016 (http://bit.ly/1RTCAU3)•Systematic Approach for Migrating to Oracle Cloud SaaS(http://bit.ly/1Xr6acs)•Oracle Magazine Jan/Feb 2016 (http://ora.cl/Vhh)•API Management Implementation (http://ora.cl/Gcw)•A Word About Microservices and SOA (http://bit.ly/25Dk5go)

{3}

Introduction

{4}

Agenda

• Modernisation strategies & approach

• Microservice architecture

• Seven conclusions

• Context, driving forces and challenges

{5}

… what is a legacy system?

Monolithic applications that significantly resists modification and evolution [1], seem frozen in time [6] and thus do not fit with the organisations future IT strategy [7]

Characteristics:

• They are business critical [1][2][4][5][6][7][8][12][15]

• Typically been around for a while (in some cases 30+ years) [2][7][8]

• End of life tech stacks [1][2][5][8][12][15]

• Not necessarily defined by age. Lack of vendor support or inflexibility also mean legacy [2][5][7][14]

• They are stable, reliable and performing [4][5][7][14]

{6}

Does the world run on legacy?• 80% of the worlds systems are legacy [3][5]• 200 billion lines of legacy code are still in use [5][7]• 1+ million COBOL programmers world wide maintaining an approximate of 100 billion lines of COBOL code [7][11]

• 75% of financial institutions in Europe are using out-dated systems (legacy) for core business [9]

• COBOL powers 80% of all daily business transactions [17]• 30+ years of legacy modernisation research and 80% of world systems are still “legacy”? [10][7]

[11]

{7}

Driving forces

• Lack of flexibility. Extremely difficult to change to take new requirements affecting new product development and time to market [1][2][3][4][5][7][8][9][12][15]

• High costs of maintenance and change [1][2][3][4][5][7][8][9][12][15]

• End of life tech-stacks(out of support, no new or major feature releases) [1][2][3][5][7][8][9][12][15]

• People [with relevant skills] are scarce[4][5][7][10][11][17]

• Integration is difficult [1][4][5][6][12][15]

• Security (no new security patches) [6][9][13]

{8}

Disruption is not just “digital”

• Amazon first physical book store in Seattle

{9}

Arguments against modernisation• High risk of failure – “we’ve tried before (several times) and failed” [1][2][4][6][7][10][12][15]

• Too costly without clear justification. Ie. When will we see ROI? [1][2][3][4][6][7][8][12][14][15]

• Lack of knowledge and skills –[1][4][5][7][12][10][11][14][17]

• “If ain’t broken don’t fix it”, the system is doing what’s supposed to why change it? [3][6][7][14]

“They [Top management] are always looking for a short term Return of Investment. Once they put the money in, they want to earn it back” [7]

We didn’t fix it last year, and survived. Why should this year be different? [7]

“I think top management doesn’t understand the issue and they don’t give budget for it [legacy modernisation]” [7]

{10}

Legacy are [more] exposed to Murphy’s Law

{11}

Faults are as big as the legacy!!

(2012) 16 million (RBS, NatWest, Ulster Bank) customer accounts

ended up being frozen

(Dec, 2014) Cancelation of 40 flights in UK affecting 1.9k

other flights and 230k passengers

(Jan, 2016) Customers couldn't access accounts for 2 days

{12}

Not failing is not success [16]

Insight into legacy modernisation success rate:

• Success rates vary from 24% to 39%, depending on study and the type of technology project [16]

• 1 in 6 projects (17%) result in "black swan” [16]

• Only 50% of projects deliver 44% of the intended business benefits [16]

“Healthcare.gov is just a frontend. The heavy lifting takes place in the back ends, a extremely complex array of legacy systems that span multiple US federal agencies” [20]

“The worry is that trying to promise a startup-style experience on the back of multiple old and lumbering IT systems. There's a long way to go, even after all the initial issues with accessing the site itself are resolved” [20]

{13}

Usual suspects: reasons for failure• Lack of documentation and knowledge of the system leading to issues like bad scoping, poor estimation and planning, inappropriate target designs and incomplete testing [1][2][3][4][7][8][12][14]

• Wrong approach specially based on “Eating the elephant in one go” (cold turkey)” [1][4][7][8][12][15][16]

• Time pressures. Rushing through project phases to meet deadlines – [1][4][5][7][12][10][11][14][17]

• Lack of people with relevant skills –[1][4][5][7][12][10][11][14][17]

{14}

Agenda





{15}

What is a Microservice? SOA 2.0?

Loosely coupled service oriented architecture with bounded context [27]

“The value of the term microservices is that it allows to put a label on a useful subset of the SOA terminology” [22]

[21]

Functional decomposition of systems into manageable and

independently deployable components[18]

{16}

3 aspects to Microservice Architecture[19]

Architectural

Technical Organisational

{17}

MSA – Technical

Microservice

Run on its own process

Deployed independently

Scales independently

Owns its data

Is stateless

Isolates faults

{18}

MSA - Technical

Legacy System Monolith

Mainframe (ie IBM System Z, S/360)

Storage(ie. DB2, IMS/DB)

Hypervisor (ie. PR/SM –Type 1)

Application Services(ie. CICS)

Hardware Resources(ie. DADB, IDMS,IEDN)

Operating System(ie. z/OS, z/VSE)

Batch Services(ie JCL,JES,3rd p.)

Modern Monolith

Relational Database

Hardware Resources

Host OS

Hypervisor (type 1 or 2)

Guest OS (VMs)

Bin/Libs(MREs, Interpreters, etc)

Application Server

Application(ie. ESB)

Services

Any Hardware

Container Engine

Microservice Architecture

Mongo

Cassandra

Oracle HBase

Neo4j

Hardware Resources

Host OS

Guest OS (VMs)

Scala Java

Ruby

Node

Jolie

Services

Any Hardware

Container Engine

Bin/Libs

Bin/Libs

Bin/Libs

Bin/Libs

Container Engine

Bin/LibsHypervisor

(type 1 or 2)

{19}

MSA – Architectural

Microservice

Bounded context

Single responsibility

Choreographed

Smart endpoint and dump pipe

Polyglot

API gateways

{20}

Order & FulfillmentDomain

Customer RelationsDomain

ERPDomain(P2C, R2C,HR, GL,Billing,etc)

MSA – Architectural

API Gateway(s)

Microservice Architecture

AsyncCommunication

Microservice Architecture Legacy

Message Pipe

Mobile Apps

Adapter

Adapter

SyncCommunication Managed API Microservice

MonolithServiceChoreography

Contact

Customer

BoundedContext

Shipment

Order Product

Web Apps Applications

{21}

MSA - ArchitecturalPattern Traditional SOA MSA

Monolith pattern (http://bit.ly/1Gjr2Y0) Yes No

Polyglot Programming & Persistence(http://bit.ly/18BvDIj & http://bit.ly/1XYiak2)

Not traditionally (use of Suites)

Yes

API gateway pattern (http://bit.ly/1WTyNLJ) Yes Yes

Orchestration (http://bit.ly/1U0SWil) Yes No

Choreography (http://bit.ly/1ssALZQ) No Yes

Event Collaboration (http://bit.ly/25Dk7oE) Yes Yes

Canonical Schema (http://bit.ly/1r6KkfK) Very common No

Schema centralization (http://bit.ly/1sVlqkc) Very common No

Decouple Contract (http://bit.ly/1O8mVpm) Yes Could be….

Bounded Context (http://bit.ly/1o7AK8B) Some times Yes

Ubiquitous Language (http://bit.ly/1c8nXQe) Some times Yes

Bulkhead (http://bit.ly/1c8nXQe) Not really… Yes

Tolerant Reader (http://bit.ly/1aa4mr9) Some times Yes

Client-side Service Discovery(http://bit.ly/1OunUyq)

Initially only(service registry)

Recommended

Server-side Service Discovery(http://bit.ly/1X3RmzA)

Yes Yes

ESB Pattern (http://bit.ly/1ZlSKeT) Yes Across bounded contexts or domains (dump pipe)

[19]

Yes = Applied most of the timeNo = Not applied most of the time

{22}

MSA – Organisational

Microservice

Teams organized around business capabilities

Small teams

You build it you run it

Decentralised governance

Culture of automation

Products not projects

{23}

MSA – Organizational

Development and support teams organized by technologies resulting in siloes(Conway’s law in action)

SOA

Support

Team

DB

Support

Team

UI

Support

team

UI Dev Team

Database Dev Teams

SOA Dev Team

Project Teams

ComsGaps

Multi-disciplinary [small] teams organized by business capability resulting in modular systems

CustomerUI D

B

MW

OrdersUI D

B

MW

ItemsUI D

B

MW

ShipmentUI D

B

MW

Traditional Operations Model

DevOps /Continuous Delivery

MSA Operations Model

[19]

CustomerMicroservice

OrdersMicroservice

ItemsMicroservice

ShipmentMicroservice

{24}

Recommend watching

https://www.youtube.com/watch?v=yPvef9R3k-M

DDD & MicroservicesBy Eric EvansGoto Berlin, Nov 2014

https://www.youtube.com/watch?v=nMTaS07i3jk

State of the art in MSABy Adrian CockcroftDockercon Amsterdam, Nov 2015

https://www.youtube.com/watch?v=wgdBVIX9ifA

MicroservicesBy Martin FowlerGoto Berlin, Nov 2014

{25}

Recommend watching (continuation)

http://bit.ly/1UBDQ2T

7 deadly sins of MicroservicesBy Daniel BryantJan 2014

https://www.youtube.com/watch?v=PFQnNFe27kU

Principles of MicroservicesBy Sam NewmanDevoxx Belgium, Nov 2015

{26}

Ok cool, but why MSA for this?

ModularityEat the elephant one piece at the time. Phased implementation approach. Starting small. Small teams owning full lifecycle of their piece(a business capability)[23]

Segmented complexitySeparate a big problem into smaller problems handled by small teams ensures mental models are retained avoiding a “legacy in the making” [5]

Easy of deployment / speedMoving away from entire system deployment (ie. “one line change to a million-line-long monolithic”). Deploy services independently and fast (ie. with containers) and introduce automation (continuous delivery) [23]

{27}

(continuation)

Scalability & resilienceScale independently and possibly on-demand. Bulkheads to isolate problems and avoid whole system failures (avoiding the cascade effect), Then purposely test resilience [23]

Enabling cloud transitionBuilding container-based modular applications whilst adhering to basic principles (like 12 factor [30], Lehman’s law [31], and the reactive manifesto [33]) cloud adoption is a real option

Breaking organisational silosOrganise small teams based on business capabilities in order to avoid organisational silos being reflected in the way systems are built (Conway's law [32])

{28}

Agenda





{29}

Modernisation Strategies

Cold Turkey [8]

• Big-bang approach• High risk and Costly (big project)• Reduced business benefits

Monolith

Ie. COBOL

Monolith

Ie. Java

Tool

Refactor / ReplaceConvert legacy code and/or replace with COTS

Monolith

Ie. COBOL

Tools, manual,

any other..

RebuildRebuild from ground up in modern language in one-go

Monolith

Ie. Java

Apps

Chicken Little [8]

• Phased approach based on coexisting with legacy using gateways• Less risky but complex

Monolith Decompose

A1

Gateways

An

Re-engineerDecompose legacy into modules and then rebuild whilst coexisting with legacy using gateways

Butterfly [15]

• Phased approach based on coexisting with legacy without gateways• Less risky but complexRe-engineerDecompose legacy into modules and then rebuild whilst coexisting with legacy using data access allocators and chrysaliser

Monolith

Decompose

A1

An

Chrysaliser

Wrapping[15] • As little modification to legacy as possible (tactical approach in nature)• Used of wrapper technologies like CICS, screen-scrapers or ESBs with legacy adapters• Lower risk, quicker results• Higher costs, complexity and risk in the mid-long term

Monolith

A1

Wrappers

An

{30}

A blended approach

1. API-fication

2. Re-engineer & coexist

3. Switch-over and start over

{31}

I) API-fication (wrap)

OnpremiseTerminals

Oracle Cloud PaaS

MainframeDB2

OrderMng

Inventory

Pricing

Product

ShipmentConsignment

s

Sales Purchasing Payables

Receivables GL Etc..

WeblogicAPI Gateway (OAPCS)

Oracle SOA Suite(11g or 12c when Adapter is certified)

Product API

CICS PROGRAMS Oracle Connect

JCA Legacy Adapter

REST Adapter

Native

Apps

Web

Apps

OACCS MCSJet on Node Mobile Backend

{32}

Oracle Cloud PaaS

APICS Gateway

Native

Apps

Web

Apps

I) API-fication (wrap)

Terminals

MainframeDB2

OrderMng

Inventory

Pricing

Product

ShipmentConsignment

s



Weblogic

Oracle SOA Suite(11g or 12c when Adapter is certified)

CICS PROGRAMSOracle Connect

JCA Legacy Adapter

REST Adapter

OACCS MCS

ICS

Jet on Node

Onpremise Agent

Mobile Backend

Onpremise

Rest endpoint

Product API

{33}

Weblogic

Oracle SOA Suite(11g or 12c when

Adapter is certified)

Oracle Cloud PaaS

APICS Gateway

Native

Apps

Web

Apps

II) Re-engineer (MSA) & coexist

Terminals

MainframeDB2

Inventory Orders

Pricing

Product

ShipmentConsignmen

ts



CICS PROGRAMS

JCA Legacy Adapter

REST Adapter

OACCS MCS

ICS

Jet on Node

Onpremise Agent

Mobile Backend

Onpremise

Rest endpoint

OACCS ODCS

ProductProductProduct API

Oracle Connect

ODIDB2 KM

Sync

{34}

Weblogic

Oracle SOA Suite(11g or 12c when

Adapter is certified)

Oracle Cloud PaaS

APICS Gateway

Native

Apps

Web

Apps

III) Switch over & start over

Terminals

MainframeDB2

Inventory Orders

Pricing

Product

ShipmentConsignment

s



CICS PROGRAMSOracle Connect

JCA Legacy Adapter

REST Adapter

OACCS MCS

ICS

Jet on Node

Onpremise Agent

Mobile Backend

Onpremise

Rest endpoint

OACCS ODCS

Product APIInventory API

ODIDB2 KM

updates

ProductProductProducts Changes in terminals blocked

{35}

Oracle PaaS Cloud

APICS

OACCS

Product

Inventory

Orders

Pricing

ShipmentConsigments

etc

Gateway(s)

OACCS Jet on Node MCSMobile Backend JCS ADF, etcOther CS’s

Sales

ODCS

ICS/SOACS

…

…

…

The goal…

Web

Apps

Native

Apps

Onpremise

MainframeDB2

xxx

{36}

Functional Decomposition

Activity

Output

Team

Business Process Flows (L1-3)

Logical Data Model

Business Rules

Document Analysis

Black-Box Analysis

SME Interviews

Process Modeling

Interface catalogues

Top-down requirement analysis

Business Analysts

Solution Architects

User Stories

DomainModel

Decomposition Approach

Technical Decomposition

Code Reverse Reengineering

Bottom-up requirement analysis

Reversed UML (L4-5)

Gap Analysis

Code Functionality Mapping

Software Engineers

Business Analysts

Solution Architects

Reversed Physical Data Model

Reversed Business Rules

Business Logic / Rules Discovery

Solution Design

Solution Design

Business Capability Model

Target State Solution Architecture

UX Patterns

Product Backlog & Sprint Planning

Build Planning

UX Designers

Business Analysts

Solution Architects

Product Manager

{37}

The WAGILE Approach

Sprint 1

Sprint 2

Sprint 3

Sprint 4

Sprint 5

Sprint N

W-Agile Methodology

Functional

Technical

Solution Design

New Requirements

DecompositionsEnd to end

testing

SIT, UAT, PPT

Release Support

Several iterations – small scopeFew iterations – large scope

Solution build & component test

Factory Management & Governance

Centre of Excellence

Release

Mgmt. &

Deployment

Detail Design

Sprint Backlog

Sprints

Scrum Master

DevOps

Pre-SITModernisation

Factory

Build & Test

Sprints

DevOps

Work Items Job Card

Cutover

Go-Live

Rollout

{38}

Agenda





{39}

I) Avoid the obvious!

Avoid legacy to legacy modernisation!!

A fundamental goal of legacy systems modernisation is that the target system

doesn’t become a legacy [8]. Modular systems are easier to evolve

{40}

II) No big bangs or cold turkeys

One piece a the time!

Understand the problem. Slice and dice your elephant by defining boundaries in the

business capabilities. Modernise one piece at the time. Starting small

Replenishment

Orders

SalesOrders

Logistics

Tracking

{41}

III)One domain to rule them all [26]

Gather together those things that change for the same reason, and separate those things that change for different reasons [23][25]

[24]

Domain driven design (DDD) divides up a large system into Bounded Contexts, each of which can have a unified model - essentially a way of structuring Multiple Canonical Models[24]

{42}

IV) The main prereq is setting the right goals

Do “You have to be this tall to use microservices[28]”? – perhaps not!

Sure you need a degree of operational maturity to adopt microservices, however

maturity is a journey that can be achieved by setting the right goals and objectives.

So set the right goals!!!

[27]

{43}

V) Optimise for speed not efficiency [29]

Speed means learning about your customers and giving them what they want at a faster

pace[29]. Use simple patterns automated by tooling [27]. Learn the walk before running.

Speed wins in the marketplace [27]

{44}

VI) 30+ years of research [5][10] proves that:

What you have seen early is just a point of view (mine!). Do dui-diligence and find the right approach for your environment. Get inspiration from lots of research work

There is no silver bullet [26]

{45}

VII) The world runs on legacy

So, let’s learn legacy!

Credible research shows that there already is a huge skill gap in the industry on COBOL, RPG, PL/I, FORTRAN, PASCAL, C++, etc [4][5][7][10][11][17].

“Study the past if you would define the future” -Confucius

{46}

References (I)[1] Legacy Information Systems: Issues and Direction by Bisbal, J.; Lawless, D.; Bing Wu; Grimson, J.; October 1999 (http://bit.ly/1sVnOYo)

[2] Software Engineering by Ian Sommerville, 10th edition 2015 (http://amzn.to/1Uo5QnB)

[3] Legacy systems definition by Techopedia(http://bit.ly/1UXRpf8)

[4] Portfolio Analysis, The business case, Aand solution for reducing risk in legacy environments by Modern Systems (http://bit.ly/22Ezw69)

[5] The burden of legacy by Dr. Toby Sucharov and Philip Rice of Erudine (http://bit.ly/1XYkSWM)

[6] Legacy systems continue to have a place in the enterprise, by computerweekly, June 2008 (http://bit.ly/XeUQ5M)

[7] How Do Professionals Perceive Legacy Systems and Software Modernization? By Utrecht University, 2014 (http://dl.acm.org/citation.cfm?id=2568318)

[8] DARWIN: On the Incremental Migration of Legacy Information Systems by Michael L. Brodie GTE laboratories and Michael Stonebreaker University of California Berkeley, March 1993

(http://db.cs.berkeley.edu/papers/S2K-93-25.pdf)

[9] Banks still handicapped by IT legacy, by computerweekly, May 2012(http://bit.ly/1U0UN6N)

[10] Moving from your legacy system- COBOL conversion, cost, and consequences bt Logapps LLC, August 2015 (http://bit.ly/24orJJ9)

[11] Who Maintains The Legacy Code by Vishwesh Bhat, October 2015 (http://bit.ly/1Y7L7du)

[12] A survey research into legacy system migration by Jesus Bisbal et al, 1997

(https://www.scss.tcd.ie/publications/tech-reports/reports.97/TCD-CS-1997-01.pdf)

[13] Security Concerns for Legacy Systems. An on-going process by Robert Annett, March 2015 (http://www.codingthearchitecture.com/2015/03/07/security_concerns_for_legacy_systems.html)

[14] Working with Legacy Systems, A Practical Guide to the Systems we Inherit and Maintain by Robert Annett. 80% complete in May 2016 (https://leanpub.com/WorkingWithLegacySystems)

{47}

References (II)[15] The Butterfly Methodology: A Gateway-free Approach for Migrating Legacy Information Systems by Bing Wu et al (http://dl.acm.org/citation.cfm?id=852129)

[16] Why legacy modernization projects fail, advice on how to how to steer your project toward Success by David Weldon, October 2014 (http://bit.ly/1t852xy)

[17] Academia needs more support to tackle the IT skills gap by Microfocus, March 2013

(http://bit.ly/22EAb7E)

[18] Microservice Architectures by Dr. Andreas Schroeder (http://bit.ly/1TOGZK8)

[19] Oracle Service Bus for Modernisation strategies by Robert Wunderlich, Ricardo Ferreira, Luis Weir, October 2015 (http://bit.ly/1ZlWtZR)

[20] Misunderstanding the Problem? By John Marshall, October 2013 (http://bit.ly/1U0TVyT)

[21] Tweet by Adrian Cockcroft (https://twitter.com/adrianco/status/542850261782237184)

[22] Microservices by Martin Fowler (minute 14), GOTO conference, Berlin November 2014

(https://www.youtube.com/watch?v=wgdBVIX9ifA)

[23] Building microservices, designing fine-grained systems by Sam Newman, October 2015

(http://shop.oreilly.com/product/0636920033158.do)

[24] Bounded context by Martin Follower, January 2014

(http://martinfowler.com/bliki/BoundedContext.html)

[25] The single responsibility principle by Robert C. Martin, November 2009

(http://bit.ly/1VDgw79)

[26] The seven deadly sins of Microservices by Daniel Bryant, January 2016

(https://www.infoq.com/presentations/7-sins-microservices)

{48}

References (III)[27] Microservice workshop – craft conference by Adrian Cockcroft, April 2015

(http://bit.ly/24os3aL)

[28] Microservice prerequisites by Martin Fowler, August 2014 (http://bit.ly/1wIjY58)

[29] Adopting Microservices at Netflix: Lessons for Team and Process Design by Tony Mauro, March 2015 (http://bit.ly/25DnkVc)

[30] The twelve factor app by Adam Wiggins (http://12factor.net/)

[31] Lehman's laws of software evolution by Meir "Manny" Lehman and László Bélády, Sep 1980 (https://en.wikipedia.org/wiki/Lehman's_laws_of_software_evolution)

[32] Conway’s law by Melvin Conway, 1967 (https://en.wikipedia.org/wiki/Conway's_law)

[33] The reactive manifesto by Jonas Bonér, Dave Farley, Roland Kuhn, and Martin Thompson, Sep 2014 (http://www.reactivemanifesto.org/)

A microservice approach for legacy modernisation

Technology

Transcript of A microservice approach for legacy modernisation