Agile enterprise integration

Balancing Agile Delivery with the Enterprise Estate

Simon Greig, Executive IT Architect, IBM Global Business Services

April 2015

Agile Enterprise Integration

About the Author

Simon is an experienced IBM Executive IT Architect with 20 years experience in

designing and delivery complex projects

He has been working on complex systems integration projects since 1999 and

over the years have been immersed in SOA, ESB and more recently cloud,

mobile and agile technologies

Over his career he has delivered projects worth cumulatively about US$2Bn

His current role in IBM is as a technical leader in the Public Sector business

within IBM Global Business Services Europe

This presentation was created following many conversations with clients and

colleagues about how to integrate agile and legacy

It is one person’s point of view on the subject…!

2

Simon GreigExecutive IT ArchitectIBM Global Business ServicesEurope

Enterprise Integration Context

3

Enterprise System Context

User Experience Transaction Management

Legacy Integration Data Integration

Agile is an excellent fit to the “user tangibles” of creating the stuff that users can see and use such as the user experience

There are additional challenges with complex systems that need to be managed

– Complex interdependencies between components– External dependencies and constraints (e.g. legacy systems, 3rd party

interfaces)– Agreement of interfaces / Data format discussions and agreements– Availability of test environments– Mismatches between components– Cost of integration and non-functional testing

And common problems with running enterprise systems that need to be addressed

– Data quality issues– Software bugs– Complex middleware configuration issues– External interface data and availability problems– Ensuring data integrity at all times– How to do point in time backups without impacting live service– How to minimise patching impacts and subsequent system downtime

Assuming that new agile systems will simply avoid or eliminate complexity is dangerous because external factors will drive up complexity and introduce significant constraints

User Tangibles are just the tip of the iceberg…

4

Agile works best with this bit

Enterprise Solution

5

USER EXPERIENCE IS THE TIP OF THE ICEBERG

Integration Spectrum

7

An Integration Spectrum

8

Agile = Opportunity

• Contemporary interfaces• Relatively small number of interfaces

and components to integrate• Single or few party delivery• Consistency• Few external dependencies

Enterprise = Complexity

• Mixture of technologies• Large number of interfaces and

components to integrate• Many parties involved, some of

whom are unable to support many changes due to their own release cycles

• Complex service management environment

• Many stakeholders• Many constraints• Complex external dependencies

System Integration solutions follow an integration spectrum from simple to complex. Each type of integration is different and the further something moves away from

standalone disconnected systems the more complex and difficult it becomes

However there is room for both:

9

Ag

ile S

erv

ice

sE

nte

rpri

se

Syste

ms

Agile services:• Able to be iterated quickly• Fast delivery of changes• Doesn’t have to be right first time• Minimal or no inter sprint dependencies• Little or no external dependencies

Enterprise Systems:• Long lead time for changes• Integration testing needs to be planned in

for a long time• Potentially lots of upstream and

downstream implications to manage• Expensive if not right first time• Complex external dependencies

• Custodians of high value business assets:

• Enterprise Master Data• Complex Business Transactions• External Interfaces• History

However there is room for both:

10

Ag

ile S

erv

ice

sE

nte

rpri

se

Syste

ms

Agile services:• Able to be iterated quickly• Fast delivery of changes• Doesn’t have to be right first time• Minimal or no inter sprint dependencies• Little or no external dependencies

Enterprise Systems:• Long lead time for changes• Integration testing needs to be planned in

for a long time• Potentially lots of upstream and

downstream implications to manage• Expensive if not right first time• Complex external dependencies

• Custodians of high value business assets:

• Enterprise Master Data• Complex Business Transactions• External Interfaces• History

What do we do if agile services

need access to legacy enterprise transactions and

data?

REBUILD AS AGILE? Many legacy systems are core to the operation of

the business

The detail of how the legacy systems is not always

understood leading to the risk that a rebuild

doesn’t fully meet the business need

An incremental rebuild may not be practical if the

business still needs to operate

UNLOCK SOMEHOW? Is it possible to trigger transactions and/or access

data via an existing interface or API?

If not, how soon can a new interface be created?

The lead time is likely to be high.

How will the integration be managed?

How will we have confidence that the change to

the legacy system will be the right one as we might

not have time to make the change twice?

11

What do we do if agile services need access to legacy transactions

and data?

1 2

REBUILD AS AGILE? Many legacy systems are core to the operation of

the business

The detail of how the legacy systems is not always

understood leading to the risk that a rebuild

doesn’t fully meet the business need

An incremental rebuild may not be practical if the

business still needs to operate

UNLOCK SOMEHOW? Is it possible to trigger transactions and/or access

data via an existing interface or API?

If not, how soon can a new interface be created?

The lead time is likely to be high.

How will the integration be managed?

How will we have confidence that the change to

the legacy system will be the right one as we might

not have time to make the change twice?

12

This requires a system integration programme solution that supports both legacy and agile development approaches

What do we do if agile services need access to legacy transactions

and data?

1 2

Platform & Services

Agile Services

Enterprise Platform

• Innovation based• Some strategic• Some short lived pilots and PoCs• Limited dependencies• Potentially reduced SLAs• Built upon the ‘platform’ of APIs offered by the

enterprise• Can be ‘hardened’ to Enterprise Platform Services

• Service Managed (ITIL)• SLAs for performance and availability• Release Managed• Solid• Robust• Dependable• Hardened• Accessed via APIs

Systems Integration Framework

14

Complex Systems Integration Framework

In order to do complex systems integration combining new agile solutions with legacy and 3rd

party systems then a systems integration release framework is required

The pace of the release is determined by the slowest participant as everything [usually] has to be delivered into production together.

Each system integration release needs to follow the structure defined below:

Enterprise Architecture

Release Governance

Application Delivery

Integration Delivery

Infrastructure Delivery

Work out what it

needs to do and

how it will be done

Bring it all together

Make sure it works

Operate

16




Work out what it needs

to do and how it will be

done

Bring it all

together

Make sure it works

Op

era

te

Phase Name INCEPTION

Description The upfront planning and solution outline phase used to work out the high level scope of the

solution, the benefits, outline cost and delivery timeframe.

Output • Delivery plan

• E2E solution design

• Initial requirements backlog

Example Activities • Determine the business goal and user needs mapped to the business strategy

• Agreeing outline objectives of the delivery and high level requirements

• user stories, epics, business outcomes, traditional requirement statements,

NFRs, etc

• Creating the E2E outline solution

• Technology selection

• Objective consideration of open source and commercial software options

• Buy vs build decisions for components

• Identification of major internal and external dependencies

• Identification of major risks and initial mitigation options

• Identification of external constraints

• Selection of delivery partners

• Specialist vendors

• SMEs

• Selection of delivery method and associated tools

Enablers • Specialist system integration methods and tools

• SMEs

• Open source software

17






done

Bring it all

together

Make sure it works

Op

era

te

Phase Name DELIVERY

Description Multi-stream delivery of the release providing the application, integration and infrastructure

solutions using a variety of delivery methods and styles best suited to each situation.

Output • Design documentation

• Unit tested code (some of which will have already been integrated)

• Unit tested infrastructure

• Test scripts (automated and manual)

• Test data

• Test stubs

Example Activities • Agile delivery of components (in particular those with user facing interfaces)

• Negotiations and information sharing with with 3rd party interface suppliers

• Integrated data model development and rollout

• Application infrastructure services development

• Infrastructure build and testing of resilience, deployment scripts, security, backups, etc

• Business logic and rules discovery workshops (if not concluded in discovery)

• Configuration management and release management processes firmed up

• Construction/procurement/provisioning of dev, test and production environments

Enablers • Agile delivery

• DevOps automation

• Cloud based IaaS/PaaS provisioning

• Design patterns

18






done

Bring it all

together

Make sure it works

Op

era

te

Phase Name INTEGRATION

Description The coming together of all components from all development streams and 3rd parties in order

to create the full system for the first time. Followed by a series of shakedown tests to ensure

that all of the joints are sound and that the system wide non-functional aspects (system

resilience, security, performance) meet expectations.

This stage is essential in systems where there is a legacy or non-agile 3rd party aspect that

cannot keep up with agile delivery and continuous integration.

Output • A ready to go live system that meets the business and user needs.

Example Activities • The multi-party development streams working together to integrate with each other

• Dependencies and constraints may have limited some components to have

been delivered sooner

• E2E integration testing through repeatable test scenarios

• Both user driven and external system interface driven

• System wide resilience testing

• Typically done once at the end of a release as it is expensive and intrusive

(e.g. requires dedicated access to an environment and physical visits to the

datacentre)

• E2E performance testing

• Complex systems typically exhibit unusual or unpredicted behaviour when put

under load

• User Acceptance testing

Enablers • Performance engineering

• Test data creation automation

• Test execution automation

19






done

Bring it all

together

Make sure it works

Op

era

te

Phase Name OPERATE

Description The live running of the system by the business and on-going maintenance and enhancement

of the system.

Ongoing • Keeping the system meeting service levels

Example Activities • System monitoring

• Problem analysis

• Incident management & resolution

• Minor enhancements

• Capacity monitoring and forecasting

• Performance monitoring and forecasting

• Technology refresh

• Patching

• Security monitoring and alert management

Enablers • Service management processes

• DevOps style management where the development team do 1st, 2nd and 3rd line support

of the system rather than a service management team

Bringing it all together: Complex Systems Integration Delivery Framework

20

Inception Delivery Integration

Operation

Define Architecture Roadmap

Procurement Approach

Business Strategy

High Level Requirements

E2E Delivery Planning (Dependency mgt, risk mgt, issue mgt, integrated planning, change mgt, etc)

E2E Delivery Tech Governance (Design assurance, troubleshooting, making sure it will work at the end, etc)

Integrated Tooling Management & Maintenance (Reqs, Design models, defects, source code control, document repository, config mgt, change mgt, incident mgt, etc)

Environment Management, scheduling and deployments

Business Change Planning

Service Management Prep

E2E High Level Design

Sprint Zero

S/W Selection

H/W & Hosting

Selection

Ready for full system

integration gate

Supplier 1

Plan Buil

d Test

FixDeploy

Waterfall Development

Legacy Change

Agile Sprints

Supplier 2

Supplier 3

Opera

tional

Accepta

nce T

est

Multi-P

art

y C

om

ponent

Inte

gra

tion T

est

Multi-P

art

y E

2E

Functional S

yste

m

Test

Non-Functional System Test(Performance, Security and Resilience)

E2E System

Integration Test

User Acceptance

Test

System Monitoring

E2E Service Management

Incident Management

AMS

Infrastructure Management

Capacity Planning

On-going Projects (Continuous Improvement & Business Change)

Inception Delivery Integration Operation

Enterprise Architecture and Governance

e.g. infrastructure

Solution Outline

Key Enablers

21

Agile Delivery

22

Point of View Agile needs to be partnered with engineering in order to create solutions with solid non-functional basis.

Systems engineering is needed in order to meet service levels and also integrate with the legacy (non-

agile) estate.

Agile may not be suitable for all elements of system delivery. Such as:

• Integration interfaces with 3rd party systems

• Data processing based function

• Deliveries with a number of dependencies

Pros • Ability to demonstrate progress of a system to the stakeholders is very valuable

• More focus on the output of working code than process and documentation

Cons • Dependencies need to be resolved before each sprint – a large number of dependencies may lead to

blocked sprints

• Risks of rework and churn need to be mitigated to avoid major re-writes

• Requires input and time investment from the business

• Building incrementally without a solid infrastructure platform puts the overall system stability at risk

• The commercial and risk ownership implications of agile delivery in a complex multi-supplier

environment are yet to be understood

Open Source Software

23

Point of View Open source software is an attractive proposition due to the low acquisition cost. However software selection

needs to objectively select the right solution based on a number of factors:

• Acquisition cost

• Maintenance cost / Enterprise support costs

• Maintainability

• Longevity of the platform

• Marketplace skills

• Robustness

• Performance / Relative hardware needs

• Features/Innovation

Pros • Low start up costs

• Typically faster start up costs

• Many products are very mature, stable and enterprise class

Cons • A lot of hype and myths surround open source software. e.g. if you don’t like the function, you can change it.

Changes cost time and money and usually void enterprise support agreements

• Variance between packages on security, stability and performance (although this is improving quickly)

• Some minor legal risks around Intellectual Property

• Variable skills and support depending on the product – which can change over time as fashions change

Cloud

24

Point of View Going forward, solutions should be designed to work on commodity platforms such as Linux on x86.

Cloud offers an opportunity to reduce the time to delivery of infrastructure to projects but it is not necessarily a cost

saver. Cost saving may be possible if systems are hosted on a commodity public cloud such as IBM SoftLayer,

however data security will limit the extent to which public cloud offerings can be utilised. Many cloud suppliers,

including SoftLayer, offer private clouds which are more attractive to an enterprise client.

Commercial private clouds have an associated hardware and deployment cost that needs to be picked up by the

customer. Additionally there may be some short term scalability constraints if more hardware needs to be

deployed quickly.

Pros • Rapid provisioning of new nodes if the capacity exists

• Inherent flexibility offered by hardware virtualisation technology

• Some suppliers may offset the upfront charge with a higher service charge which may be helpful for budgeting

and a business case

Cons • There remains concerns with enterprise data security – more a perception than a reality

• Cloud may not necessarily provide a cost saving as enterprises [currently] mostly prefer the majority of systems

hosted on a private cloud with dedicated non-shared hardware and storage

• Modifications such as the addition of security components and specific network separation zones within an

environment may limit some of the cloud benefits such as dynamic provisioning

Conclusion

25

ConclusionAgile can offer significant benefits to the

business, in particular in areas where the user

requirements are not fully understood

External dependencies can heavily constrain

agile delivery

Solution delivery will always go at the pace of the

slowest participant regardless of how fast some

participants can be

‘Full stack’ enterprise system development and

integration needs a hybrid approach between

agile and traditional in order to support a

combination of agile and traditional delivery

methods

26

Agile enterprise integration

Technology

Transcript of Agile enterprise integration