IBM InterConnect 2016 Greg Hodgkinson 2238 Thriving DevOps at BMI (Prolifics)

CONNECT WITH US:IT: Customized to Your Advantage

Thriving DevOps at Broadcast Music Inc. DOD-2238Growing from nothing to world class.

Public | Copyright © 2014 Prolifics

Greg Hodgkinson – Prolifics, Global Director of Lifecycle Tools and

Methodology

Jim Harvey - Broadcast Music Inc., Sr. Dir. of Quality Assurance and Release

Management

http://www.youtube.com/prolificstv

http://www.linkedin.com/company/prolifics

http://www.facebook.com/prolificstech

http://twitter.com/prolifics/

CONNECT WITH US:

What attendees will learn What has made DevOps thrive at Broadcast

Music Inc.? What are the conditions and driving forces

behind a thriving DevOps practice? What does this thriving practice look like? Who is involved, what tools are they using, how

have they benefitting from these tools? What are they achieving? How is DevOps making a difference to the

business?

2






Introductions






CONNECT WITH US:

Who Are We Making Happy?

4

Everyone! (that listens to music)





CONNECT WITH US:

Speaker Introduction

5

JIM HARVEYSenior Director of Quality Assurance and Release ManagementBroadcast Music Inc.Jim Harvey is the Senior Director of Quality Assurance and Release Management for Broadcast Music Inc. (BMI). He has over 16 years’ experience in Software Development. Jim is currently evolving the Software Development Life Cycle at BMI as it relates to Requirements Management, Software Testing and Change Control management.

GREG HODGKINSONDirector of Lifecycle Tools and MethodologyProlificsGreg has 20 years of experience in assisting customers achieve success by optimizing their teams. Optimizations include adopting agile practices, DevOps tools and practices, and SOA techniques.

He is very much a practitioner with responsibility for successful delivery at a number of leading companies. He presents on DevOps and agile SOA approaches at major IBM events, has co-authored a Redbook on SOA solutions, and contributes to DeveloperWorks.





CONNECT WITH US:

Broadcast Music Inc. at a Glance

Broadcast Music, Inc. - 1939 Performing Rights Organization

(PRO) - Pay public performance royalties

Operate on a non-profit-making basis

7 locations: Nashville, New York, Los Angeles, Atlanta, Miami, Puerto Rico, London

600 employees, 7.5 million works, over 500,000 songwriters and composers

6





CONNECT WITH US:

The Performers

7

Israel Kamakawiwoʻole

Louis Armstrong

Judy Garland





CONNECT WITH US:

The Writers

8

“What a wonderful world”George David

WeissBob Thiele

“Somewhere over the rainbow”1939

1967





CONNECT WITH US:

2016: The Digital Music Ecosystem

Over 500 digital music services worldwide

Offer consumers the opportunity to legally access up to 26 million songs

Challenge: Stay relevant and competitive?

9





CONNECT WITH US:

Prolifics at a Glance

10

5-Years CompoundAnnual Growth Rate

19%

EmployeesWorldwide1,500

Global PresenceUnited States, United Kingdom, Germany, India

20+Technology

Accelerators

550+Technical

Certifications

Over 10 Technology and Solutions Awards since 2009 including

Business Agility, Customer Integration and Digital Experience, the first ever Beacon Laureate for Business Agility

Over 160 global customersare currently Fortune

1000 companies

Best-in-class architects andspecialty experts:

BPM, Integration, Digital Experience, Security, Testing, Business Analytics and Enterprise Content Management

End-to-End Project Expertise

Rate of Repeat Engagements*

91%

Years inBusiness

35+Offices

14 Awards

Technology Expertise

Fortune 1000

*based on % revenue Source – December 2013 internal revenue metrics






Challenge and Success

2/18/16 11





CONNECT WITH US:

Why the need for DevOps at BMI?

Rapidly innovating music industry

Means BMI needs rapid innovation themselves

New music services => new integrations

New offerings = > new systems

Explosion of artists => increased need for scale

===========================Net: Need to deliver a lot of software innovation, quickly.

12





CONNECT WITH US:

The BMI Transformation

Over a period of 4-5 years BMI have fundamentally changed their ability to deliver software innovation.

13

New Approaches New Runtime Platforms

New Development Tools

Agile

BPM

SOA

DevOps

IBM BPMIBM WebSphere Portal

IBM ODMWebSphere DataPower

IBM Integration BusWebSphere ESB

IBM CastIronIBM DataStage

OracleIBM WSRR

Focal Point

DOORS NG

Rational Software Arch.

Rational Team Concert

Rational Quality Mgr.

IBM UrbanCode Deploy





CONNECT WITH US:

How Did They Change (Successfully) So Fast?

They key to this rapid transformation: Good processes and tools!

14

To be successful with these…

…required that they be successful with these.

In a nutshell: all the things that helped them be efficient! DevOps especially relevant.





CONNECT WITH US:

Adding Agile and RTC

First challenge: Herding the cats!

Scrum

Structure for teams Focus of sprints Course correction Simple – get going quickly!

RTC

Progress visible Keeps everyone on same

page Easy configuration

management Opportunity for integrated

builds

15





CONNECT WITH US:

Toe in the Water: Continuous Integration

Opportunity: Integrated builds for quality feedback loop

Continuous integration of WESB apps

Java apps easy, start tackling newly adopted WESB

Developed build machine plugin to automate builds of WESB ears

Used in CI cycles: immediate feedback for devs (learning new tech)

Little effort -> lots of value

16





CONNECT WITH US:

Obvious Next Step: Automated Deploys

Challenge: Delays to get changes into test

Continuous deployment of WESB apps

Added automated deployment of WESB ears to plugin

Either on demand, or as soon as ready

No more waiting; testers testing earlier

Reduced delays mean more testing done; quality increases

17





CONNECT WITH US:

Expand Across Enterprise Architecture

Opportunity: Success breeds success

Full-stack build and deploy

Made use of Prolifics investment

Common framework Leveraged common steps Already had plugins for IIB

and Portal Expanded to all app types Integrated stack no longer

held back by slowest individual release pipeline

18





CONNECT WITH US:

Added Database Deployments

Challenge: But what about the data? New plugin for automating deploy of schema updates

Implemented using same framework and standards as apps for simplicity

DBAs place scripts into source control

Automation picks up and upgrades schema

Works across all environments

19





CONNECT WITH US:

Added Publishing of Services

Challenge: And what about the service registry?

New plugin to publish to WSRR

Again, same framework and standards leveraged for simplicity

Uploads WSDLs, XSDs, and other file changes

Updates version and other metadata

Consistent across all environments

One less thing to remember to do

20

*Bad luck Brian





CONNECT WITH US:

Ramping up on Automated Deployment

Challenge: Push button deployments – too many buttons!

Purpose-built deployment engine: UrbanCode Deploy

Wrap existing deployment automation as UCD plugins - quickly and painless move

Orchestration Cross-tier deployments

simple Easier to elaborate deploys

of individual components too

Snapshots of complex apps simple

Conditional gates on environments

21





CONNECT WITH US:

Integrated Automated Testing

Opportunity: Include existing automated tests Plugin to wrap testing tool(s)

Tests moved into source control Builds pull out tests and execute

them automatically Can then

Trigger when you need, Schedule, Or include as part of

build/deploy/test process Lots more value from test scripts

Regression testing Smoke testing

22





CONNECT WITH US:

Dockerising Build Agents

Challenge: Build infrastructure inflexible time sink

Created Docker images of all build agent configurations

Install build agent software Install automation Install supporting software

– in some cases this is considerable

Easier to isolate build agents

Easier to spin up new agents

Easier to refactor infrastructure

23





CONNECT WITH US:

Next: Automated Provisioning of Environments

Challenge: Full set of test environments takes up a lot of tin

Create Docker images for all test environment configurations

Saves time – also allows new behaviors

Spin up new environments on demand

Stand-up/tear-down as part of test automation

Much better usage of tin – easier to share across projects and teams

24





CONNECT WITH US:

Beyond That: Into the Future

Opportunity: What else can we do with this automation?

Think big for the future!

Increasingly hands-free automation

Self scaling environments Detect shortage Provision new environment Lay on top app and config

Self healing environments Detect issue Provision different version Apply known fix

25






Our Approach

2/18/16 26





CONNECT WITH US:

Practice Agility!

The key to getting started and keeping going.

27

Some might… We did…

Start by envisioning a big program of work.

Started simple – Then build on initial success, now into 4th year of continued improvement – best multi-year projects are based on a small success.

Static delivery plan

Good principles of agility – Do something, course correct, adapt to changing needs, focus on strategic plan but still keep agility to respond to immediate need

Fix-it-all in first 6 monthsPick an area to improve - Don’t “boil the ocean”. We started by just fixing Continuous Integration.





CONNECT WITH US:

Focus on Practices and Benefits!

Focus on what is important.

28


Just focus on putting in place new tools and technologies.

Consider the practices – We thought through what it is we’re trying to do and chose the right tools and technologies to support.

Get caught up in “what is hot in DevOps”.

Focus on benefit – As we intended on keeping resource/budget small, we had to make sure that everything we did gave “best bang for buck”.

Practices & Benefits





CONNECT WITH US:

Cross Boundaries!

Important to keep eyes open to wider opportunities.

29


Stick with the areas they are comfortable with e.g. Java.

Cover it all! - Thrived on figuring out how to automate assembly and deploy of the trickier areas e.g. database, repository-based dev, service registries. Ensure that all parts of the app are included.

Just focus on developers and operations teams.

Consider all involved teams – Any role that is involved in releasing apps, including DBAs, testers.





CONNECT WITH US:

Keep Efficient!

DevOps sounds like it needs a big team.

30


Go it alone, or outsource it all.

Best of both worlds – Prolifics involvement allows BMI to focus on writing and testing apps, while BMI release engineering team has still had direct control of app releases.

Create a big team to solve a big problem.

Small as possible - Achieved almost unbelievable levels of efficiency with team of just 2.5 FTEs (and one of those is a manager). Low cost, disproportionate benefits.

Either do no training, or overcommit to training.

Pragmatic value-driven training - Train by example, support with JIT workshops and Wiki-based documentation.





CONNECT WITH US:

Leverage Expertise!

Working with a partner has allowed BMI to make much more progress.

31


“Not invented here!” – lets build our own automation.

Reuse! - Leveraged automation from the Prolifics DevOps Labs. Huge difference in how much has been achieved as well as level of quality.

Give us 6 months and we’ll have our own automation development capability.

Engage specialists - Prolifics DevOps labs efficiencies have allowed us to be responsive to BMIs needs – we’ve created the engine.






What Does Success Look Like?






CONNECT WITH US:

Planning and Prioritizing Work

Apply agility to planning

Start from backlog Release plans track what

we’re currently working on Thrice weekly touchpoints Focus on prioritizing:

Hot current issues Hot opportunities for

improvement Strategic improvements

Constantly maximizing value!

33





CONNECT WITH US:

Prototyping New Automation

Uncover risk, uncover opportunity Seek to understand before we

automate What are artifacts? Where do

artifacts live? What are APIs/tools?

Approach Do it manually. Then automate

manually – command-line. Then script automation

Examples! Involve experts and teams -

Demo and get feedback

34

DeveloperWorks Knowledge Center

Samples Try it all out!





CONNECT WITH US:

Implement Automation in RTC

Great automation development support Configuration management of

automation and example apps Inject automation (with changes)

into example build workspace Build engine to do test runs – attach

logs, perform activity logging Link up development work to plan

items in release plan Record of progress, issues, solutions JBE simple automation engine – use

Ant as standard

35





CONNECT WITH US:

Architect for Ease of Maintenance and Extensibility

Well factored plugin model Ant-based core Standard assembly and deploy steps

atomic steps Additional atomic steps e.g. stop app,

undeploy app, set default app, etc. Composite steps are mini-processes Reusable utilities in common Package up libs where appropriate Helper scripts where appropriate Interface with APIs and tools

appropriate to target platform Version to handle upgrades of target

platforms

36

APIs and Tools

Target Platform

Core Automation Plugin

build.xml

libs

scripts

Atomic•assembleApp•deployApp•etc.

Composite•assembleAndDeployApp

common.xml





CONNECT WITH US:

Wrap Plugins for UrbanCode Deploy

Port deploy automations to enterprise deployment platform Simple set of wrapper files

around proven automation Benefits

Keep orchestration (deploy processes) flexible

One button deploys for multi-component, multi-tier apps

Snapshot whole app and move through environments

Gates, approvals, etc.

37

UCD Plugin

Core Automation Plugin

build.xml

libs

scripts

Atomic•assembleApp•deployApp•etc.

Composite•assembleAndDeployApp

common.xml

plugin.xml

wrapper scripts.groovyAtomic• deployApp• etc.

component template.json

• component processes





CONNECT WITH US:

Continuous Integration and Provisioning of Plugins

Practice what we preach – optimized delivery of plugins Automate plugin assemble and

deploy Core plugins and UCD plugins Reduce time-to-test

Run in a continuous integration cycle to detect issues early

Combine with running of automation steps to test working

Feedback loop Snapshot plugins – promote to PROD

38

DEV/TEST PROD

DEV/TESTDeploy

PROD Deploy

DEV/TESTDeploy

PROD Deploy

JBE

UCD Serve

r

JBE

JBE

JBE

UCD Serve

r





CONNECT WITH US:

Containerize build agents Creating a flexible DevOps

infrastructure with Docker Build agent host installs take

time Docker gives simple way of

Codifying install process Shipping around “golden”

images Starting/stopping as many

instances (containers) as you wish

Keeping build machine stacks isolated

Provisioning new machines “instantly”

Prevent build machines from clogging up with installs and temporary files

39

Before:

After:






How Has This Helped?






CONNECT WITH US:

What has been achieved?

Automated assemble and deploy across 10 technology platforms.

Accelerated delivery pipelines for 11+ application development streams.

Oversight of deployments for all involved projects.

Core Build/Release team of 2.5 FTEs

Big Difference, Little Budget!

41





CONNECT WITH US:

What Benefits are Being Seen?

Able to reinvest upwards of 10 resources into more valuable activities.

Delivery is sooner. Testers are able to test more. Quality of code delivered is

higher. Visibility of delivery has

increased. Outages due to deployment

error close to eliminated.

42





CONNECT WITH US:

Impact on the Business, Customers, and Us?

Able to keep up with exploding number of music delivery channels.

Able to deliver more features that make it easier to do business.

Giving a better and more reliable quality of service to customers.

Collected royalties result in a healthier music industry, meaning better music for all of us!

43





CONNECT WITH US:

Thank YouYour Feedback is Important!

Access the InterConnect 2016 Conference Attendee Portal to complete your session surveys from your

smartphone, laptop or conference kiosk.





IBM InterConnect 2016 Greg Hodgkinson 2238 Thriving DevOps at BMI (Prolifics)

Software

Transcript of IBM InterConnect 2016 Greg Hodgkinson 2238 Thriving DevOps at BMI (Prolifics)