DevOps for the Mainframe Leveraging Continuous Integration, the cloud, and beyond to deliver z/OS applications Session SC-2402 David Myers, Product Manager, Cross-Brand DevOps myersda@us.ibm.com @Dave_Does Rosalind Radcliffe, Distinguished Engineer, Enterprise Modernization Solution Architect rradclif@us.ibm.com

Please note

IBM’s statements regarding its plans, directions, and intent are subject to change or withdrawal without notice at IBM’s sole discretion. Information regarding potential future products is intended to outline our general product direction and it should not be relied on in making a purchasing decision. The information mentioned regarding potential future products is not a commitment, promise, or legal obligation to deliver any material, code or functionality. Information about potential future products may not be incorporated into any contract. The development, release, and timing of any future features or functionality described for our products remains at our sole discretion.

Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon many factors, including considerations such as the amount of multiprogramming in the user’s job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve results similar to those stated here.

3

Social, Local, Mobile Smart Infrastructure Analytics

ERP  Legacy  DB  

CRM   HR  

Manage workloads and maintain security

Rapid innovation in the cloud

User experience and mobile management

New era systems integrate existing operational systems with rapid delivery of new client-facing apps

Evolving customer and market expectations

Capabilities and User Experience Today Emerging

Primary Workload Types Systems of Record Transactional

Systems of Engagement (+ Record) Big Data, Analytics, Mobile/Social Channels

Time to Value Planned Opportunistic

Delivery Model Planned Incremental (DevOps)

Development and Operations Team Sizes 100s and Costly 10s with built-in DevOps automation

Release Frequency Months to Years Hours to Days, based on business opportunity

Integration Frequency Weeks Continuous

Service Sourcing Develop Consume and Assemble (Public and Private)

Operational Model Systems Management Built in to application, Recovery Oriented Computing, Continuous Availability

Infrastructure Deployment Days Minutes

Risk Profile Big-Bang (High Risk) Incremental

DevOps is… A set of principles and values that facilitate collaboration across disciplines to…

1. Enable rapid evolution of deployed business services

2. Reduce risk, decrease cost, and improve quality across the portfolio

DevOps Principles

§  Collaborate across disciplines

§  Develop and test against a production-like system

§  Deploy frequently using repeatable and reliable processes

§  Continuously monitor and validate operational quality characteristics

§  Amplify feedback loops

People

Process

Tools

Solution: A Continuous Delivery Pipeline

Development Testing Staging Production Business Owners Customers

•  Validate on more production-like conditions earlier

•  Automate hand-offs/promotions to increase velocity through the different stages

•  Standardization on processes and assets between Dev and Ops

•  Automated monitoring and dashboarding of quality and performance against service level agreements at multiple stages

Ensure applications are production-ready throughout the lifecycle and can be released at any time while minimizing rollback due to quality issues

Mainframe Delivery Pains…

§ Multiple teams working across restricted dev and test capacity lead to conflict, delays, or bad test results in shared environments

§ Complex and manual management and configuration tasks result in errors and delays

§ Too much bad code going into test and production causes crit sits and emergency fixes

§ Bottlenecks due to inefficient communications between disparate platforms and teams (Dev/Test - System Programmers; mobile – distributed-mainframe)

Go on Offense

Play Defense

…solutions from IBM

§ Provide cheap, isolated, development and test environments for project teams

– Rational Development and Test Environment

– Rational Test Virtualization Server – SmartCloud Provisioning – Cloud Ready for Linux on System z

§ Automate consistent build, configure, and deploy processes across all stages

– Rational Team Concert – UrbanCode uDeploy – SmartCloud Orchestrator

§ Enforce base quality standards automatically prior to promotion

– Rational Test Workbench – Rational Quality Manager – SmartCloud Application Monitoring – Omegamon

§  Improve communication and collaboration with cross-platform release planning

–  IBM Collaborative Lifecycle Management

– Smart Cloud Control Desk – UrbanCode uRelease

Test Test

Dev Dev

1

DevOps for the common mainframe delivery cycle

Dev Test Staging Prod

Development" Test" System Programmer"

3

2 2

4

10 10

Test LPAR

z/OS

…

Typical z/OS Testing Architecture Organized by project team, vertically scaled, sharing resources, limited automation

Project Team [April Maintain]

Project Team [Prototype SOA]

Project Team [June New Func]

Project Team [Dec Sys Upgrade]

Test Data

App

App

App

Problems Encountered

1. Shared resources combined with overlapping schedules can elicit conflicts, impede innovation and slow code delivery

2. Coordination of environmental changes and releases cause bottlenecks, delays and additional overhead

3. Shared test data is difficult to manage and can lead to over testing or incorrect test results

The ultimate in modern application development for System z

§  Liberate developers to rapidly prototype new applications § Develop and test System z applications anywhere, anytime! §  Free up mainframe development MIPS for production capacity §  Eliminate costly delays by reducing dependencies on operations staff Note: This Program is licensed only for development and test of applications that run on IBM z/OS. The Program may not be used to run production workloads of any kind, nor more

robust development workloads including without limitation production module builds, pre-production testing, stress testing, or performance testing.

RDz & ISPF user

ISPF user RDz user

RDz user RDz user

COBOL, PL/I, C++, Java, EGL, Batch, Assembler, Debug Tool

x86 PC running Linux

RDz user

IMS

z/OS

WAS

DB2

MQ CICS

Rational Development and Test Environment for System z

Simulate subsystem dependencies –  Test impact of latency –  Test application response to unresponsive

services Create a stable test environment –  Simulate “public” services –  Reduce capacity requirements on

infrastructure –  Improve security by reducing access from

external services

Improve testing with Green Hat service simulation Decouple subsystem teams using Green Hat

Databases Mainframe applications

Third-party Services

Rational Test Virtualization Server

Application changes being tested

virtualized services

Testing with dependency virtualization

Phase 1 Phase 2 Phase 3

App

Test Case

Virtual Services

3rd Party Call

IMS Data Access

CICS Commarea Call

App

Test Case

Virtual Services

3rd Party Call

IMS Data Access

CICS Commarea Call

App

Test Case

Virtual Services

3rd Party Call

IMS Data Access

CICS Commarea Call

Controlled large system testing by isolating components under test •  Easier problem determination •  Lower test environment capacity requirements •  Improved component quality

Provision new dev / test systems in minutes •  Fast VM provisioning for near-instant

deployment of 100s of virtual machines and scale based on business needs

•  Rich set of web interfaces into the cloud that can be driven by a user or scripts for complete automation

•  Reliable, non-stop cloud capable of automatically tolerating and recovering from software and hardware failures

•  Near-zero downtime due to faults, hypervisor/management software upgrades or addition/removal of hardware

Requested  VMs  will  be  up    and  running  under  a    

minute  using  standard  HW  

Function Test

Performance Test

§  Fast, dependable, automatic feedback speeds time to market §  Lower cost of application testing using off-mainframe z/OS test environment §  Enables confidence by automatically tracking and promoting code health

Continuous Integration for z

Developer Frequent, rapid feedback earlier to improve quality

Rapid Feedback

Code Check In + Build + Deploy +

Run Automated Tests

Change Request

Automated, continuous testing with isolated development and test environment

Reusable integration tests simplify testing across layers

Promote To QA

Higher quality going into QA

Reduced delivery time, end-to-end visibility of test activities, safer and faster upgrades (V2V) Fail fast and drive better quality downstream

System z Continuous Integration for System z Scenario

16

SCM/CI Server

(Rational Team Concert)

Test Environment (RD&T or LPAR)

Test Execution Manager

(Rational Quality Manager)

Test Automation

(Rational Test Workbench) 1.  Check-in code

2.  Build code and zUnit tests

3.  Deploy build results and test data to Test Environment

4.  Execute zUnit Tests

<Deploys App+zUnit Tests> <D

rives>

5.  Kick-off Automated Test Plan 6.  Run automated

interface tests against Test Environment

7.  Mark execution records Pass/Fail in Test Execution Manager

<Starts Tests>

8.  Report test results in dashboard/build results/defect records in CI server.

<Results>

§ Each tool came with its own •  UI - Web and desktop

presentations of views and tasks

•  Logic – Workflow, process, search, query, scale, security and collaboration

•  Storage – Availability, traceability

•  Privacy, backup/archive § Resulting in...

•  Brittle integrations •  Silos everywhere •  High cost to maintain and

administer •  Proprietary API's

Tool E SCM

UI

LOGIC

DB

UI

LOGIC

DB

UI

LOGIC

DB

UI

LOGIC

DB

UI

LOGIC

DB

Tool B Release Planning

Tool D Incident

Management

Tool C Reporting

Tool A Dev work Items

Separate tools makes collaboration inefficient or difficult...

Tie development release to operational release Rational Team Concert Release Plan and work items

SmartCloud Control Desk

19 19

Delivery Organized for Flexibility and Speed Organized by application team, horizontally sliced, dedicated resources, highly automated

Problems Encountered 1.  Shared resources

combined with overlapping schedules can elicit conflicts, impede innovation and slow code delivery

2.  Coordination of environmental changes and releases cause bottlenecks, delays and additional overhead

3.  Shared test data is difficult to manage and can lead to over testing or incorrect test results

4.  Provisioning, managing, and synchronizing project test environments including data

QA

/Inte

grat

ion

Test

LPA

R

z/OS LPAR

…

Project Team [April Maintain]

Project Team [Prototype SOA]

Project Team [June New Func]

Project Team [Dec Sys Upgrade]

Integr. Test Data

LPAR

Data

RD&T

Data

RD&T

Data

LPAR

Data

What are we working on in the lab…

Go on Offense

Play Defense

…solutions from IBM

§ Provide cheap, isolated, development and test environments for project teams

– Rational Development and Test Environment

– Rational Test Virtualization Server – SmartCloud Provisioning – Cloud Ready for Linux on System z

§ Automate consistent build, provision, configure, and deploy processes across all stages

– Rational Team Concert – SmartCloud Continuous Delivery – SmartCloud Orchestrator*

§ Enforce base quality standards automatically prior to promotion

– Rational Test Workbench – Rational Quality Manager – SmartCloud Application Monitoring – Omegamon

§  Improve communication and collaboration with cross-platform release planning

–  IBM Collaborative Lifecycle Management

– Smart Cloud Control Desk

Cloud-style deploy of zLinux and z/OS dev and test patterns on RD&T or

zHardware

Infrastructure as Code

Source Control Management

Source Artifacts

.cbl

.asm

.cpy

CLIST .rexx

Provisioning using Cloud

#!/usr/bin/env ruby

class DevopsDeployer def initialize(build_url, build_id) @log = Logger.new(LOG_FILE) @log.level = LOG_LEVEL @iaas_gateway = IaasGateway.new(HsltProvider.new(),LOG_FILE, LOG_LEVEL) @server_instance = nil rtc_build_system_provider = RtcBuildSystemProvider.new(RTC_REPOSITORY_URL, RTC_USER_ID, RTC_PASSWORD_FILE) @build = rtc_build_system_provider.resolve_build(build_url, ENV['buildResultUUID'], build_id) @build_system_gateway = BuildSystemGateway.new(rtc_build_system_provider, LOG_FILE, LOG_LEVEL) end def add_build_stamp template_file = WEB_APP_ROOT + "/app/templates/pages/page.html" @log.info "Adding build ID stamp #{@build.id} to \#{template_file}" # Read in the file's contents as a string, replace # the build_id, then overwrite the original contents # of the file text = File.read(template_file) new_text = text.gsub(/\{\{ build_id \}\}/,"<a href=\"#{@build.uri}\">#{@build.id}</a>") File.open(template_file, "w") { |file| file.puts new_text } end

# ...

Delivery Pipeline Fundamentals

Deploy

Environment Running System

Source Control Management

Source Artifacts

.cbl

.asm

.cpy

CLIST .rexx

Deployable Artifacts

Build, Package, & Unit Test Application Binaries & Platform Configuration

Library

Consolidate, standardize, simplify deployment

COBOL

Java

Copybooks

WSDL

DB2 DDL IMS DBD/

PSB

HTML

MQ Queue

Def

DB2 Plans

Procs

JCL

Deploy

Environment Running System Deployable Artifacts

Build, Package, & Unit Test Application Binaries & Platform Configuration

Library

Enterprise Applications have thousands of disparate parts •  Currently maintained in separate systems, if maintained at all •  Limited linkage between systems for application and configuration

dependencies •  Missing assets and information is rampant DevOps force linkages, automation, and standardized packaging…

Source Management

Create, secure, and deploy test data

2TB

-Subset

Production or Production Clone

-Compare -Refresh

Create “right-size” production-like environments

for application testing De-identify sensitive information with realistic but fictional data for testing & development purposes

De-identify sensitive information with realistic but fictional data for testing & development purposes

z/OS

UT RD&T

Automated test data deployment for each

build and test

1 GB

z/OS

UT LPAR

1 GB Masked Test Data

Automated Build & Deploy Test Data

Automated Testing

Optim RTC

Test Data

Standardize z/OS region configurations to ease deployment

•  Standard topologies exist today (production LPARs)

•  Standardized regions which can be repeatedly deployed are rare

•  Standardized/Automated deployment of COMPLETE system is spotty

To adopt the cloud for testing… Standardize and automate provisioning of everything …

Standard Middleware Pattern

CICS 5.1 IMS 11.1

DB2 10.1 Z/OS 1.12

Configuration

Data

Application

RTC

/CIz

Ava

ilabl

e C

apab

ilitie

s

Standardize z/OS region configurations to ease deployment

Standard Middleware Pattern

CICS 5.1 IMS 11.1

DB2 10.1 Z/OS 1.12

Configuration

Data

Application

Pattern/Platform

Application load modules

Test Data

JCL and REXX

Evolving to deploying regions … cloud-style

“Cloud” RD&T Or LPAR

RD&T Or LPAR

RD&T Or LPAR

RD&T Or LPAR

SmartCloud Orchestrator +

<Deployment Automation for Mainframe project>

Region/Middleware Patterns

CICS+DB2

CICS+ IMS DB

Batch

29

Daily Apple TV giveaway §  Complete your session surveys online each day at a conference kiosk or on

your Innovate 2013 Portal!

§  Each day that you complete all of that day’s session surveys, your name will be entered to win the daily Apple TV!

§  On Wednesday be sure to complete your full conference evaluation to receive your free conference t-shirt!

30

Acknowledgements and disclaimers

© Copyright IBM Corporation 2013. All rights reserved. –  U.S. Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp.

IBM, the IBM logo, ibm.com, Rational, the Rational logo, Telelogic, the Telelogic logo, Green Hat, the Green Hat logo, and other IBM products and services are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both. If these and other IBM trademarked terms are marked on their first occurrence in this information with a trademark symbol (® or ™), these symbols indicate U.S. registered or common law trademarks owned by IBM at the time this information was published. Such trademarks may also be registered or common law trademarks in other countries. A current list of IBM trademarks is available on the Web at “Copyright and trademark information” at www.ibm.com/legal/copytrade.shtml If you have mentioned trademarks that are not from IBM, please update and add the following lines: [Insert any special third-party trademark names/attributions here] Other company, product, or service names may be trademarks or service marks of others.

Availability: References in this presentation to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates.

The workshops, sessions and materials have been prepared by IBM or the session speakers and reflect their own views. They are provided for informational purposes only, and are neither intended to, nor shall have the effect of being, legal or other guidance or advice to any participant. While efforts were made to verify the completeness and accuracy of the information contained in this presentation, it is provided AS-IS without warranty of any kind, express or implied. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, this presentation or any other materials. Nothing contained in this presentation is intended to, nor shall have the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use of IBM software.

All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics may vary by customer. Nothing contained in these materials is intended to, nor shall have the effect of, stating or implying that any activities undertaken by you will result in any specific sales, revenue growth or other results.

© Copyright IBM Corporation 2012. All rights reserved. The information contained in these materials is provided for informational purposes only, and is provided AS IS without warranty of any kind, express or implied. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, these materials. Nothing contained in these materials is intended to, nor shall have the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use of IBM software. References in these materials to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. Product release dates and/or capabilities referenced in these materials may change at any time at IBM’s sole discretion based on market opportunities or other factors, and are not intended to be a commitment to future product or feature availability in any way. IBM, the IBM logo, Rational, the Rational logo, Telelogic, the Telelogic logo, and other IBM products and services are trademarks of the International Business Machines Corporation, in the United States, other countries or both. Other company, product, or service names may be trademarks or service marks of others.

www.ibm.com/software/rational