Post on 16-Apr-2017
DevOps Best Practice for Oracle SOA and BPM
Rubicon Red
The Facts
Founded in 2009, Rubicon Red’s mission has been to lead customers to success on their Oracle middleware journey.
This is the origin of our name; helping customers cross the “Rubicon” in their successful adoption of the Oracle Middleware “Red” stack.
Largest dedicated Fusion Middleware consulting and managed support services business in Australia.
Offices in Brisbane, Sydney, Melbourne, Adelaide and Redwood Shores California
Offshore development and support centre in Hyderabad India.
One of the founding members of the Red Expert Alliancehttps://www.redexpertalliance.com
Recognised global leader in Oracle Fusion MiddlewareThought Leadership. Innovation. Unrivalled Expertise.
Oracle Customer Advisory Boards• Oracle SOA• Oracle BPM• Oracle WebLogic
Rubicon RedRecognised global leader in Oracle Fusion MiddlewareThought Leadership. Innovation. Unrivalled Expertise.
Oracle A/NZ Specialised Partner of the Year 2014
Middleware
2nd year
I N N O V A T I O N A W A R D 2 0 1 4
I N N O V A T I O N A W A R D 2 0 1 1
I N N O V AT I O N A W A R D 2 0 1 0
I N N O V AT I O N A W A R D F I N A L I S T
2 0 1 5
x3
x3
The value organizations deliver through products & services is increasingly defined by the software that underpins them
Software is a Competitive Advantage
Organizations are in a Digital Race
Solution not delivered by a single systemRather a patch work of applications, each one performing a particular business function
5
Oracle Middleware provides the platform to combine these business apps, into an integrated solution
Majority of IT projects fail to deliver on-time and on-budget.
Software project waste 40%+ of resources
Gene Kim - Why Everyone Needs DevOps Now: 15 Year Study Of High Performing Technology Orgs
The Human Cost…
The Impact of Technology FailuresHigh Profile Technology Failures Increasingly Common
NYSE Says Wednesday Outage Caused by Software Update
July 10th 2015
United Airlines blames grounding of hundreds of flights on computer glitch
July 8th 2015
U.S. visa system will be offline until at least next week
June 17th 2015
• On average - a single major technology failure can cost a business as much as $US 10.8 million.• 45% of organisations experience a loss of brand equity or market share following a major
technology failureTechradar – Dec 2014
DevOpsA better way?
200x
*over 3 years
Faster lead times than their peers
60x Fewer failures
50% Higher market cap growth*
30x Deploy code more frequently
• Featured on 800 CEO Reads Top 25: What Corporate America Is Reading June, 2013
• Novel on how DevOps can be used to• Address many of the process issues that
plague IT Delivery / Operations• Align Development & Operations• Align IT with the Business• Help the Business Win
“3 Ways” of DevOps• The First Way: Systems Thinking emphasizes the performance of the entire
system as opposed to the performance of a specific silo of work or department• The Second Way is about creating and amplifying the right to left feedback loops• The Third Way is about creating a culture that fosters both continual
experimentation, taking risks and learning from failure; and understanding the repetition and practice is the prerequisite to mastery.
13
Work in Small BatchesGetting Feedback Fast – FAIL FAST• Faster Feedback• Validate assumptions
• Solve Problems Quickly• Easier to find & fix defects
• Reduce Overhead• Organizations get better at things
they do frequently
• Reduce Risk
Release
Get FeedbackImprove
Release small chunks of functionality frequently
Get regular validation on the production readiness
of your application
Incorporate feedback rapidly. Fail fast and learn
fast
Breaking the Bottlenecks In The Flow1. Environment Creation2. Code Deployment3. Test setup and run4. Overly tight architecture5. Development6. Product Management
Leading A DevOps Transformation: Lessons Learned – Gene Kim
Takes weeks/months to provision and configure Oracle Middleware Environments
Manual build & deployment of codePainful, Resource Intensive and Stressful stage of Oracle Middleware Implementations• Manual deployments
• SLOW & ERROR PRONE• Inconsistent across environments• Neither repeatable nor reliable• Require extensive documentation (often outdated)
• Which leads to • Long delivery times• Ongoing Project delays• Significant risk of Major Production Issues with
associated financial and brand damage
“What do you mean, ‘it’s not working in production?’ I TESTED IT BEFORE WE
RELEASED!”
17
Delaying integration means…Invalid assumptions discovered late in the process!
• Integration designs are full of “un-identified” assumptions• Need to integrate with actual systems
to validate• Correcting issues with core design
patterns can result in significant time delays• Frequent and early releases into SIT
critical for integration projects. Systems implemented in isolation make assumptions about other
systems with which they will integrate.
Minimise differences between environmentsVersion Control all Middleware Configuration
18
Issues caused by Configuration Drift are often the most difficult to diagnose and result in many wasted weeks/months of man effort to resolve.
Configuration Drift !
19
Test Teams standby Idle ….Waiting for deployment errors to be fixed
Test Team Idle whilst Deployment is being performed Performing smoke tests and
resolving deployment issues Turning around blocking defects etc.
• Lack of Testing
“How long would it take your organization to deploy a change that involved just one
single line of code?”Mary and Tom Poppendieck, Implementing Lean Software Development
“The test team has approx. 18% down-time, waiting for deployment issues to be resolved, every 8 days of delay is costing me 1 FTE” Programme Manager - Major Bank
Defects discovered late in deliveryCost of defect removal increases exponentially as the development lifecycle progresses.
The later defects are found and fixed, the greater the risk to the business they pose.
21
Outage In Production….Waiting for deployment errors to be fixed
60% of Production failures caused by
human error or lack of automation
80% of all IT services outages caused by unauthorized configuration changes
99% of DR failures caused by Config Drift
• Lack of Testing
“Configuration drift and unauthorized configuration changes account for nearly
80% of all IT service outages”source: Gartner Research
Technical DebtEvery time we take a “short cut” we create technical debt:• Manually configure an environment• Create code without automated tests• Manually do deployment
Technical debt is what you feel the next time you want to make a changeGene Kim – Author The Phoenix Project
23
DevOps; The First Way…Accelerate Flow to Production
Work in Small Batches Create Safety / Flow Through Automation
Automated Environments, means identical DEV, TST, PRD
Continuous Integration / Testing Automated Regression Testing Continuous Delivery / Deployment Blue / Green Deployment
Define a Reference Architecture Should “Target” Flow, i.e. reduce Technical
Debt
• Lack of Testing
“The process for releasing/deploying software MUST be repeatable and reliable”
Continuous Delivery is a Key Practice of DevOps
24
Code Build Integrate Test Release Deploy Operate
Agile Development
Continuous Integration
Continuous Delivery
Continuous Deployment
DevOps
Reduce the cost, time and risk of delivering incremental changes into Production
Automating Environment Creation
Lessons Learned on our Journey of Discovery
A declarative based solution for the install,
configuration and continuous deployment of
Oracle Fusion Middleware and
Applications.
• Still need to manually build first environment to create “Gold” Image.
• Significant amount of manual work to update clone with environment specific details.
• Any changes to “gold” environment requires new gold image to be created.
Cloning / Gold Images
100% Automated Rollout of Oracle FusionNo way to propagate changes to previously created environments.
• Significant Investment to implement and maintain
• Scripts get “big” very quickly, complex to maintain.
• Brittle, tied to a particular topology/configuration.
• Need re-write /re-factor each time Oracle make a new release.
WLST
Scripting Software Defined Environments
Automating Oracle Middleware Platform LifecycleSoftware Defined Environments
Fully automated provisioning of Oracle
Middleware Environments in
minutes, at the push of a button
Platform Provisioning
Fully automated ongoing patching and
configuration management of your Oracle Middleware
environments.
Lifecycle Management
Establish a standard process for delivering
Oracle Middleware environments
on-premise and on-cloud.
Enables Hybrid Cloud
Delivering Platform as a ServiceDrives Strong Governance & Consistency
• Activate push button automation of Environment provisioning / tear
down Configuration updates Continuous Integration
• Implement governance and version control for Platform Configuration
• Define Integration Reference Architecture
• Standardise on single integration technology stack for on-premise that can in future move to the cloud
• Define & implement consistent FMW blueprint for all environments.
Software Defined EnvironmentsDrive Strong Governance & Consistency
29
Use a consistent Platform Blueprint across ALL environments.
PRODCI PRE-PRODSIT
Environments become more production-like
Version Control and Promote Configuration Changes
DEV
Issues caused by Configuration Drift are often the most difficult to diagnose and result in many wasted weeks/months of man effort to resolve.
Platform Blueprints• Define standard topology• Ensure consistency across all
environments• EDG Templates provided out of the
box
Platform Models• Map Blueprint to Specific
Environment• Defines environment specific details
Blueprints & Models version controlled• Control promotion to new
environments• Easy to rollback to previous
environment
Fully automated provision• Provision new environments in
minutes
Demo: Install & Configure
Automating the Path to Production
SOA & BPM EnvironmentsPath to Production
Develop & Test Code
Automated Build
CIDeploy & Test
SITDeploy & Test
UATDeploy & Test
CAPACITYDeploy & Test
PRE-PRODDeploy & Test
PRODDeploy & Test
• Dev Environment• Developers build, fix & unit test code
prior to committing
• CI Environment• Automated Build & Deployment of Code• Automated Tests
• Non-Production Environments• Automated Deployment• Automated Tests• Long running / manual tests come last
• PROD
Automating the Software Delivery Pipeline
33
“How long would it take your organization to deploy a change that involved just one single line of code?”Mary and Tom Poppendieck, Implementing Lean Software Development
The set of processes, procedures and tools used to promote code from development into production.
34
Orchestrating the Delivery PipelineLeverage CI Tools such as Jenkins, Hudson, BambooOrchestrates Delivery Pipeline• Pipeline consists of various jobs
• Build, Deploy, Test, etc
• Split out by phase / environment• Dev, CI, SIT, UAT, etc
For Example• DEV Pipeline
• Triggered whenever code change committed
• CI Pipeline• Scheduled Nightly
• Non-Prod Pipelines• Triggered Manually – when business decides.
Build AutomationBuild Binaries Once… Deploy to Multiple Environments
Keep environment specific configurations OUT of the buildThis includes environment specific SCA Configuration Plans, OSB Customization Files
36
Environment Delivery PipelineFor Non-Production Environments
Re-Provision Middleware
Platform
Configure Release
Configure Middleware
PlatformDeploy Release
Smoke Test Environment Specific Tests
• Re-Provision Middleware Platform• Prevents Configuration Drift
• Configure Release• Update environment specific config, e.g.
Web Service Endpoint
• Configure Middleware Platform• Resources, e.g. Data Sources, JMS• Create & Configure Adapters
• Perform Environment Specific Tests• Longer running tests• Manual Test Performed as late in the
processes as reasonable
Application PromotionMinimise differences between environments
37
Use a consistent Platform Blueprint across ALL environments.
Re-Provision Non-Prod Platforms to eliminate
configuration drift.
PRODCI PRE-PROD
CAPACITY
UAT
SIT
Environments become more production-like
Increasing Confidence in build’s production readiness
DEV
Issues caused by Configuration Drift are often the most difficult to diagnose and result in many wasted weeks/months of man effort to resolve.
Automated DeploymentDrives Strong Governance & Consistency
Automated Builds• Compilation and packaging of code
centrally in an automated fashion
Application Blueprints• Defines which artefacts, resources
make up a release package• Build once, deploy to multiple
environments.
Application Models• Captures environment specific
details.
Fully Automated Build and Deployment
Quickly Deploy and Manage Releases across Environments
39
Build Quality In“Cease dependence on mass inspection to achieve quality. Improve the process and build quality into the product in the first place”
W. Edwards Deming
40
Different Kinds of Testing• Treat test code as production code
• Includes Test Data
• Not all tests call external systems• Make use of Mock Services
• Run smoke tests first• Perform short running tests before
long running tests• Perform Manual Testing after
Automated Tests• Continually maintain your tests
Functional Acceptance Tests
Journey Tests
ShowcasesUsability Testing
Unit testsIntegration tests
System tests
Non-Functional Acceptance Tests
Performance, scaling, …
Automated
Automated Manual / Automated
Technology Facing
Business Facing
Supp
ort p
rogr
amm
ing
Critique Project
Diagram invented by Brian Marick
Manual
41
Design Code to Simplify Testing …Use a layered reference architecture
• BPM Projects• Mock Service Layers• Use Automated Testing
• SOA Project• Same Approach• Mock IVS Layer
Service layering, use of Mock Services & Automated Testing Increase productivity and improves code quality.
Blue / Green Deployments• Allows time to validate new
release is behaving as expected before go-live• Zero down-time for go-live.• Makes rollback very simple
Blue Environment
Release N
Green Environment
Release N+1
Router
43
Continuous Deliver Tool Chain For…Oracle SOA Suite and Oracle BPM Suite
Pipeline Orchestration
Source Code Management
OSB, SCA Build Automation Software Repository
Oracle Middleware Platform
Provisioning
Configure Oracle Middleware Platform
Configure & Deploy OSB, SOA, BPM
ArtefactsAutomated Testing
Nexus
44
Benefits of DevOpsReduce Risk, Decrease Costs, Speed Up Time to Market
Reduce Risk.
Reduce risk of projects delays, provide better
visibility.Significantly reduce
risk of defects in production.
Decrease Cost.
Reduce waste and increase developer and operational
efficiency.
Provide the business with a strategic advantage in its ability to be more responsive in delivering new solutions faster, cheaper and more often.
Speed Up Time to Market
Increase agility of development and test
teams. Shorten the
development lifecycle.