OOW16 - Oracle E-Business Suite 12 Upgrade Experience for a 14 TB Oracle E-Business Suite Instance...
-
Upload
vasuballa -
Category
Technology
-
view
96 -
download
3
Transcript of OOW16 - Oracle E-Business Suite 12 Upgrade Experience for a 14 TB Oracle E-Business Suite Instance...
19/15/2016
GE Healthcare R12 Upgrade Overview
- Umesh Date( Tech. Program Mgr.- R12 Upgrade)
GE HE ALTHCARE TODAY• $18 billion: global business unit of GE
• 46,000: number of employees worldwide
• $1 billion+ per year investment in R&D
• 35,800 patents filed by GE since 2000
• Core strengths in bio-sciences, medical imaging
& information technologies
• Headquartered in Chicago, USA
39/15/2016
BROAD SOLUTIONS FOR HE ALTHCARE
Diagnostic and Interventional Imaging
Patient Monitoring
Maternal and Infant Care
Anesthesia and Respiratory Care
Diagnostic Cardiology
Healthcare IT
Life Sciences
Healthcare Consulting
Global Services
CUSTOMERS
O U R K E Y C U S TO M E R S I N C LU D E :
• Healthcare systems, hospitals, and clinics
– Physicians
– Clinicians
– Nurses
– Executives and administrators
– IT managers
– Biomedical engineers
– Construction managers
• Government officials
• Pharmaceutical firms
• Genetics researchers
• Bio-science researchers
59/15/2016
GE Healthcare ERP Context
4 Pillar instances – SAP, PS & EBS ( 2)
Largest EBS instance upgraded to 12.2.4
80% of revenue transacted
53 countries
Approx. 10000 users
DB Size approx. 14 TB
150+ companies
Upgraded from 11.5.10 to 12.2.4
Our Upgrade Journey…..
Upgrade vs Re-implementR12 Technical Upgrade R12 Upgrade Plus R12 Reimplementation
Shortest
Tech upgrade to R12.2
• Customization remediation
• Integration and Bolt-on checks
• Regression & Performance Testing
• New Features: SLA, inv org, multi-org, role- based access controls
• Retire Customizations:
• Convert GL to Corporate COA
New R12.2 Instance
• Rationalize customizations
• Leverage new R12 features
• Potential for Phased Roll-out
Lowest
Lowest
Supportability
Real-time Patching
Timeline
Cost
Funct.Resource
Needs
BusinessBenefits
New
Features
Retire
Customizations
Corporate
COAs+ +
Medium - Long Longest
Medium - High Highest
Medium - High Highest
Supportability+ Re-engineering
Technical Upgrade allowed for other business priorities.
Incremental Change in Business process instead of Big Bang
89/15/2016
Initial Project Timeline
Code Remediation
2014 2015
Q3 Q4 Q1 Q2
CoE
Tech
Steady increase in program resourcing needs over time
Global
Teams
Core
Team
Planned Timeline
Resources
Leveraging a small core team to maximize value
R12 Upgrade Testing
99/15/2016
Global Functions
SVCGSC FIN Commercial ENG
IT Technical / Quality Team
GlobalManagementTeam
Senior LeadershipTeam
ProgramManagement
Region / P&L
Region/ Modality CIOs
Functional Program Manager
VP-Service
VP-ITVP- Supply
ChainCFO
CTO Regional GMsSupply chainG
SC
Reginal GMFinF
IN Sales GMC.E
.
GM Engg
EN
G
CIO
Fin Leader
IT Fin Leader
GSC Leader
IT GSC Leader
Service Leader
IT SVC Leader
ENG Leader
IT Eng Leader
Bus Integration
Business Integration Leader
Region / P&L Integration Leader
ITProgram Leader
Quality & Compliance Leader
Commercial ExLeader
Comm. IT Leader
Program Organization
SV
C
Service GMS
TechnicalProgram Manager
109/15/2016
Team Functional IT
Senior Leadership Team (SLT)
• Top Down Support – CAP / Communication
• Strategic Direction• Help with staffing engagement
• Top Down Support – CAP / Communication
• Strategic Direction• Help with staffing engagement
Guidance Management Team (GMT)
• Drive func leadership selection• Monitor progress, provide help
needed• Authorize major project variances• Manage conflicting priorities• Assist with functional staffing• Make the final Go / No-Go decision
• CIO engagement of functional leadership
• Drive IT leadership selection• Approve budget changes• Resolve cross program issues• Monitor progress, provide help needed• Make the final Go / No-Go decision
OverallProgramExecutive
• Manages SLT engagement• Organize and Lead GMT Sessions• Develops overall program strategy & organization• Manages top-level risk mitigation
ProgramManagers
• Organizes the overall program and mobilizes teams
• Drives overall program operating mechanisms, issues, and risk mitigation• Manages program Plan, risk mitigation, tollgates• Manages program budget• Manages communication plan
Business IntegrationLeader
• Creates Overall Program Test strategy• Establishes and drives test reporting• Drives cross functional integration
• Identifies simplification opportunities for future releases
Region IntegrationLeader
• Drives regional engagement through best practices and common work structure• Pivots between program, global processes and region / P&L leaders• Plans the work for the regional leads and monitors the overall region plan• Identifies top regional issues and drives to closure• Leads commercial staffing, supports functional staffing
Top Management Support
Staff Leaders
Make Go/No-go
LeadershipEngagement
Mobilize & Orchestrate
Cross-functionalConsistency &Integration
LeadRegionEngagement
Roles & Responsibilities
119/15/2016
Team Functional IT
Global Process Leader
• Recommend Go / No-Go• Assure functional coverage &
obtains resources• Validate functional test plan• Escalates critical risks• Organizational readiness
• Identifies process eco-system • Develop staffing coverage strategy and
resource plan• Staffs GE & contractor positions• Drives issue resolution• Liaison's to function
Global Track Leader
• Defines critical processes• Defines test scenarios• Manages test execution• Escalates critical issues
• Identifies testing coverage for customizations & interfaces
• Gathers data for testing• Drives fixing defects• Drives system documentation
maintenance
RegionalLeader
• Engages regional leadership and commercial teams for testing & readiness• Identifies local system integrations, country localizations, and CFD’s• Coordinates regional testing as needed – particularly where local tools are involved• Identifies regional risks and issues and drives to closure• Define the region go-live readiness (acceptance) criteria
IT TechnicalProgram Leader
• N/A • Plans & manages outage window execution
• Performs stress test & tunes system• Governs customization designs
• Drives Overall eco-system connectivity• Manages validation & compliance• Leads central break/fix for Oracle defects• Identify & modify BI fixes
OrganizationalReadiness
Process & ModuleReadiness
Run the business…Find the Bugs!
Run the Systems…Fix the Bugs!
Assure Regional Readiness
Upgrade Execution & System Performance
Roles & Responsibilities
129/15/2016
2014 2015 2016
Actual Program Timeline
NovOct Nov Dec Jan Feb Mar Apr May DecJun Jul Sep OctAug
Go LiveJan
CRP1 CRP2 UAT Validation Test
Development End to End Process Testing User AcceptanceTest
Compliance documentationTraining & Communication
DefectResolution
Defect Resolution
Key Activities:
CRP1 – Code remediation and functionality exploration
CRP2 – End to end testing and Bug fixes
UAT – User acceptance testing
Upgrade simulations to optimize the time
Back out planning and Cutover planning
Simulate Back out plans.
Communication & Training
Go-Live
6 Upgrade Simulations
139/15/2016
Upgrade completed on time – system came up 6pm Sunday as planned
Critical Business transactions exceeded daily averages (starting day 1 )
No post go-live outages
Successful January Close
Program Go-Live Highlights
Post Go-Live Executive Summary
“Just an awesome, awesome job!!!!” ~ CFO GEHC
“Best upgrade I have seen in 20 Years” ~CFO SC P&L
“As close to perfect as you could expect in an ERP upgrade” ~ VP &
CIO GEHC
149/15/2016
Key Learnings
159/15/2016
Key Learnings
ERP is not standalone 350 Interfaces ;Inbound and outbound
120 connecting systems
3 BI tools
Transactions flow from Source to ERP to destination systems
Code remediation and testing done across systems
…… But… How do we shutdown this web of systems?
Key is to understand dependencies and cross-impacts
How do we startup this web of systems?
A pilot can fly alone easily but when 15 pilots fly together, it’s a
different experience…. That’s how it feels like in the ecosystem !!
Think about your Ecosystem; not just the ERP
169/15/2016
Key Learnings
Use of tools to remediate code Automation of code remediation saved us months in the
project schedule
Code remediation activity could be run like a factory
Total number of code Objects remediated ~ 2300
…… But… We found that like all software, our tools had issues..
We missed code in ecosystem, missed SSWA objects,
missed embedded queries in apps like QA, Pricing etc.
Online patching compliance
Trust your tools but verify the results !
179/15/2016
Key Learnings
Test plan and Test coverage We executed 13000 test scenarios in CRP’s
Fantastic functional team engagement
Defect monitoring and root cause analysis helped
Exited UAT with 12 defects across all tracks.
Load testing and Regression Testing
…… But… Initially we did not test transactions which start in 11i but
end in R12. Practice period close. Get used to period close
analyzers.
Our root cause analysis did not look at underlying trends
and systemic defects…. We improved on that.
Start your Load testing early.
If using i-modules, please test them from outside the
firewall.
Test boundary conditions & exceptions !
189/15/2016
Key Learnings
Appreciate your team Any Oracle upgrade is very demanding on people
Work hours will be long and often at odd hours
Workload for different teams like developers, middleware team,
testers, users will be fluctuating.
Workload for DBA’s tends to be high all the times.
Keep morale up. Take pride in milestone accomplishments and
learn from every incident.
This is team work. Trust is key. Help in building it
…… But… Think about back ups
Think about shifting work hours
Think if you run a risk of knowledge concentration in a few
Think about your communication plan across the globe if you are
global
Remember DBA’s are the centerpiece of this project
199/15/2016
Key Learnings
Functional Users Need a Strong buy-in.
Knowledgeable super users are the key to success
Users need to test the system as they use it everyday
Reports, labels are the key
Spend ample time in communications, awareness and training.
Posters and banners at major hubs and Mfg/ Dist sites.
Think about having a dedicated communications leader
Involve them in sanity testing during simulations
Use your super users as your messengers and damage controllers
in case of small hick ups.
Think about your communication plan across the globe if you are
global
Project success = Happy Users
209/15/2016
Key Learnings
Upgrade Considerations from a DBA perspective Archive or Purge unwanted data
Refresh all MVs to reduce upgrade time
Concentrate on long running jobs
Parallelize tasks if possible
Clean up database objects before enabling ADOP (duplicate tables
etc.)
Implement SPMs and SQL profiles for long runners
…… But… Do end to end simulations
Plan for backups before critical steps
Check logs always for any unexpected errors
Follow the upgrade manual from Oracle. No Short cuts please !!
219/15/2016
Key Learnings
Cutover planning & execution We had a 1900 line project plan just for the cutover tasks
Understand the dependencies of tasks.
Understand dependencies of people. They need to trust each
other. Handshakes on tasks are important
Co-location helps a lot
Simulate like it is Production. Avoid the temptation of shortcuts.
…… But… Note every single incident for investigation later
Think user lock down is simple…it might not be.
Plan your simulations for every single failure. Don’t take things for
granted.
Practice, practice and practice….
229/15/2016
R12 Upgrade Outage planProvide high quality instance as efficiently as possible
What actions were taken?
• Practiced and improvised the shutdown process.
• Created save points which were key to BCP
• Worked with Oracle to get best practices
• Oracle fix was practiced and verified
• Conducted multiple simulations to iron out the process
• Added time for GS and introduced a functional checkout process
Save Points
What Did We Learn?
Go/No Go
Monday Tuesday Wednesday Thursday Friday Saturday Sunday24-Jan
Monday25-Jan
R12.2Code,
Patches Ch
eck
O
utSystem
shutdown
R12.2.4/Online Patching
3rd
Party,M/W
• Graceful shutdown adds 12 hours (flushing data through system)
• Issues need to be recreated. Painful.
• VCP first collection was slow
• Issues with R12.2.driver in the first 6 months vs the last 6 months
• Code migration, VCP collection, Sabrix install and Brazil localizations took 42 hrs.
• System startup was not smooth
VCP
What were the outcomes?
• Finally reduced it for 7 hrs.
• Gained ability to run multiple simulations at same time.
• VCP collections was not a bottleneck
• Reduced the time for the last phase to 27 hrs. from 42 hrs.
• Resulted in a very high user confidence
Practice to optimize the plan and understand dependencies
Outage at ~5 Days… additional time needed to ensure Quality
239/15/2016
Key Learnings
Where is your Plan B? We had a 300 line back out plan
Plan for back out from every step of the upgrade
Playbook for executive engagement in the worst case
Have Plan B incase of delays. Understand steps where you could
lose time and where you could gain time
Use the time gained wisely. Try to stay ahead and be ready for
contingencies.
Have playbooks for concurrent activities going out of sequence.
Practice your back out plans and Plan B’s to make sure they can
work.
Your Back out plans should cover the ecosystem not just the ERP.
Have a Plan B for every possible thing you can think about.
249/15/2016
Key Learnings
System Performance Comprehensive load testing
Verified performance between 11i & R12 up to the second
API and Interface performance is important too.
Watch out for first collection run for ASCP.
…… But… Watch out for performance post-upgrade
Allow time for post upgrade steps to improve performance
Simulate post upgrade performance to take corrective actions
Productions system will be faster than your test environments
Gather Stats and Table re-orgs will help. Ask your DBA’s to plan for
those pre and post Upgrade
Plan for post-upgrade performance. Don’t make it an after thought !
259/15/2016
Key Learnings
Post Go-live support Engage Production support teams during UAT
Pair the project SME’s with the support SME’s
Be ready for a high call volume
Have regional contact points by defect types for level 1 support
If you are changing java versions, be ready for login Issues.
Over communicate about go-live . Communicate not only during
the project but even after go-live.
Be ready with swat teams especially for finance.
All the best for your upgrade & hope that our learnings will make you smarter.
269/15/2016
279/15/2016
Operating Mechanism
Purpose / Objective Participants Frequency
Senior Leadership & GMT Review
Inform corporate officers of project status, issues, and help needed
Senior Leadership Team, aligned GMT members, selected team members, Global functional & IT leads, Gilles, Jason
Monthly
Program Leadership Forum
Strategic review, develop recovery plans, make course corrections
Program management, Global functions, Gilles, Region Leaders, Jason
Monthly
Program Review Review scorecard status at end of week; weekly status report preparation
Program management, Global functions, Gilles, Jason
Weekly
Global functional Review
Manage track progress, issue escalation & mitigation, update on program information
Global functions, track leads, region integration leader, break/fix manager
Weekly
Region LeadershipReview
Manage region progress, issue escalation & mitigation, update on program information
Region leader, Gilles, Brian,Optional: Functional leads
Weekly
Training & ImprovementForum
Add / update training & improvement lists Functional Leads, Gilles, Brian, Architecture leader
Bi-Weekly
IT Technical Integration
Session
Manage IT Platform progress, issue
escalation & mitigation, dependencies
IT Technical Team Weekly
Defect ManagementReview
Defect assignment, status, critical issues by module
Defect manager, IT track leads, BACOE architects, BI COE, GAMS
Weekly
Project Plan Review Progress vs plan update on activities Program management, Gilles, Jason
Weekly
Operating Mechanisms