1
<Insert Picture Here>
Oracle E-Business Suite Release 12.1 Upgrade Best Practices -
Technical Insight
Udayan Parvate, Director, EBS Release Engineering
Lester Gutierrez, Senior Architect, EBS Applications Performance
3
The following is intended to outline our general product
direction. It is intended for information purposes only,
and may not be incorporated into any contract. It is not a
commitment to deliver any material, code, or functionality,
and should not be relied upon in making purchasing
decisions.
The development, release, and timing of any features or
functionality described for Oracle’s products remains at
the sole discretion of Oracle.
4
Objectives
• Provide an overview of the R12.1 release, the technology stack
and supported upgrade paths
• Outline best practices for the R12.1 technical upgrade, with a
focus on downtime reduction
• Review runtime performance monitoring and tuning tips
• Provide sample customer upgrade profiles and share Oracle’s
GSI upgrade experience
5
Agenda
• R12.1 Upgrade Overview
• Upgrade Best Practices
• Customer Upgrade Snapshots
• References
• Additional Resources
• Q&A
6
R12.1 Upgrade Overview
7
R12.1 Rapid Install (RI)
R12.1 Maintenance Pack (MP)
• R12.1 was generally available (GA) in May 2009
– Via Oracle Electronic Product Delivery (EPD) and Oracle Store
– For all the supported platforms and languages
• Can be used by new and upgrading customers (11.5.9 and
above) to go directly to R12.1
– If you are on R11i, use R12.1 RI from EPD. Follow instructions from
the “Upgrade Guide: 11i to 12.1” and “12.1.1 Release notes”
– If you are on R12.0.X, use R12.1 MP (7303030) from My Oracle
Support (MOS). Follow instructions from “R12.1 Maintenance Pack
Install Instructions (752619.1)”
8
R12.1 Technology Stack
For details “Oracle E-Business Suite Release 12.1 Maintenance Pack Installation Instructions” (752619.1)
TECHNOLOGY
COMPONENT
VERSION
INCLUDED
11i10CU2
VERSION
INCLUDED
12.0.4 RI
VERSION
INCLUDED
12.1 RI
VERSION
CERTIFIED
WITH
MINIMUM
REQUIRED
VERSION
• Apps Mid tier-
Forms/Reports 6.0.8.25 10.1.2.2 10.1.2.3 - 10.1.2.3
• Apps Mid tier-
Java Oracle
Home/
• Apps Mid tier-
JDK
1.0.2.2/1.4.2 10.1.3.0/1.5 10.1.3.4/1.6.0 10.1.3.5 10.1.3.4/1.6
• Database/
JDBC drivers 9.2.0.6/9.2.0.6 10.2.0.3 11.1.0.7/11.1.0.7 10.2.0.5/11.2.0.1 10.2.0.4/11.1.0.7
9
R12.1 Upgrade Paths
• Minimum EBS suite level for direct upgrade to R12.1
– 11i9, 11i9CU1, 11i9CU2 or above
– 11i10, 11i10CU1, 11i10CU2 or above
– R12.0 and above
• Minimum EBS suite level required for database versions
– 10.2.0.4 11i9CU2, 11i10CU2
– 10.2.0.5/11.0.7/11.2.0.1 11i10CU2, R12.0.4
• Based on R12.1 DB guide (761570.1), certified upgrade path options can
be categorized into :
A. Upgrade database and EBS level in a single downtime
B. Upgrade database and EBS level in separate downtimes
C. Apply upgrade interoperability DB patches and then upgrade EBS level
10
R12.1 Upgrade Paths Continued.. 11.0/11i.X => 12.1
< = 11.5.8
11.5.9/cu1
11.5.10/cu1
12.1
10.2.0.4
12.1
10.2.0.5
12.1
11.2.0.1
12.1
11.1.0.7
11.0
11i10cu2
A
B ,C
B ,C
B ,C
B ,C
B ,C
A. Upgrade database and EBS level in a single downtime
B. Upgrade database and EBS level in separate downtimes
C. Apply upgrade interoperability DB patches and then upgrade EBS
SOURCE
TARGET
11i9cu2
11
R12.1 Upgrade Paths Continued.. 12.0.X => 12.1
12.1
10.2.0.4
> = 12.0.4
12.1
10.2.0.5
12.1
11.2.0.1
12.1
11.1.0.7
< = 12.0.3 A
B ,C
B ,C
B ,C
B ,C
B ,C
A
A. Upgrade database and EBS level in a single downtime
B. Upgrade database and EBS level in separate downtimes
C. Apply upgrade interoperability DB patches and then upgrade EBS
SOURCE
TARGET
12
R12.1 .1 Key Facts
11i10CU2 R12.0.4 RI R12.1 RI
# of Product Schemas 209 195 (25 removed, 11 added
when compared with
11.5.10 CU2)
201 ( 25 removed, 17 added
when compared with
11.5.10 CU2 )
#file calls in DB
portion of the U driver 104242 144940 156622
PROD database size
File system size
31 GB
26 GB
45 GB
28 GB
50 GB
28 GB
#files shipped in RI 268359 357778 385755
#of Changed+ New
files in DB portion of
the U driver
NA ~95488 ( Vs 11.5.10.2 ) ~104057 ( Vs 11.5.10.2 )
~31843 ( Vs 12.0.4 )
13
Upgrade Best Practices
14
Upgrade Best Practices
• Identification of required patches
– Prepare a complete list of pre and post patches and recommended
code levels
• Keep the system current on AD/ATG/OAM code e.g. latest AD/ATG
RUPs on 11i/R12.0 and once on R12.1
• High priority patches from MOS.
• Consolidated Upgrade Patches (CUP) and required one-offs that
need to be applied in pre-install mode. R12.1 CUP1 (7303029:12.1.0)
is available now
• Review “Known-issues” sections from key documents to determine
additional pre or post upgrade patches: Release notes, MP Install
Instructions
15
Upgrade Best Practices Continued..
• Patch merging and sequencing
– Take advantage of patch merge, hot patching and apply patches in the
correct sequence
• Merge patches
• Merge NLS patches per language. Apply NLS patches as “hot”
patches
• Perform uptime maintenance when possible e.g. hot patching of
iHelp or NLS patches, upload patch history
• HRGLOBAL step before NLS ( Document 145837.1 & 414434.1 )
• Stage patches locally accessible by mid-tier machines instead of on
nfs mounted or remote servers
16
Upgrade Best Practices Continued..
• Adpatch options and other reporting tools
– Use applicable adpatch options to reduce downtime and track key
upgrade aspects using in-built tools
• Use non-interactive patching
• Use adpatch options such as nomaintainmrc, phtofile, nocompiledb,
nolink, nogenform, nogenrep, nocompile jsp, noautoconfig, novalidate
(1078973.1)
• Use optimal values for batch size and #workers
• Use OAM reports and Patch Wizard functionality to track patch
timing, files applied, functionality patched, impacted customized files
etc. (976188.1, 976688.1)
17
Upgrade Best Practices Continued..
• Downtime reduction procedures – Use “Shared APPL_TOP” with “Distributed AD” for upgrades and
regular maintenance for multi-node instances
• No need to apply the same patch on multiple tiers
• Distributed AD adds to the degree of parallelism by distributing AD workers across application tier nodes and improves D/G portion of the patch driver.
– Use “Staged APPL_TOP” for regular maintenance and upgrade
• Saves time to patch the file system (C/G portion) by using a patched up copy of production instance file system
• Use in 11i => R12.1 upgrade to avoid applying NLS C/G portion
• Can use for R12.0.X => R12.1 upgrade and once on R12.1
18
Upgrade Best Practices Continued..
• Downtime reduction procedures continued..
– For the duration of the upgrade, consider…
• Disabling custom triggers and business events
• Disabling auditing if enabled
• If possible run in noarchivelog mode
• Disabling flashback DB
• Removing TDE from volume tables
19
• Pre-upgrade activities
– Identify and execute tasks that could be completed in a
separate downtime period, prior to the production upgrade
• Use applicable steps mentioned in the "Downtime reduction" and “Upgrade By Request” appendices E and G of the R12.1 upgrade guide
• Upgrade RDBMS version to latest certified for the current APPS level ( 11.2.0.2 / 11.1.0.7 / 10.2.0.5 )
• Convert to Oracle Applications Tablespace Model (OATM)
– Compacts data, optimizes storage settings and I/O
• Other planned HW or OS upgrades
Upgrade Best Practices Continued..
20
• Pre-upgrade activities continued...
• Use TUMS (“The Upgrade Manual Script”) to avoid running
tasks that are not relevant for your system
• Convert to multiple organization architecture (Document 210193.1)
• Drop MRC schema ( 11.5.10 and above )
• Assign post upgrade jobs to specialized CM queue (by
request_type, see Document 399362.1)
Upgrade Best Practices Continued..
21
• Pre-upgrade activities continued...
• Flush all the interfaces, such as Autoinvoice, Journal entry
import, order import etc..
• Review and disable all debug, logging at site, responsibility,
user level
• Gather stats with GATHER_AUTO option for all schemas
close to the start of downtime
Upgrade Best Practices Continued..
22
• Pre-upgrade activities continued...
• Pre-create new large indexes created by upgrade (constrained
to indexes on existing columns)
• Minimize historical data to be upgraded as per business
requirements – “Upgrade By Request”
• Purge old and/or transient data before upgrading
– Over 260 standard purge programs in R12
– Use new “Purge Portal” in OAM to administrate
Upgrade Best Practices Continued..
23
• Purging – Use OAM to configure, initiate and monitor purge programs
• Set the execution frequency and view program history
• Programs tagged with the “Purge” program type
System Administrator >Oracle Applications Manager >Purging/Critical Activities
Upgrade Best Practices Continued..
24
• Key recommended fixes
– Affecting Upgrade Performance
• AD-Parallel improved scalability: 8917381 (Part of 9179588 AD
CUP)
• Forms/reports generation regression with 11g: 8557019
– General
• Review document 244040.1 for latest EBS recommended
performance fixes
Upgrade Best Practices Continued..
25
• Database tier configuration
– Maximize SGA and PGA sizing
• An upgrade only involves 10's of concurrent sessions;
starting rules of thumb ...
– log buffer = 30 to 100 Mb
– shared pool = 1 to 4 Gb
– pga target = 3 to 20 Gb
– buffer cache = multi Gb, be generous without causing
excessive paging or swapping
• Adjust with help from AWR pool advisories
Upgrade Best Practices Continued..
26
• Database tier configuration
– Other upgrade specific init.ora changes
• If specified, remove db_file_multiblock_read_count
– Maximize multiblock I/O sizes
• Set job_queue_processes = # of CPUS
– adobjcmp.sql (Phases : plb+90 and last+63)
• Set parallel_max_servers = 2 X CPUs
– Helps with large index creation, stats gathering and some
large upg+ phase jobs
– Need to test with production-like DB server and I/O subsystem
– Shutdown other RAC instances
Upgrade Best Practices Continued..
27
• Batch size and #workers
– Batch size
• 10K is suitable for most installs, you can test other values from 1K
up to 100K if time allows
– # Workers
• Starting rule-of-thumb is between 1 and 1.5 x #CPUs
– It is critical to do multiple rounds of testing, adjusting above settings to
maximize server utilization, but constrained by factors such as
• Memory utilization (no swapping/ excessive paging)
• CPU utilization (scale down if at 100%)
• I/O response times (scale down if averages > 20 ms)
Upgrade Best Practices Continued..
28
• Performance testing, monitoring and additional
optimizations...
– Analyze long runners via timing report, ad_task_timing analysis
– Mine ad_task_timing to identify low worker utilization due to
phasing waits and review responsible culprits
– Review targeted AWR Instance and SQL reports
– awrrpt.sql and awrsqrpt.sql
– Use My Oracle Support to check for known issues and
workarounds for the longest running jobs
Upgrade Best Practices Continued..
29
• Performance testing, monitoring and additional
optimizations continued... – Outside-the-box optimizations – Requires Support Approval*
• Long running index creation and Statistics gathering
– As these tasks run with little or no concurrent tasks, consider using alter system commands to increase parallel_max_servers and pga_aggregate_target
(Need to test so as to not exhaust machine resources)
– Large indexes: Pre-create, but do not alter column definitions
– For stats (adsstats.sql ; phase: last+63) : On RAC, consider using all nodes when gathering stats or use exp/imp of stats
Upgrade Best Practices Continued..
30
• Performance testing, monitoring and additional
optimizations continued... – Outside-the-box optimizations – Requires Support Approval*
• Long running mv related xdf’s (“en” phases)
– Cleanup and truncate large MV logs (requires MV complete refresh)
• Look more closely at long running jobs that you suspect may not be required for your system
– Thousands of jobs and customer product/data/setup configurations
– Some of the heavy lifting work may not be required for a specific site
– Check with Oracle Support to validate
– A fix maybe available or possible to optimize for your case
– New 12.1.3 AD timing report by module
Upgrade Best Practices Continued..
31
• Runtime Performance Testing Tips – Use Automated, scripted tools
• EBS Test Started Kits (Winrunner/QTP)
– Bundled QA based automated scripts for EBS testing - Patch 8408886
• Oracle Applications Testing Suite (Accelerators for EBS)
– Web and Forms based flows
– Complement with user participation tests and batch load tests with frequent
and critical jobs
– References
http://blogs.oracle.com/stevenChan/2009/10/oats_ebs_certified.html
http://blogs.oracle.com/stevenChan/2009/08/evolutionary_steps_automated_testing_ebs.html
Upgrade Best Practices Continued..
32
• Runtime Performance Testing Tips – Review the white paper: Upgrade to Oracle 11g - Performance Best
Practices
http://www.oracle.com/apps_benchmark/html/white-papers-e-business.html
• Points to note include11g features for
– Testing of 11g or 10g workloads with Real Application Testing
– Managing SQL Execution Plans and minimizing performance degradation with11g SQL Plan Management
– Session OOW2010: Tuning the Oracle E-Business Suite Environment
– Review EBS Benchmark Publications • http://www.oracle.com/apps_benchmark/html/white-papers-e-business.html
Upgrade Best Practices Continued..
33
Customer Upgrade
Snapshots
34
Oracle E-Business Suite Release 12.1 Live
Customers
35
Customer Upgrade Snapshots
11i to 12.1
• CPS (Chicago Public Schools) – Release: 11.5.10.2 to 12.1
– DB size: 900GB
– #Workers and batch size: 32, 10000
– #CPUs on DB server: 2 node RAC, 8 CPUs per node
– Downtime reduction measures
• Distributed AD with shared APPL_TOP
• Upgrade RDBMS to 10.2.0.4 in a separate downtime
• # hrs for the D driver: 22 hrs
– Customer snapshot http://www.oracle.com/customers/snapshots/chicago-public-schools-ebs-snapshot.pdf
36
Customer Upgrade Snapshots Continued...
11i to 12.1
• Cisco
– Release: 11i to R12
– DB size: 600GB
– #Workers and batch size: 32, 20000
– #CPUs on DB server: 16
– Downtime reduction measures
• Distributed APPL_TOP
– #hrs for the D driver: 5.5 hrs
37
Customer Upgrade Snapshots Continued...
11i to 12.1
• Dell
– Release: 11i10 to R12.1.2
– DB size: 15TB , 16 node RAC Cluster
– #Workers and batch size: 32, 10000
– #CPUs on DB server: 8
– Downtime reduction measures
• Distributed APPL_TOP
• Pre-create large indexes
– #hrs for the D driver: 30 hrs (latest test results)
38
Customer Upgrade Snapshots Continued...
12.0 to 12.1
• Zebra Technologies Corporation – Release: 12.0.6 to 12.1
– DB Size: 106GB
– #Workers and batch size: 32, 10000
– #CPUs on DB server: 8
– Downtime reduction measures
• Staged APPL_TOP
– #hrs for the U driver: 12 hrs
– Customer snapshot http://www.oracle.com/customers/snapshots/zebra-technologies-corporation-ebs-snapshot.pdf
39
• Oracle GSI – Release: 12.0.3+ to R12.1
– DB size: 17TB
– #Workers and batch size: 60, 10000
– #CPUs on DB server: 88 processors
– Downtime reduction measures
• Staged APPL_TOP for US and ten languages
• Ran data fixes for problems found in test upgrades prior to production upgrade to minimize stoppages
• Distributed APPL_TOP (4 servers - 15 workers each)
• Pre-create large indexes (6 hrs savings)
– #hrs for the D driver: 14 hrs
Customer Upgrade Snapshots Continued...
12.0 to 12.1
40
• Oracle GSI – Release: 12.1+ to R12.1.3 -- (Latest Testing)
– DB size: 17TB
– #Workers and batch size: 200, 10000
– #CPUs on DB server: 150 processors
– Downtime reduction measures
• Staged APPL_TOP for US and ten languages
• Ran data fixes for problems found in test upgrades prior to production upgrade to minimize stoppages
• Distributed APPL_TOP (4 servers - 50 workers each)
• Pre-create large indexes
– #hrs for the D driver: 4 hrs
Customer Upgrade Snapshots Continued...
12.1 to 12.1.3
41
• AT&T – Release: 12.0+ to R12.1.2
– DB size: 10 TB
– #Workers and batch size: 40, 10000
– #CPUs on DB server: 32 Processors
– Downtime reduction measures
• Staged APPL_TOP for US and ten languages
• Distributed APPL_TOP
– #hrs for the D driver: 9 hrs
Customer Upgrade Snapshots Continued...
12.0 to 12.1
42
References
43
References
• R12.1 documentation roadmap (790942.1)
• “Oracle E-Business Suite Release 12.1 Info center” (806593.1)
• Database preparation guidelines for R12.1 upgrade (761570.1)
• Patching FAQs (459156.1, 225165.1)
• Using staged or shared APPL_TOP and distributed AD (734025.1, 384248.1, 236469.1)
• OAM “Patch Wizard” overview and FAQ (976188.1, 976688.1)
• AD Command Line Options for Release R12 (1078973.1)
• Recommended Performance Fixes (244040.1)
• R12 Upgrade Sizing & Best Practices (399362.1)
44
Additional Resources
45
Additional Resources
• White paper
– Planning Your Oracle E-Business Suite Upgrade from Release 11i to
Release 12.1 (987516.1)
– R12 Upgrade considerations by product: Financials (889733.1)
• Have upgrade questions ?
– Post on OTN R12 upgrade forum
http://forums.oracle.com/forums/forum.jspa?forumID=395&start=0
46
Finding Additional Information Accelerate your evaluation and planning
Contains
• Presentations
• RCDs
• RVPs
• Videos
• Customer
Stories
• White Papers
• etc
http://launch.oracle.com/?OOW
47
Oracle Products Available Online
Oracle Store
Buy Oracle license and support online today at
oracle.com/store
48
We encourage you to use the newly minted corporate tagline
“Software. Hardware. Complete.” at the end of all your
presentations. This message should replace any reference to
our previous corporate tagline “Oracle Is the Information
Company.”
49
Oracle OpenWorld
Latin America 2010
December 7–9, 2010
50
Oracle OpenWorld
Beijing 2010
December 13–16, 2010
51
EBS 12.1 UPGRADE DEMOPOD
MOSCONE SOUTH, S-089
MON : 9:45 AM – 5:30 PM
TUE : 9:45 AM– 5:30 PM
WED : 9:00 AM – 4:00 PM
52
SUMMARY
DO’s..
•Read and follow official documentation and have a project plan
•Engage Support early on to validate your upgrade plan
•Identify and execute tasks you can do today
•Test ! Test ! Test !
•Keep track of patches applied in test upgrades
•Be smart about using the right tools and explore published downtime reduction techniques
•Optimize patch runs to suite your H/W
DON’Ts..
• Ignore including relevant functional SMEs in the planning process from the beginning
• Skip failed or long running jobs as a regular practice
• Drop indexes
• Run adpatch in serial mode
• Perform workarounds without approval from support/DEV
• Insufficient time and/or # of rounds of upgrade and critical functional flow testing