Copyright © 2010 Rockwell Automation, Inc. All rights reserved. ALM226 SAP Migrations Made Easy...

Copyright © 2010 Rockwell Automation, Inc. All rights reserved.

ALM226SAP Migrations Made Easy

Heiko Zuerker

Copyright © 2010 Rockwell Automation, Inc. All rights reserved. 2

Agenda

Rockwell Automation’s SAP Landscape

Why a different Process?

OS Migration High Level

OS Migration in Detail

Lessons Learned

Rockwell Automation

• Manufacturer of industrial automation control and information solutions• World headquarters in Milwaukee, WI• Annual sales about $4.3 billion • Serving customers in more than 80 countries• About 19,000 employees• Over 15,000 internal SAP users• Over 5000 external SAP users (distributors)


SAP Single Global Landscape


CRM(C)

InternalPortal

(E)*XI(X)

ERPR/3(R)

PR1 PC1 PX1

Production(P*0)

Dev0(D*0)

SRM(S)

PCDE(B)

PS1 PB1

SCMAPO(A)*

PA1PE1

PC0 PX0

Prod Sup(P*1)

PS0 PB0PA0PE0

GRC - Process Control

(D)

PD0

QR0 QC0 QX0QS0 QB0QA0QE0 QD0QA (Q*0)

DR0 DC0 DX0DS0 DB0DA0DE0 DD0

SM2 (Solution Manager)

APOLive

Cache(L)

PL1

PL0

QL0

DL0

PW0 (WebAS) CUA/CCMS

BW 7.0(J)

PJ1

PJ0

QJ0

DJ0

MDM(M)

PM1

PM0

QM0

DM0

BIA(Z)

PZ1

PZ0

GTS(T)

PT1

PT0

QT0

DT0

LS0 (System Landscape)

SCM APO

Gateway(G2*)

G21

G20

G22

G24

Sandbox(S) SR0 SC0 SX0SS0 SB0SA0SE0 SD0 SL0SJ0 SM0 ST0 G26

En

viro

nm

en

t

QR1 QC1 QX1QS1 QB1QA1QE1 QL1QJ1 QM1 QT1 G23QA1(Q*1)

Dev1(D*1) DR1 DC1 DX1DS1 DB1DA1DE1 DL1DJ1 DM1 DT1 G25

TREXSRM/

ECC5.0(U)

TREX_48PU0

PW2 (WebAS) CUA/CCMS

Training(T*0) TR0 TC0 TX0TS0 TB0TA0TE0 TL0TJ0 TM0 TT0 G27

JD2 (JDI)

JD0 (JDI)

SD(Y)

PY1

PY0

QY0

DY0

SY0

QY1

DY1

TY0

SM1 (Solution Manager)

TREXCRM(H)

PH1

PH0

QH0

DH0

SH0

QH1

DH1

TH0

PR0 PP0

ExternalPortal

(P)*SNC(N)

PN0

TREX_48PU1

TREX_48QU0

TREX_48Qu1

TREX_48DU0

TREX_48DU1

TREX_48TU0

TREX_48SU0

PP1

QP0

DP0

PN1

QN0

DN0

SN0

QN1

DN1

TN0

PV0

QV0

DV0

SV0

Access Control

(V)

TREXExt.Portal

(Q)

PQ1

PQ0

QQ0

DQ0

LS6 (non Prod SLD)

LS2 (System Landscape)

PF0

PF1

QF0

QF1

DF0

DF1

TF0

SF0

TREXERP 6.0

(F)

Mobile Infrastructure

(O)

SO0

Everything we do in our SAP Environment, we do 8 times…

Production databases are currently 12 TB with 25% growth per year.


Why Migrate?

• Reached capacity limits on existing hardware• Reached end of life for existing hardware• Leveraging commodity hardware decreases maintenance costs

– Easier to reuse/repurpose hardware– Easier to switch hardware vendors

• Completed full evaluations of different server architectures– SAP is fully supported on Linux operating system– x86 based architecture provided a significant price difference in the cost per “SAPS”

• All other components (i.e. Veritas Cluster, Control-M, HPOMS, etc.) supported on Linux• Not all SAP applications are supported on Solaris x86

– NW6.40 Java stacks are not supported– Adobe document services are not supported

• TREX 7.10 only supported on Linux x86 and Microsoft Windows

SAP Production Environment Pre & Post

Sun Solaris 9

Sun V4904 CPUs (2 Cores each) 1.35 - 1.8 GHz / 32GB RAM

4 Servers for Central Instances (Clustered)

1 Server for TREX’s

24 Servers for Dialog instances

57 Central and Dialog Instances


SUSE Linux Enterprise Edition 10

HP Blades BL460c2 CPU’s (4 Cores each) 2.93 GHz / 96GB RAM

2 Servers for Central Services (Clustered) (SCS, ASCS, IPC, TREX)

16 Servers for Application Servers (Central and Dialog Instances)

39 Central and Dialog Instances (retired 18 Instances)

SAP’s Standard OS Migration Process

• SAP recommends and supports the use of the “Heterogeneous System Copy” for OS/DB Migrations– Note 82478 - SAP system OS/DB migration

• Procedures shown deviate from SAP’s standard process• SAP AGS

– Worked with SAP account team to ensure continued support after the OS migration– Not in violation of any support, maintenance or warranty agreements

• SAP Services– Scheduled several OS / DB migration checks

• http://service.sap.com/osdbmigration– Scheduled several Go-Live checks

• http://service.sap.com/goinglivecheck


http://service.sap.com/sap/support/notes/82478

http://service.sap.com/osdbmigration

http://service.sap.com/goinglivecheck


Why did we not use the Standard Process?

• Fast track OS Migration– Estimated time 1 hour executing the migration, plus 3 hours of post steps (mainly SGEN)– Minimal changes to the system configuration less risk

• Heterogeneous System Copy too time consuming– Estimated time, including post steps, 12 hours per system– Each basis resource is able to complete 2 systems in parallel. More can be dangerous.– 12 production systems and only 8 basis resources– Basis resources need to be staffed for post-migration support– Outage of all production systems for at least 24-36 hours

• Requires complete re-installation of dialog instances too time consuming– All instance specific settings need to be reapplied (certificates, SSL, proxymap, memory, threads, etc.)

• Lots of “manual” labor high risk of errors/mistakes, especially during night shifts• Many of the tasks need to be done anyway, no matter which process

– SAP profile settings– Java parameters (Configtool)– SAP kernel and crypto library updates– SGEN, SPAD, SM69– Architecture changes

• Testing requirements are the same

What did we do in 16 hours?

• Migrated all production SAP instances to Linux• Upgraded the 12 production Oracle databases from 10.2.0.2 to 10.2.0.4• Separated ASCS/SCS (enqueue and message server ) from central instance, even on NetWeaver 6.40• Moved ASCS/SCS into clustered central services layer• Installed enqueue replication server for both SCS and ASCS• Standardized all start profiles

– Process naming– SAPCPE corrections (mostly for upgraded systems)– Gateway on SCS/ASCS

• Standardized all instance and default profiles– Minimum heap/extended memory– Minimum work processes– Password complexity– Timeouts

• Standardized Java Configtool settings– Minimum heap– Minimum threads– Java VM settings

• Standardized Kernel and CryptoLib versions• Moved all NetWeaver 6.40 systems to EX2 kernel and Oracle 10 instant client• Changed IP addresses of all SAP systems (each instance got its own VLAN) DR & virtualization• Validated all business-critical transactions


Fast track Migration Process High Level


SAP Kern

el Upgrade

Database Clien

t Upgrade

Java Runtime Upgrade

Config

Settings

Fasttrack OS Migration

Only works when going from one flavor of Linux/UNIX to another flavor of Linux/UNIX

Why does it work?

• ABAP code is platform independent (SGEN required)• Java code is platform independent• SAP’s Java stacks use some additional binaries (i.e. ADS), but

– The binaries for all supported operating systems are stored in the database– Instance just needs to be told to extract the correct binaries during bootstrap


Automate, Automate, Automate

• Automate as many tasks as possible• Scripts can be executed by one individual, in parallel, on many systems• Ensures consistency• Ensures quick execution• Ensure the team understands what the scripts do• Every basis resource should do at least one “manual” migration• Almost all steps shown in this presentation are automated• Many of the scripts created can be re-used during patches or upgrades• Focus on the benefits that will be realized during the production migration


Main Custom Automation Tools we used

• saprun.pl– Perl script which allows execution of Unix commands on multiple SAP instances,

systems or environments in parallel, with a minimal amount of effort on the user’s side. It leverages “expect” to control the SSH sessions.

– Example to shut down all instances in the production environment :saprun.pl -t appservers -c "stopsap #INSTANCE#" -e PROD

• j2eebatchconfig.sh– Automated configuration of Configtool settings via templates– Leverages Configtool’s BatchConfig functionality– https://cw.sdn.sap.com/cw/docs/DOC-108724

• sap_profiles.sh– Ensure that the DEFAULT.PFL, instance profiles and instance start profiles follow

company standards (password complexity, file format, variable use)– Defines minimum values for certain settings (heap, buffers, dialog wp’s)– Has the optional feature to email the profiles as a CSV file and re-import them later.

This makes mass changes to multiple instance profiles a breeze.Copyright © 2010 Rockwell Automation, Inc. All rights reserved. 13

https://cw.sdn.sap.com/cw/docs/DOC-108724

What we tested?

• Anything I/O related– File– Email– FTP– Printing– Job Scheduling

• Interface types, not all the interfaces• All scripts

– Scripts for interfaces– Administration scripts

• Business-critical transactions (system validation)


Migration Detailed Timeline

T-2 weeks• Install Hardware• Install Operating

System• Get SAP Licenses

T-1 week• Clean queues (daily)• Initial File Sync• Export Profiles to CSV

9:30pm• Shutdown (~ 30

minutes)

10:00pm• DNS changes (~ 30

minutes)• Disable Database

Archive Logs• Database Upgrades

(~ 4 hours)

10:30pm• OS Migration SAP

Systems (~ 1 hour)

Sunday 3:00am• Post Steps (~ 2 hours)• Enable Database

Archive Logs• Final System Restart

(~ 30 minutes)

7:00am• System Validation

(Business Critical Transactions)

11:00am• Unlock all Users

Noon• Productive Operations

ResumeT+2 weeks• Hypercare Support


The Migration Process – Preparation / Uptime

• Prepare the new hardware– Install and configure the new operating system– Create necessary SAP directory structure, mount points, etc.– Install Java runtime

• Initial file sync– Copy (rsync) all files from the local file system to the new servers ( /usr/sap/SID )

• Export profiles to CSV– Clean up the profiles in Excel– Copy CSV files to staging directory

• SAP Licenses– Request the licenses for the new hardware-keys– Import manually via Visual Admin into Java-only systems


The Migration Process – Downtime

• Stop all PI communication channels• Clean all queues• Shutdown SAP• Final file sync

– Execute rsync again to get the delta from last rsync• Re-Import the cleaned-up profiles from the CSV files• Remove all log files from every SAP instance

– /usr/sap/SID/*/work/* – /usr/sap/SID/SYS/global/SLOGJ* – /usr/sap/SID/*/log/SLOG* – /usr/sap/SID/*/data/rslgsta– /usr/sap/SID/*/j2ee/cluster/server?/log/*– /usr/sap/SID/*/j2ee/cluster/dispatcher/log/*– /sapmnt/SID/global/*LG


Replacing Binaries

• Replace database client binaries• Replace the SAP kernel & crypto libraries

– Delete contents of the /sapmnt/SID/exe directory– Extract SAPEXE.SAR– Extract SAPEXEDB.SAR– Extract IGSEXE.SAR (7.x Systems only)– Extract the crypto library

• Java kernel SAP NetWeaver 6.40 and older– Delete contents of the /sapmnt/SID/exeU directory– Extract SCS.SAR (from correct support package)– Extract SCSCLIENT.SAR (from kernel patch)

• Verify all binaries were replaced– find -type f -exec file {} \; | grep SPARC


Application Server (Instance) Migration

• NetWeaver 7.x– Delete contents of instance exe directory

• NetWeaver 6.40 and older Java stacks– Extract SCSCLIENT64UC.SAR (from SP) into os_libs directory– Copy os_libs/jstartup.jar into cluster/dispatcher/bin/kernel directory– Copy os_libs/jstartup.jar into cluster/serverX/bin/kernel directory– Copy exeU/libjmon.so into os_libs directory– Extract IGSEXE.SAR into igs/bin directory

• All Java stack versions– Delete the instance.properties.vmprop file to force a re-generation– Set element.resynch=force in bootstrap.properties

• All NetWeaver stacks– Recreate password file for SNC PSE– Clean up /usr/sap/tmp/dev_*


Java Configuration on the Central Instance

• Update the dispatcher and Java parameters depending on the JDK– Note 718901 - How to Change the JDK of the J2EE Engine– Note 709140 - Recommended JDK and VM Settings for the WebAS630/640/7.0– Note 723909 - Java VM settings for J2EE 6.40/7.0

• Set new JDK for SDM– Delete SDM/program/config/sdm_jstartup.properties.vmprop– Note 743583 - How to change the JDK of SDM

sdm.sh newjdk "javahome=$JAVA_HOME"

• Java stacks older than NW7.1 store the operating system type in the database– Update via Configtool


https://service.sap.com/sap/support/notes/718901




Updating the Operating System in Configtool


Switch to Configuration Editor Mode

Where to find os.name values?Install SAP once on new platform

to compare settings

ABAP and ABAP+Java Licenses

• Use the command line tools saplicense & saplikey to remove all old licenses and then add the new ones

• SAP NetWeaver 7.x– saplikey pf=/path/to/profile -delete \* \* \*– saplikey pf=/path/to/profile –install /path/to/license

• SAP NetWeaver 6.40– Need to use “expect” script to remove licenses or just do it manually– saplicense -install ifile=/path/to/license


Post Configuration

• Start the systems• Execute SGEN and generate all objects for all components• Import the changed profiles into RZ10• Update custom OS commands in SM69 for new platform• Update OS Commands for your output management system in SPAD


TREX Migration

• Can not export/import indexes when Endian-ness (byte-order) changes• New installation and full re-indexing required• Re-indexing report does not support parallel processing• TREX tuning can improve re-indexing performance significantly!• ECC TREX had 1.4 million documents, re-indexing took about 24 hours


Postponed TREX migration to 3 day upgrade weekend, due to time required for re-indexing.

Migration Done! Time for System Validation!

• Validate business-critical transactions• Validate interfaces• Validate printing, emails, faxes


Lessons Learned

• Test and get sign off on all interfaces - Do NOT just test the “types”• Test all I/O related processes• Watch out for character sets, some binaries may have different defaults• Stage as much as you can• Automate as much as you can• Ensure basis has a clear understanding of the underlying concepts


Closing Comments

• We did this because– Have the ability to re-use many components of this process– Had the ability to execute the migration within a standard maintenance window– Had the ability to include architecture changes due to time saving– Had the ability to include Oracle upgrades due to time savings– Less risk due to minimal configuration changes

• Test, test and then test some more• Don’t be afraid to try out new ideas• Use the opportunity to implement architecture changes• Use the opportunity to standardize• Use the opportunity to clean up the landscape• Schedule SAP OS/DB migration check and Going-Live checks


Questions ?


Copyright © 2010 Rockwell Automation, Inc. All rights reserved. ALM226 SAP Migrations Made Easy...

Documents

Transcript of Copyright © 2010 Rockwell Automation, Inc. All rights reserved. ALM226 SAP Migrations Made Easy...