All Change:The Story of a MigrationDr. Jason Arneil

• Introduction

• The Installation

• The Migration

• Conclusion

AGENDA

Introduction

• Jason Arneil

• Oracle DBA/System & Storage Administrator

• Using Oracle since 1998

• At Nominet since 2001

About Me

Introduction

• Nominet is the internet registry for .uk domain names

• Nominet has just celebrated its 11th anniversary

• Nominet is run as a not-for-profit company

• Nominet is owned by its members

• There are almost 6 Million .uk domain names

About Nominet

Introduction

Hardware/Software Stack

Oracle10.1

Veritas

Powerpath

Solaris 8

EMC

Introduction

Motivation

• Replace ageing SPARC hardware

• New 4-way Sun SPARC servers ~ £50K EACH

• Equivalent 4-way AMD Opteron cost £20K in TOTAL

• Reduce complexity in the software stack

• Ability to store accented characters

Introduction

Changes

10.1.0.3 10.2.0.2

Solaris 8 RedHat AS 4

VxFS ASM

US7ASCII UTF8

SPARC AMD

Introduction

Issues

• This was not just a case of running an upgrade script

• Database was some 100GB in size

• Change in endianess an additional complication

• Downtime allowable - but needed to be the minimum possible

The Installation

Initial Server Install

• Server installation kept to as minimal rpm’s as possible

• Recommended install rpms have many package dependencies

• Both 32-bit and 64-bit versions of certain packages are required– rpm --query --queryformat “%(NAME)-%(VERSION).%(RELEASE)

(%(ARCH))\n” rpm name

• Binutils rpm incorrect with RedHat Update 2 – rpm -Uvh --force binutils-2.15.92.0.213.0.0.0.2.x86_64.rpm

The Installation

ASMLib

• Eliminates need to bind raw devices

• Need to install 3 rpms:

– oracleasm-support

– oracleasm-your-kernel

– oracleasmlib

• Must get ASMLib from

http://www.oracle.com/technology/tech/linux/asmlib/index.html

The Installation

Other tips

• Follow recommendations in the Oracle Validated Linux configurations:

http://www.oracle.com/technology/tech/linux/validated-configurations/index.html

• Ensure DNS resolving works for hostnames, VIPs

• Use Cluster Verification Utility

• Bond the Interconnect

The Installation

Hugepages

• Translation Lookaside Buffer maps virtual -> physical memory

• Normal memory page size is 4K

• Hugepage size is 2MB resulting in more efficient memory access

• Method of moving tablespaces between databases

• Export metadata then copy the actual datafiles

• Plug in the metadata into the target database

• Transporting across platforms new in 10g

• A change of endianess means an rman convert is needed

The Migration

Transportable Tablespaces

• Check your platform is supported for transporting:

– sql> select * from V$TRANSPORTABLE_PLATFORM

PLATFORM_ID PLATFORM_NAME ENDIAN_FORMAT

----------- -------------------------------- --------------

1 Solaris[tm] OE (32-bit) Big

2 Solaris[tm] OE (64-bit) Big

3 HP-UX (64-bit) Big

4 HP-UX IA (64-bit) Big

5 HP Tru64 UNIX Little

6 AIX-Based Systems (64-bit) Big

7 Microsoft Windows IA (32-bit) Little

.

.

.

The Migration

Transportable Tablespaces

The Migration

Transport

• Must have a self-contained set of tablespaces to move

– sql> EXECUTE DBMS_TTS.TRANSPORT_SET_CHECK('TBNAME', TRUE);

– sql> select * from TRANSPORT_SET_VIOLATIONS;

• You must make the tablespaces to be moved read only

• Convert the endianess of the datafiles

– rman> convert tablespace TBNAME to platform 'Linux 64-bit for AMD’ format='/home/oracle/%U' ;

The Migration

Plugging In• Now you need the metadata:

– exp ‘as sys’ transport_tablespace=y file=tbname.dmp tablespaces=tbname

• Plug tablespace in as a “normal” filesystem tablespace:

– imp ‘as sys’ transport_tablespace=y file=‘path to export’ datafiles=‘path to filesystem datafile’

• Convert from filesystem datafile to ASM datafile:

– rman> convert datafile ‘path to filesystem datafile’ format ‘ASM PATH’

The Migration

The Plan

Solaris Linux

Rm

an

con

vert

Rm

an

con

vert

Copy datafiles

metadata

The Migration

A problem

• Started to notice space calculations incorrect

• Particular problem with 1 critical tablespace

• Assumed we had problems with our created types & custom indexing

• Clearing database down had catastrophic problem:

– sql> drop user TRANSPORTED cascade;

ERROR at line 1:

ORA-00600: internal error code, arguments: [ktecgetsh-inc], [2], [], [], [], [], [], []

The Migration

Oracle Response

The Migration

The cause of the problem

• The cause of the instance crashing was the following:

– ORA-00600: internal error code, arguments: [ktecgetsh-inc], [2], [], [], [], [], [], []

– Non-fatal internal error happened while SMON was doing temporary segment drop.

– Instance terminated by PMON

• Internal table SEG$ info wrong after TTS

• Fixed by patch 5186784

The Migration

Data Pump to the Rescue!• Could not wait for a fix to the Transportable Tablespace problems

- did not know how long it would take.

• Data Pump new in 10g - export/import on steroids

• Oracle claim Data Pump is 2x faster than export and 15-45 times faster than import

• Restartable with detailed job monitoring

• Automatic tuning and parallel server side execution

• direct import across the network

The Migration

Data Pump Architecture

Client Process

Client Process

Status Queue

Master table LOGMaster Control Process

Command and Control Queue

Worker Worker Dump file set

Data, Metadata & Master Table

The Migration

Data Pump Basics

• Create dumpdir:

– sql> create directory dumpdir as '/home/oracle/dumpdir’

– sql> grant read, write on directory dumpdir to USER;

• Actually do a Data Pump export:

– expdp user/password schemas=user directory=dumpdir dumpfile=output.dmp

• Do the Data Pump import:

– impdp user/password directory=dumpdir dumpfile=output.dmp

The Migration

Data Pump Issues

• Data Pump expdp did not run trouble free:

– ORA-31693: Table data object ”USER"."DOMAINS" failed to

load/unload and is being skipped due to error: ORA-06502: PL/SQL: numeric or value error

LPX-00210: expected '<' instead of '?'

• Apparently the xml/xsl stylesheets were not loaded into our 10.1 database

• The fix was running $Oracle_Home/rdbms/admin/initmeta.sql

The Migration

Data Pump Issues

• Data Pump impdp also was not trouble free:

– ORA-31693: Table data object ”USER"."DOMAINS" failed to load/unload and is being skipped due to error:ORA-44002: invalid object nameORA-06512: at "SYS.DBMS_ASSERT", line 283ORA-06512: at "SYS.KUPD$DATA", line 1336ORA-12801: error signaled in parallel query server P004,instance lr2:NOM2 (2)ORA-29913: error in executing ODCIEXTTABLEOPEN calloutORA-31626: job does not existORA-06512: at "SYS.ORACLE_DATAPUMP", line 19

• Seems like Data Pump import running in parallel across RAC nodes does not work. Shutdown of 2nd node cleared this problem

The Migration

Data Pump Issues

• We encountered even more problems during an import:

– ORA-02374: conversion error loading table ”USER"."INVOICES” ORA-12899: value too large for column CONTACT (actual: 162, maximum: 100)ORA-02372: data for row: CONTACT0X'C3A1C2B3CA98C3C5C28DC2A3C283C2A1C3A3C283C2B3C3'

• This time it was due to changing charactersetUS7ASCII -> UTF8

• Fortunately not too many rows affected so data munging was possible

The Migration

Data Pump Issues

• Tried playing with importing directly across network:

– ORA-31693: Table data object "SNOWDROP"."EMPLOYEE_OLE" failed to load/unload and is being skipped due to error:ORA-06502: PL/SQL: numeric or value error

• Seems like this is either a Bug:

– 3369197 - Dpimp : Network Mode Hr.Per_Addresses Ora-00997: Illegal Use Of Long Datatype Errors

– Metalink Note 313496.1

• Or maybe this is a feature - the above bug & Metalink Note are NOT publicly visible

The Migration

Final Data Pump Issues

• When we came to the day of the migration we hit this problem when exporting:

– expdp system parfile=param.exp

Export: Release 10.1.0.3.0 - 64bit Production on Sunday, 16July, 2006 10:10 Copyright (c) 2003, Oracle.

ORA-31626: job does not existORA-31637: cannot create job SYS_EXPORT_SCHEMA_01 for

user SYSTEM ORA-06512: at "SYS.DBMS_SYS_ERROR", line 95 ORA-06512: at "SYS.KUPV$FT_INT", line 601

ORA-12850: Could not allocate slaves on all specified instances: 2 needed, 1 allocated

• We managed to export 1 schema without issue but second schema was starting Parallel slaves on both nodes

The Migration

Data Pump Timings

• Use a parameter file:

– dumpfile=user.dmpdirectory=dumpdirparallel=4logfile=user.logschemas=user

• Some timings:

size

(GB)

Expdp

(time mins)

Impdp

(time mins)

network

(time mins)

35 11 55 75

49 13 95 240

Tips & Conclusions

ODBC Issue• Rule based Optimizer meant to be de-supported in 10g - see

Metalink Note: 189702.1

• ODBC driver was adding /*+ RULE */ hint to certain queries - still performing ok in 10.1.0.3

• /*+ RULE */ hinted queries in 10.2.0.2 were performing x 100 slower

• This is covered in Metalink Note:373129.1

• Added logon trigger for application to ignore optimizer hints _optimizer_ignore_hints

Tips & Conclusions

Post-Migration Issues

• There were quite a few niggling issues post migration

• perl-dbi applications silently dying

– update DBD driver to 1.17

• Pro*C applications silently dying

– update Oracle Client to 10.2

Tips & Conclusions

Post-Migration Issues

• Java JDBC problem:

– java.sql.SQLException: OALL8 is in an inconsistent state.

• Character set problem with domain registration system

– ORA-01756: quoted string not properly terminated

• Developers finding counting characters “harder”

– Can no longer guarantee 1 character == 1 byte

Tips & Conclusions

Migration Benefits

• Load on servers is greatly reduced

• Most ran query has gone from 1.6ms to 0.8ms

• RAC related waits decreased

• Stable for 6 months +

Tips & Conclusions

Conclusion

• TEST, TEST, TEST

• Transportable tablespaces is quickest way of moving data

– changing platforms really complicates & slows it down

• Data Pump is much, much faster than old export/import

– Roughly 25GB per hour

Questions?

All Change:The Story of a Migration

Contact:

• jason@nominet.org.uk

• http://blog.nominet.org.uk