Infrastructure Alternatives for Setting Up an Oracle … Alternatives for Setting Up an Oracle DB on...

Infrastructure Alternatives for Setting Up an Oracle DB on Linux

on IBM System z

Session 308

Tom [email protected]

IBM

Notice Regarding Specialty Engines (e.g., zIIPs, zAAPs and IFLs):

Any information contained in this document regarding Specialty Engines ("SEs") and SE eligible workloads provides only general descriptions of the types and portions of workloads

that are eligible for execution on Specialty Engines (e.g., zIIPs, zAAPs, and IFLs). IBM authorizes customers to use IBM SE only to execute the processing of Eligible Workloads of specific Programs expressly authorized by IBM as specified in the “Authorized Use Table for

IBM Machines” provided at www.ibm.com/systems/support/machine_warranties/machine_code/aut.html (“AUT”).

No other workload processing is authorized for execution on an SE.

IBM offers SEs at a lower price than General Processors/Central Processors because customers are authorized to use SEs only to process certain types and/or amounts of

workloads as specified by IBM in the AUT.

IBM trademark information can be found at Copyright and trademark information

Agenda• IBM System z basics

• Linux native or virtualized

• Oracle MAA on Linux on z

• Networking and MAA

• Disk or DASD considerations

• Linux implementation considerations

IBM SYSTEM z

IBM SYSTEM zHow big is a mainframe?

More information on zEnterprise

• IBM zEnterprise / System z Redbooks Portal: http://www.redbooks.ibm.com/portals/systemz

• IBM zEnterprise Announcement Landing Page: http://www-03.ibm.com/systems/z/hardware/zenterprise/zec12.html

• IBM System Storage: http://www-03.ibm.com/systems/storage/disk/index.html?LNK=browse

PR/SM – LPARsHardware partitioning

Servers

zEC12: 3 TB max, 60 LPARs max, 101 cores max

PR/SM – LPARsHardware partitioning

The IBM mainframe can be partitioned into separate logical computing systems. System resources (memory, processors, and I/O devices) can be divided or shared among many such independent logical partitions(LPARs) under the control of the LPAR hypervisor, which comes with the standard ProcessorResource/ Systems Manager (PR/SM™) feature on all mainframes. The hypervisor is a software layer to manage multiple operating systems running in a single central processing complex. The mainframe uses a Type 1 hypervisor. Each LPAR supports an independent operating system (OS) loaded by a separate initial program load (IPL) operation.

The System z* PR/SM* is designed for Common Criteria Evaluation Assurance Level 5+ (EAL 5+) certification for security, so an application running on one partition can’t access another application on a different partition,

z/VM - virtualization

Comments:- Think of z/VM as a big city. Parts of the city can be allocated to guests. Some guests live in skyscrapers

and some in single family homes.

- Guest are given access to IFLs, memory, and I/O AS NEEDED

- Unused resources can be made available to all and that increases efficiency through complete utilizationof resources

z/VM sample Oracle Guest• Suppose the System z has

64 GB of memory, 20 IFLs, and dozens of I/O connections

• HR production guest4 GB for SGA and PGA so we make it a 6 GB guest Sized for four IFLs. Can use any four of the 20 available IFLsPrioritize this guest in z/VM relatively high.

• HR test/dev/QA/sandbox10 GB for multiple instances of Oracle so we make it a 12 GB guestSized for 6 IFLs. Can use any six of 20 available IFLsPrioritize below HR production guest so in times of high demand HR productiongets preferred access.

• Can run many other Oracle or non-Oracle guests sharing the IFL and memoryresources.

Don’t call it zLinux – It is really Linux on z

zLinux is a commonly used term for running Linux on z

zLinux is NOT a special distribution of Linux created by IBM.

SLES and Red Hat have Oracle certified distributions for Loz

SLES 10 and 11

Red Hat 5.x and Red Hat 6.2

Linux can run natively on System z or under z/VM (i.e. virtualization)

Linux on z (Loz) is a better term and is less confusing in the Oracle space

Linux on System z

Choices

• z/VM – in one or more LPARs• Production and Dev/Test in different LPARs each with z/VM• Production, Dev/Test in the same LPAR

• Linux in an LPAR natively

• Linux in LPAR(s) with z/VM in different LPARs• z/VM hosting Dev/Test• LPAR mode for Production Linux

Basic Guest Infrastructure

Implementing Oracle DB and RAC

Oracle database Linux for System z

© 2011 IBM Corporation

Oracle Database - building Oracle MAA

Node 1

Linux

z/VM LPAR 1

ApplicationServer(s)

Oracle Database EE11gR2 or 10gR2

Guards against:•Hardware failure – System z

Comments:•Added Oracle’s Automated Storage Manager (ASM) which is similar to a LVM but with more capabilities

•ASM is a separate Oracle DB but is not shown

A

S

M

VSWITCH

Oracle DB Instance

MAA – Oracle’s Maximum Availability Architecture



Oracle Database - building Oracle MAA – no RAC

Node 1

Linux

z/VM LPAR 1


Oracle Databases

Guards against:•Hardware failure – System z

Comments:•The solution can scale with multiple virtual Linux guests, each with an Oracle database instance as well as running multiple instances within a single guest.

A

S

MVSWITCH

C

R

S

Oracle DB Instance

Node 1

C

R

S

A

S

M

Node 1

C

R

S

Oracle DB Instance

A

S

M

Node 1

C

R

S

Linux

Oracle DB Instance

A

S

M

Node 1

C

R

S

Linux

Oracle DB Instance

A

S

M

Node 1

C

R

S

Linux

Oracle DB Instance

A

S

M

Node 1

C

R

S

Linux

Oracle DB Instance

A

S

M

Node 1

C

R

S

Linux

Oracle DB Instance

A

S

M

Node 1

C

R

S

Linux

Oracle DB Instance

A

S

M

Node 1

C

R

S

Linux

Oracle DB Instance

A

S

M

C

R

S


ApplicationServer(s)Application

Server(s)ApplicationServer(s)



Oracle Database building Oracle MAA -

Prod

Linux

z/VM LPAR 1


Oracle Database

Guards against:•Hardware failure – System z•Linux OS or Oracle DB failure•Allows for rolling maintenance to Linux and possibly Oracle.

Comments:•Unlike hot stand by there will be little impact to the end users resulting from a Linux node or instance failure.

•All IFLs and memory can be shared across nodes.

A

S

M

&

C

R

S

Prod

Linux A

S

M

&

C

R

S

VSWITCH

Private interconnect

RAC

Oracle DB Instance

Oracle DB Instance

Interconnect - The private network communication link that is used to synchronize the memory cache of the nodes in the cluster. Can be a VSWITCH, Linux channel bonding or Oracle HAIP.



Oracle Database - building Oracle MAA – RAC

Prod

Linux

z/VM LPAR 1


Oracle Database

Guards against:•Hardware failure –System z•Linux OS or Oracle DB failure•Allows for maintenance to either z/VM, Linux and possibly Oracle DB in either Prod guest

A

S

M

&

C

R

S

Prod

Linux A

S

M

&

C

R

S

VSWITCH

z/VM LPAR 2

VSWITCH

HiperSockets

Oracle DB Instance

Oracle DB Instance

PrivateInterconnect

Comments:•IFLs can be shared across LPARs but memory cannot.

•HiperSockets is a memory based technology that provides high-speed TCP/IP connectivity between LPARS within a System z.



Oracle Database - building Oracle MAA – RAC

Prod

Linux

System z #1 – z/VM


Oracle Database

Guards against:•Hardware failure –System z•Linux OS or Oracle DB failure•Allows for maintenance to either z/VM, Linux and possibly Oracle DB in either Prod guest

Comments:•Physically separate System z machines

A

S

M

&

C

R

S

Prod

Linux A

S

M

&

C

R

S

VSWITCH

System z #2 – z/VM

VSWITCH

OSA and TCP/IP

Oracle DB Instance

Oracle DB Instance

PrivateInterconnect

Networking


Focusing on z/VM capabilities

z/VM Virtual Switch

•A special-purpose Guest LAN

• Layer 2 or Layer 3• Built-in IEEE 802.1q bridge to outside network• IEEE VLAN capable• IEEE 802.3ad - Link Aggregation

•Each Virtual Switch has up to 8 separate OSA-Express connections associated with it




Node 1

Linux

z/VM LPAR 1A

S

MVSWITCH

C

R

S

Oracle DB Instance

Node 1

C

R

S

A

S

M

Node 1

C

R

S

Oracle DB Instance

A

S

M

Node 1

C

R

S

Linux

Oracle DB Instance

A

S

M

Node 1

C

R

S

Linux

Oracle DB Instance

A

S

M

C

R

S

Linux

Oracle DB Instance

A

S

M

Node 1

C

R

S

Linux

Oracle DB Instance

A

S

M

Node 1

C

R

S

Linux

Oracle DB Instance

A

S

M

Node 1

C

R

S

Linux

Oracle DB Instance

A

S

M

Node 1

C

R

S

Linux

Oracle DB Instance

A

S

M

C

R

S

OSA10 GbE

OSA10 GbE vNICs in

each guest

vNIC

Comments:- OSA is a physical

network port

- vNIC – virtual NIC defined in z/VM mapping to a NIC definition in Linux

- There can be many VSWITCHes defined and running in z/VM




Node 1

Linux

z/VM LPAR 1A

S

MVSWITCH

C

R

S

Oracle DB Instance

Node 1

C

R

S

A

S

M

Node 1

C

R

S

Oracle DB Instance

A

S

M

Node 1

C

R

S

Linux

Oracle DB Instance

A

S

M

Node 1

C

R

S

Linux

Oracle DB Instance

A

S

M

C

R

S

Linux

Oracle DB Instance

A

S

M

Node 1

C

R

S

Linux

Oracle DB Instance

A

S

M

Node 1

C

R

S

Linux

Oracle DB Instance

A

S

M

Node 1

C

R

S

Linux

Oracle DB Instance

A

S

M

Node 1

C

R

S

Linux

Oracle DB Instance

A

S

M

C

R

S

OSA10 GbE

vNICs ineach guest

vNICOSA

10 GbE

Comments:- Additional Network

CapacityUsing VSWITCH IEEE 802.3ad Link Aggregation



Oracle Database building Oracle MAA – in one z LPAR

Linux

z/VM LPAR 1A

S

M

&

C

R

S

LinuxA

S

M

&

C

R

S

VSWITCH

Private interconnect

RAC

Oracle DB InstanceNode 1


VSWITCHor

dedicated guest lan

vNIC - vlan

vNIC

vNIC

OSA10 GbE

OSA10 GbE

vNIC - vlan

Comments:- VSWITCH supports vlan

tagging

- vlan tagging occurs inside Linux networkdefinition



Oracle Database - building Oracle MAA – RAC in separate z LPARs

Linux

z/VM LPAR 1A

S

M

&

C

R

S

LinuxA

S

M

&

C

R

S

VSWITCH

z/VM LPAR 2

VSWITCH



HiperSocketPrivate interconnect

HiperSocket

vNIC

vNIC - vlan

vNIC

OSA10 GbE

OSA10 GbE

OSA10 GbE

OSA10 GbE

vNIC - vlan

Comments:- HiperSockets is a

memory to memory connection

- When using Hipersockets disable TCP/IP checksum

- Increase MTU size for better performance

- HiperSockets supports vlan



Hipersockets Checksumming Disable

HiperSockets does not require network checksum since it is a memory to-memory operation.

– To save CPU cycles, switch checksumming off:SUSE SLES10: in /etc/sysconfig/hardware/hwcfg-qeth-bus-ccw-0.0.F200 add QETH_OPTIONS="checksumming=no_checksumming"

SUSE SLES11: in /etc/udev/rules.d/51-qeth-0.0.f200.rules addACTION=="add", SUBSYSTEM=="ccwgroup", KERNEL=="0.0.f200",ATTR{checksumming}="no_checksumming“

Red Hat: in /etc/sysconfig/network-scripts/ifcfg-eth0 addOPTIONS="checksumming=no_checksumming"

27



Oracle Database - building Oracle MAA – RAC in separate z machines

Linux

z/VM LPAR 1A

S

M

&

C

R

S

LinuxA

S

M

&

C

R

S

VSWITCH

z/VM LPAR 2

VSWITCH



vNIC

vNIC - vlan

vNIC

OSA10 GbE

OSA10 GbE

OSA10 GbE

OSA10 GbE

VSWITCHOSA

10 GbE

VSWITCHOSA10 GbE

vNIC - vlan

Comments:- Use VSWITCH for

multiple guests running RAC private interconnections

Disk or DASD



Oracle database on Linux for System z


Disk or DASD options

Disk: XIV, V7000 or SCSI- For the Oracle database

DASD (3390 or ECKD)- Use for z/VM, Linux and Oracle binaries- For Oracle database

• PerformanceBest – HyperPav in the DASD subsystem and HyperPav driver

support in Linux distributionSLES 11 update 1REL 6

Good – HyperPAV or PAV in the DASD subsystem. Additional setup work in z/VM and Linux required

Decent – No PAV. Setup work in z/VM and Linux required

HyperPAV

0 10 20 400%

20%

40%

60%

80%

100%

120%

140%

160%

0

20

40

60

80

100

120

ECKD Devices: Scaling HyperPAV aliases

Normalized Transactional throughput and total Disk I/O (read + write)

normalized transactional throughput Total MB/sec

# alias devices

norm

aliz

ed tr

ansa

ctio

nal t

hrou

ghpu

t

MiB

/sec

Applies only to DASD and not SCSI

4 CPUs 6 CPUs0%

20%

40%

60%

80%

100%

120%

140%

160%

Comparing FCP and ECKD

Transactional throughput

ECKD (20 aliases) FCP (rr_min_io=100)

norm

aliz

ed tr

ansa

ctio

nal t

hrou

ghpu

t

ECKD / FCP Comparison

ECKD / FCP comparison - 2

4 CPUs 6 CPUs90%

95%

100%

105%

110%

115%

120%

Comparing FCP and ECKD

CPU cost per transactional throughput

FCP ECKD

CP

U c

ost p

er tr

ansa

ctio

nal t

hrou

ghpu

t

ECKD / FCP comparison - 3

• FCP (SCSI) offers better throughput and performance

• ECKD uses less CPU per transaction

• You have to tune both environments

• Recommendation: it depends

3



Overview of Major RAC Components

RAC

Node_1

RAC

Node_2

Voting

OCR

Datafile

Datafile

Datafile

VIPA -- Backup

Private Network -- PrimaryPublic Network

$Oracle

Swap

Linux

Product Binaries

$Oracle

Swap

Linux

Product Binaries

Position inside ASM for 11gR2



Verify I/O Performance with Oracle Orion/CalibrateOracle ORION utility - I/O simulator simulates reads and writes without creating a database.Now included with Oracle install code.Get your I/O subsystem tuned ahead of time.

$ORACLE_HOME/bin/orion -run oltp -testname test -num_disks 5 -duration 20 -simulate raid0

ORION: ORacle IO Numbers -- Version 11.2.0.3.0test_20130312_1652Calibration will take approximately 9 minutes.Maximum Small IOPS=41535 @ Small=26 and Large=0

Existing database you can use Oracle’s I/O Calibrate to test your I/O performance:

SET SERVEROUTPUT ONDECLARE

lat INTEGER;iops INTEGER;mbps INTEGER;

BEGIN-- CALIBRATE_IO (<DISKS>, <MAX_LATENCY>, iops, mbps,

lat);

DBMS_RESOURCE_MANAGER.CALIBRATE_IO (2, 10, iops, mbps, lat);

DBMS_OUTPUT.PUT_LINE ('max_iops = ' || iops);DBMS_OUTPUT.PUT_LINE ('latency = ' || lat);dbms_output.put_line('max_mbps = ' || mbps);

end;/

max_iops = 45386latency = 0max_mbps = 1236

PL/SQL procedure successfully completed.



Storage – Testing with Oracle Orion – Bad result

ORION Simulates Oracle reads and writes, without having to create a databaseand helps to isolate I/O issues. When a database is optimally configured you can expect to get up to 95% of the thorughput of Orion.

./orion_zlinux -run simple -testname mytest -num_disks 2 -duration 30 -simulate raid0

ORION VERSION 11.2.0.0.1Commandline: -run oltp -testname mytest -num_disks 2 -duration 30 -simulate raid0This maps to this test: Test: mytestSmall IO size: 8 KB Large IO size: 1024 KBIO Types: Small Random IOs, Large Random IOsSimulated Array Type: RAID 0 Stripe Depth: 1024 KBWrite: 0% Cache Size: Not EnteredDuration for each Data Point: 30 secondsSmall Columns:, 2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26,

28, 30, 32, 34, 36, 38, 40Large Columns:, 0 Total Data Points: 22

Name: /dev/dasdq1 Size: 2461679616Name: /dev/dasdr1 Size: 24616796162 FILEs found.Maximum Small IOPS=5035* @ Small=40 and Large=0Minimum Small Latency=0.55 @ Small=2 and Large=0

* Results are NOT representative of normal DASD performance



I/O Subsystem – Testing with Oracle Orion

Be careful with the Orion options you choose. The writes are destructive.

Perform Orion testing immediately after installing the Oracle database to validate the I/O subsystem

Moving data is like moving water – must have adequate flow throughout

OR



AWR – I/O statistics

Oracle’s Orion

• Can be used on both Disk and DASD

• If you have both Disk and DASD can help you decide which to use for the Oracle Database(s)

Linux ImplementationRed Hat or SLES


DASD or Disk

• Use Linux multipathing for SCSI

• Not required for DASD with Red Hat 6 or SLES 11 SP2 and HyperPAV

• z/VM handles pathing automatically

• If using a Linux Logical Volume Manager (LVM)

• Disable read ahead on Oracle volumes or LUNs



Linux HugePagesConsider implementing Linux HugePages for large SGAs

(Oracle DB 11gR2 only)- The more connections the larger the page table reduction.- Limitations: No Oracle’s AMM (i.e., memory_target).

Notice the HugePages size differs among different version of Linux

- Red Hat 5.x and SLES 10 use 2 MB page increments- Red Hat 6 and SLES 11 use 1MB page increments

See 1392497.1 on support.oracle.com for more details.



Oracle DB without HugePages Linux 4K Page Tables after 70 minutes

Linux Swap and Page Tables using 87.7 GB of Memory!



Oracle using HugePages after two hours

Page tables: 0.371 GB and No Swap

cat /proc/meminfo

Oracle Database Implementation11gR2


Utilize Direct I/O with LVM

• Eliminate Linux caching allowing Oracle DB to properly stage data

• init.ora

• filesystemio_options=setall*

• disk_asynch_io=true

*Not necessary with Oracle’s ASM


Oracle 11.2.0.3 Improvements

Oracle’s VKTM (virtual keeper of time) process uses slightly fewer CPU minutes – (about 0.08 vs. 0.09 with 11.2.0.2)

Only Install the database modules that are needed– DB installed with NO options

The "gettimeofday" function is called 300 times every 15 seconds.

– DB installed with all options : (Java, XML, Text, etc ....... ) The "gettimeofday" function is called 1500 times every 15 seconds.

My Oracle Support LinksNote 1306465.1 Getting Started - 11gR2 Grid Infrastructure, SI(Single Instance), ASM and DB (IBM: Linux on System z)Note 1470834.1 - Requirements for Installing Oracle 11gR2 on RHEL 6 on IBM: Linux on System z (s390x)Note 1290644.1 - Requirements for Installing Oracle 11gR2 on SLES11 on IBM: Linux on System z (s390x) Also review note: OHASD fails to start on SuSE 11 SP2 on IBM: Linux on System z [ID 1476511.1]Note 1308859.1 Requirements for Installing Oracle 11gR2 on SLES 10 on IBM: Linux on System z (s390x)Note 1306889.1 Requirements for Installing Oracle 11gR2 on RHEL 5 on IBM: Linux on System z (s390x)Note 1086769.1 - Ensure you have prerequisite rpms to install Oracle Database and AS10g(midtier) on IBM: Linux on System z (s390x) Note 1377392.1 How to Manually Configure Disk Storage devices for use with Oracle ASM 11.2 on IBM: Linux on System z)Note 1400185.1 How to Upgrade Oracle Restart i.e. Single Node Grid Infrastructure/ASM from 11.2.0.2 to 11.2.0.3Note 1276058.1 Oracle GoldenGate Best Practices: Instantiation from an Oracle Source DatabaseNote 1413787.1 How to completely remove 11.2 Grid Infrastructure, CRS and/or Oracle Restart - IBM: Linux on System z

Note 259301.1 CRS and 10g Real Application ClustersNote 268937.1 Repairing or Restoring an Inconsistent OCR in RACNote 239998.1 10g RAC How to clean up after a failed CRS InstallNote 220970.1 RAC Frequently Asked Questions Topic

Note 420382.1 Requirements for Installing Oracle 10gR2 RDBMS on RHEL 4 on zLinux (s390x).Note 431443.1 Requirements for Installing Oracle 10gR2 RDBMS on SLES 9 zLinux (s390x)Note 1082253 Requirements for Installing Oracle 10gR2 RDBMS on SLES 10 zLinux (s390x)Note 741646.1 Requirements for Installing Oracle 10gR2 RDBMS on RHEL 5 on zLinux (s390x).Note 415182.1 DB Install Requirements Quick Reference - zSeries based Linux .

Note 741146.1 Installing Standalone Agent 10.2 on Linux on z

Other Resourceshttp://www.ibm.com/redbooks- SG24-6482-00 Experiences with Oracle Database 10g on Linux for zSeries- SG24-7191-00 Experiences with Oracle 10gR2 Solutions on Linux for System z- SG24-7573-00 Using Oracle Solutions on Linux on System z- SG24-7634-00 Experiences with Oracle Solutions on Linux for IBM System z- REDP-4788-00 Installing Oracle 11gR2 RAC on Linux on System z

http://linuxmain.blogspot.com

http://www.vm.ibm.com/perf/tips- General z/VM Tuning Tips

http://www-124.ibm.com/developerworks/oss/linux390/index.shtml- Lot’s of information on Linux for System z

http://www-128.ibm.com/developerworks/linux/linux390/perf/index.html- Hints and Tips for tuning Linux on System z

http://www.zseriesoraclesig.org- Special Interest Group of Oracle users on the System z

http://www.mail-archive.com/linux-390%40vm.marist.edu/- Marist List Server

Questions? Comments?

Trademarks and special notices © Copyright IBM Corporation 2012. All rights reserved. References in this document to IBM products or services do not imply that IBM intends to make them available in every country. IBM, the IBM logos are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both: Java and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both. Microsoft, Windows, Windows Server, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both. Intel and Xeon are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries. AMD and AMD Opteron are trademarks of Advanced Micro Devices, Inc. Red Hat, the Red Hat "Shadow Man" logo, and all Red Hat-based trademarks and logos are trademarks or registered trademarks of Red Hat, Inc., in the United States and other countries. UNIX is a registered trademark of The Open Group in the United States and other countries. Linux is a trademark of Linus Torvalds in the United States, other countries, or both. Other company, product, or service names may be trademarks or service marks of others. The information provided in this document is distributed “AS IS” without any warranty, either express or implied. The information in this document may include technical inaccuracies or typographical errors. All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics may vary by customer. Information concerning non-IBM products was obtained from a supplier of these products, published announcement material, or other publicly available sources and does not constitute an endorsement of such products by IBM. Sources for non-IBM list prices and performance numbers are taken from publicly available information, including vendor announcements and vendor worldwide homepages. IBM has not tested these products and cannot confirm the accuracy of performance, capability, or any other claims related to non-IBM products. Questions on the capability of non-IBM products should be addressed to the supplier of those products. All statements regarding IBM future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only.

Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners.

Infrastructure Alternatives for Setting Up an Oracle … Alternatives for Setting Up an Oracle DB on...

Documents

Transcript of Infrastructure Alternatives for Setting Up an Oracle … Alternatives for Setting Up an Oracle DB on...