© 2010 Quest Software, Inc. ALL RIGHTS RESERVED.

Optimize Oracle on VMware

Tips for Optimal Virtualized Oracle Databases

Bert Scalzo and Guy Harrison

2

Bert Scalzo …Database Expert & Product Architect for Quest Software

Oracle Background:• Worked with Oracle databases for over two decades (starting with version 4)• Work history includes time at both “Oracle Education” and “Oracle Consulting”

Academic Background:• Several Oracle Masters certifications• BS, MS and PhD in Computer Science• MBA (general business)• Several insurance industry designations

Key Interests:• Data Modeling• Database Benchmarking• Database Tuning & Optimization• "Star Schema" Data Warehouses• Oracle on Linux – and specifically: RAC on Linux

Articles for:• Oracle’s Technology Network (OTN)• Oracle Magazine,• Oracle Informant• PC Week (eWeek)

Articles for:• Dell Power Solutions

Magazine• The Linux Journal• www.linux.com• www.orafaq.com

3

Books by Bert …

Coming in 2009 …

Guy vs. Bert

• Similarities– Also a Quest professional (Director

of R&D, creator of Spotlight, etc)– Also a Prentice Hall author (Oracle

Performance Survival Guide and others)

– Also many (20+) years experience in Oracle

• Dissimilarities– Less experienced (no Oracle 4

experience!)– Australian accent – Lack’s Bert’s Texas charm

Couple of Questions…

• How many people are doing Oracle on– VMware Workstation only– VMware ESX or ESXi – Other Virtual Platforms

– Oracle VM– Other Xen Server– Hyper-V– OS-specific Hypervisor (Sun, AIX )

– How many Toad users?

Agenda• Motivations for Virtualization

• VMware ESX resource management:

– Memory

– CPU

– IO

• Optimization tips and techniques

Motivations for Virtualization

Resistance to Database virtualization

Fundamental ChallengeUnderstanding the impact of resource sharing

CPUMulti-Core Processors

NetworkShared Connectivity

Load Balancing

Redundancy

MemorySharing

Over-commitment

DiskShared storage

Fiber, iSCSI, NAS

Core 4 Resources

Configuring VM memory

Relative Memory Allocation for this VMMinimum Memory for this VM

VM Competes for memory in this range

Maximum memory for the VM (dynamic)

Monitoring VM memory

ESX Swap

ESX virtual memory

ESX physical memory

VM VM virtual memory

ESX swap

Effective VM physical memory

ESX and VM memory

ESX Swap

ESX virtual memory

ESX physical memory

VM

Apparent VM physical memory

ESX swap

Effective VM physical memory

Vmmemctl

“balloon”

VM SwapVM Swap

ESX Ballooning

ESX Ballooning• As memory grows, ESX balloon driver (vmmemctl) forces VM to page

out memory to VM swapfile

ESX Ballooning

• Inside the VM, paging to the swapfile is observed.

• The guest OS will determine which pages are paged out

• If LOCK_SGA=TRUE, then the SGA should not be paged.

ESX Swap

ESX virtual memory

ESX physical memory

VM Apparent VM physical memory

ESX swap

Effective VM physical memory

ESX Swapping

ESX Swapping

• ESX swaps out VM memory to ESX swapfile

ESX Swapping

• Within the VM, swapping cannot be detected.

• ESX will determine which memory pages go to disk

• Occurs when VMware tools are not installed

• Even if LOCK_SGA=TRUE, SGA memory might be on disk

Avoiding Ballooning and swapping

• memory reservations help avoid ballooning or ESX swapping

VMware memory recommendations

• Paging or swapping of PGA or SGA is almost always a Very Bad ThingTM.

• Use memory reservations to avoid swapping• Install VMware tools to avoid “invisible” swapping• Set Memory reservation = PGA+SGA+process Overhead• Be realistic about memory requirements:

– In physical machines, we are used to using all available memory– In VM, use only the memory you need, freeing up memory for other

VMs– Oracle advisories (or Spotlight) can show you how much memory is

needed– Reduce VM reservation and Oracle memory targets in tandem to

release memory

VMware CPU management

• If more virtual CPUs than ESX CPUs, then VCPUs must sometimes wait.

• Time “stops” inside the VM when this occurs (for multi-CPU VMs, it’s all or nothing).

• A VCPU can be in one of three states:– Associated with an ESX CPU but idle– Associated with an ESX CPU and executing instructions– Waiting for ESX CPU to become available

• Shares and reservations determine which VM wins access to the ESX CPUs

Configuring VM CPU

VMs compete for CPU in this range

Shares determine relative CPU allocated when competing

• “CPU Ready” is the amount of time VM spends waiting on ESX for CPU

• Inside the VM, CPU stats can be misleading

CPU utilization VM

Spotlight Virtual CPU monitoring

VM CPU Overhead

• CPU intensive workloads can show significant degradation even without contention from other VMs

Physical machine was dual core 1.8GHz. ESX Server was dual core 3.5 GHz with single 2 CPU VM

CPU intensive workloads

• Latch contention may be exacerbated.

Disclaimer: Quick and simple workload measurements, not benchmarks; no disk activity; no para-virutualization: Your mileage will vary.

VM CPU Overhead

VM had exclusive access to the ESX server – no “ready” waits.

No “spare” ESX CPU to carry ESX overhead

VMware CPU Summary

• Use reservations and shares to prioritise access to ESX CPU

• Performance of CPU critical workloads may be disappointing regardless of CPU reservations or shares.

• Monitor ESX Ready time to determine the “penalty” of competing with other virtual machines

Typical VMWare disk configuration

Disk Resource Allocation

• Disk shares can be used to prioritize IO bandwidth.

• However, ESX often does not know underlying storage architecture

Performant VMware disk configuration

Optimal configuration

• See “Oracle Database Scalability in VMware® ESX” at www.vmware.com/oracle

• Each virtual disk directly mapped via RDM to dedicated RAID 0 (+1) group

41 Spindles!

VMWare disk configuration

• Follow normal best practice for physical disks• Avoid sharing disk workloads • Consider Raw Device Mapping (RDM)• If not, consider disk Shares• If you can’t optimize IO, avoid IO:

– Tune, tune, tune SQL– Prefer indexed paths– Memory configuration– Don’t forget about temp IO (sorts, hash joins)

Berts toolbox…

• Not pitching or selling anything here – just disclosure

• There are lot’s of good tools out there - pick whatever

• In brief – here’s what I used and why (its purpose):

– Load Generator = Benchmark Factory for Databases– Virtualization Monitor = Foglight for VMware

– Database Performance = Spotlight for Oracle & Foglight

Foglight for VMwareMonitor Shared Resources

Find & FixGenuine VMPerformance Issues

Help to SqueezeMaximum“Overall”Throughput

Benchmark Factory for DatabasesGenerate Database Workload

Stress theDatabase IntroducePerformance Issues

Help to ExposePotentialCandidates

Each step above is an individual test for my proposed best practices – and their results

440% Improvement for Proper Setup

1080 msAverage ResponseTime

200 msAverage ResponseTime

Bert’s Top Ten Tricks …• Obtain a baseline test for relative comparisons• On the VM host, exclude VM clients from active, online virus scans• Remove Windows Indexing Service (because really don’t need fast

file system searches for an Oracle database)• Remove other extraneous Windows services (there are lots)• Change the VM host registry settings to improve file system IO for

databases• Optimize the VM host configuration and options• Optimize the VM client OS configuration and options for Oracle

database• Remove other extraneous VM client OS services and daemons• Change VM client file system settings to improve IO performance

for databases• Adjust VM client file system block size to more closely match

Oracle block size

Guy’s top ten tips

• Use a tuning methodology (tuning by layers)• Design with performance in mind• Index wisely• Know your tools• Optimize the optimizer• Tune SQL and PL/SQL• Minimize contention• Optimize memory• Tune IO last, but tune it• Exploit and tune RAC

See www.toadworld.com article: http://bit.ly/9lnxyy

Tuning by layers

Problems in one database layer can be caused or cured by configuration in the higher layer. Therefore:

2.Reduce application demand to its logical minimum by tuning SQL and PL/SQL, and optimizing physical design (partitioning, indexing, etc).

3.Maximize concurrency by minimizing contention for locks, latches, buffers, etc.

4. Minimize amount of physical IO by optimizing Oracle memory.

5.Reduce IO latency by providing adequate IO bandwidth and evenly distributing the resulting load.

Database Code Layer:Oracle software parses & optimizes SQL, manages

locks, security, concurrency , etc

Application Layer: Application code issues SQL (and PL /SQL) requests

to database

Memory layer:

Buffer cache (data blocks)Other shared memory caches

PGA (sorting and hash memory )

Disk layer :

Read/write table/index data Read/write temporary work areas

Redo and other log IO

Presenter:Bert Scalzo: Bert.Scalzo@Quest.com

www.bertscalzo.com

Guy Harrison: guy.harrison@quest.comwww.guyharrison.net

www.twitter.com/guyharrison

Questions and Answers …

Thank You

© 2010 Quest Software, Inc. ALL RIGHTS RESERVED.

End of Presentation

너를 감사하십시요 Thank You Danke Schön

Gracias 有難う御座いました Merci

Grazie Obrigado 谢谢