Optimizing Oracle on VMware

7/31/2019 Optimizing Oracle on VMware

1/10

Optimizing Oracle on VMware

Bert Scalzo PhD & Oracle ACE

Quest Software: Database Expert

Intro

Many people swore that databases on virtual machines would not fly orat best, would be accepted very, very slowly. I was not one of thoseskeptics so I wrote a book on just that subject: Oracle on VMware:Expert Tips for Database Virtualization.

http://www.rampant-books.com/book_2008_1_oracle_vmware.htm

However I also dont expect people in these tough economic times torush out to buy every new book, so this paper will attempt to provide avery brief synopsis of Oracle on VMware books overall thesis and somehigh value proposition recommendations.

Other Sources

One of the first and most robust papers Ive seen on this subject is from VMware itself:

Deployment of Oracle Databases on VMware Infrastructure.

http://www.vmware.com/partners/alliances/technology/oracle-database.html

This fine paper is 119 pages of highly useful DBA information for both effectively andefficiently deploying Oracle databases on a VMware infrastructure.

At first Oracle was a little slow to fully embrace the new concept of databases on virtualmachines. But then they dbuted their own virtual machine solution and things quickly

changed. There are now papers that showbenchmarks of virtual machines versus bare

metal that are achieving acceptable results.

http://www.oracle.com/technologies/virtualization/docs/ovmbenchmark.pdf

There are now even papers for successfully deploying RAC (Oracle Real ApplicationClusters) on virtual machines.

http://www.oracle.com/technology/products/database/clusterware/pdf/oracle_rac_in_oracle_vm_environments.pdf
http://www.rampant-books.com/book_2008_1_oracle_vmware.htmhttp://www.rampant-books.com/book_2008_1_oracle_vmware.htmhttp://www.rampant-books.com/book_2008_1_oracle_vmware.htmhttp://www.vmware.com/partners/alliances/technology/oracle-database.htmlhttp://www.vmware.com/partners/alliances/technology/oracle-database.htmlhttp://www.oracle.com/technologies/virtualization/docs/ovmbenchmark.pdfhttp://www.oracle.com/technologies/virtualization/docs/ovmbenchmark.pdfhttp://www.oracle.com/technologies/virtualization/docs/ovmbenchmark.pdfhttp://www.oracle.com/technology/products/database/clusterware/pdf/oracle_rac_in_oracle_vm_environments.pdfhttp://www.oracle.com/technology/products/database/clusterware/pdf/oracle_rac_in_oracle_vm_environments.pdfhttp://www.oracle.com/technology/products/database/clusterware/pdf/oracle_rac_in_oracle_vm_environments.pdfhttp://www.oracle.com/technology/products/database/clusterware/pdf/oracle_rac_in_oracle_vm_environments.pdfhttp://www.rampant-books.com/book_2008_1_oracle_vmware.htmhttp://www.rampant-books.com/book_2008_1_oracle_vmware.htmhttp://www.rampant-books.com/book_2008_1_oracle_vmware.htmhttp://www.vmware.com/partners/alliances/technology/oracle-database.htmlhttp://www.vmware.com/partners/alliances/technology/oracle-database.htmlhttp://www.oracle.com/technologies/virtualization/docs/ovmbenchmark.pdfhttp://www.oracle.com/technologies/virtualization/docs/ovmbenchmark.pdfhttp://www.oracle.com/technologies/virtualization/docs/ovmbenchmark.pdfhttp://www.oracle.com/technology/products/database/clusterware/pdf/oracle_rac_in_oracle_vm_environments.pdfhttp://www.oracle.com/technology/products/database/clusterware/pdf/oracle_rac_in_oracle_vm_environments.pdfhttp://www.oracle.com/technology/products/database/clusterware/pdf/oracle_rac_in_oracle_vm_environments.pdfhttp://www.oracle.com/technology/products/database/clusterware/pdf/oracle_rac_in_oracle_vm_environments.pdf


2/10

So the trend winds have changed, virtualization is no longer verboten for databases

So Lets Begin

At Oracle Open World 2008, I presented some materials that show that deploying Oracleon VMware with default configuration and settings across the board could cost you as

much as 440% in terms of achievable database server performance. However, we mustapproach configuration, optimization and tuning in a slightly different manner because

we must now fully consider the four core, shared resources shown here more deeply:

This is really nothing new DBAs have always cared about the CPU, memory, disk andnetwork bandwidths versus utilization. But generally speaking in the past, DBAs tended to

have one database instance per physical server thus they historically didnt worry too

much about shared resource consumption. Of course with SAN, NAS and iSCSI shared

storage servers DBAs began to feel, spot and optimize around such shared resources.

Virtualization merely extends that same paradigm across all of the key component layers

that the database engine relies upon so we must now account and accommodate for that.Suppose that we have a SQL Server and Oracle database sharing the same resources and

running radically different workloads concurrently. We must now then setup, configure,

optimize and tune everything with the assumption that every resource of value is sharedand thus not ours alone. This means that there will be even less room for error than ever.


3/10

What Was Tested

Of course the very best possible workload to test with would be that from production which

one needs to deploy onto a virtualized server. For the sake of this paper, I simply needed tochoose something that a general readership population might either already know or at least

easily understand. Therefore I chose the TPC-C benchmark a well-known and fairlyrespected industry standard OLTP benchmark (although soon to be replaced by the newer

TPC-E benchmark). I ran the TPC-C benchmark for 50 to 300 concurrent users and not

larger due to my hardware size limitations. My goal was to meet a Service Level

Agreement (SLA) of no transaction taking more than 2 seconds, and to determine just howmany concurrent users my server could ideally accommodate.

Luckily I work for a software vendor (i.e. Quest Software) so its easy for me to obtaintools to make this testing and optimization effort trivial. I dont mention these tools as a

sales pitch, but rather merely to inform you as to what I used. There are of course severalother tools out there that offer similar capabilities. So feel free to utilize those that best suityour needs, budget and/or preferences. Again, I only list these so that you know how I

performed all of these tests. The 440% improvement results would have probably been

also fairly easily obtained using different tools because its really the concepts here thatapply, and not the tools per se used to arrive at and/or derive them.

1. Load Generator = Benchmark Factory for Databases

2. Virtualization Monitor = Foglight for VMware

3. Database Performance = Foglight Database Performance Analysis

4. Database Ad-Hoc Diagnostics = Spotlight for Oracle

So returning to the technique used, I incrementally and in an evolutionary style modifiedall the default settings across the board to obtain that 440% improvement for the TPC-C

OLTP database benchmark (your mileage will of course vary). Look at the chart below

(Figure 1) at how the response time for 300 concurrent users decreased from 1080 ms to

200. And while 1080 ms (i.e. 1 second) may have been acceptable (i.e. met the requiredSLA), what would happen if the user load increased to 1000 or more users. Obviously the

tuned scenario would scale much better to thus to a much higher concurrent user load.

So what black magic was applied to get these results? The ten simple steps covered in thispaper are all thats required to see the 440% improvement. Note that the VM server used

was running Windows 2003 Server Enterprise Edition Release 2 64-bit, but similar issueswould also apply had it been Linux or some other OS.


4/10

Figure 1 Performance Achieved for Various Default Settings

Ten Tricks to Try

Sometimes the best things are quite often obvious and easy. So heres a simple plan at how

to incrementally optimize all the relevant database virtualization default settings:

1. Obtain a baseline test for relative comparisons

2. On the VM host, exclude VM clients from active, online virus scans

3. Remove Windows Indexing Service(because dont need fast file searches)

4. Remove other extraneous Windows services

5. Change the VM host registry settings to improve file system IO for databases

6. Optimize the VM host configuration and options

7. Optimize the VM client OS configuration and options for Oracle database

8. Remove other extraneous VM client OS services and daemons

9. Change VM client file system settings to improve IO performance for databases

10. Adjust VM client file system block size to more closely match Oracle block size


5/10

My key contention is that as long as there are sufficient resources and/or bandwidth to

handle the net requests, requiring database servers to be islands unto themselves is nowpass. Even if we say that virtualization adds a 10-20% overhead (which Im inclined to

say is far closer to 10 than 20), with cheap hardware these days the benefits far outweigh

the negatives just spend 10% more and get a bigger server to handle multiple databases.But it may be a few years before that opinion is generally or more widely accepted.

The remainder of this paper focuses on just three somewhat less obvious key steps from thelist above (which are highlighted in green): #5 and #9 (which kind of go together as one

concept), and #10. All the rest are somewhat more obvious and thus in need of less

explanation. Plus they also tend to be more subjective (i.e. what services dont we need).

Optimize All File Systems for Databases

Remember, a database ultimately makes IO requests and we all know that IO is theslowest part of the database equation. So look again above at what choices #5 and #9 reallyare and/or mean. As a reminder:

#5 = Change the VM host registry settings to improve file system IO for databases

#9 = Change VM client file system settings to improve IO performance for

databases

In reality, this is one and the same improvement simply being applied to two differentvirtualization levels: the host and each client. Thus Im simply going to present how to

accomplish this technique for both Windows and Linux, and then leave it to the reader to

make sure to apply it properly across all their various virtualization layers.

Both the Windows NTFS and Linux ext2/3 file systems maintain multiple meta-data

information related to file access such as date created, last time updated, etc. So an IOrequest might actually generate multiple physical IOs one for the data file and one or

more for updating the related meta-data. In the case of the VMware server, we really dont

care to keep OS detailed file system information about the hosted clients data file access its simply neither useful nor critical (unless doing snapshots). And at the Oracle database

level, we know Oracle accesses its files every so many seconds so why pay to update the

meta-data with that information. Oracle keeps its own timestamps (i.e. SCN) in the file

headers.

For Windows, we simply adjust the following registry entry:

HKEY_LOCAL_MACHINE\System\CurrentControlSet\

Control\FileSystem\NtfsDisableLastAccessUpdate = 1


6/10

By the way, Quest Softwares Toad for Oracle offers a screen to do this for Windows:

For Linux, there are several ways to accomplish the same result. We can set the arrtibute

for the individual Oracle files as follows:

chattr +A file_name

Or we can do it for an entire directory:

chattr R +A directory_name

However the best method (because it automatically handles any file additions) is to edit the

/etc/fstab and add the NOATIME attribute:

/dev/sda6 / ext3 defaults,noatime 1 1

/dev/sda1 /boot ext2 defaults,noatime 1 2

/dev/cdrom /mnt/cdrom iso9660 noauto,owner,ro 0 0

/dev/sda5 swap swap defaults 0 0

Many people ask if I know similar settings for other operating systems such as AIX, HPUXand Solaris I dont. But please email me and share that information, because I get asked

this one all the time and would love to have an answer for those people.


7/10

Match File System and Database Block Sizes

explain what choice #10 is. As a reminder:

#10 = Adjust VM client file system block size to more closely match Oracle block

size

The default block size for both the host and client OS file systems is generally not thesame as your Oracle block size (although hopefully the Oracle block size is a multiple of

the OS file block size). Since the host may be servicing multiple Oracle databases with

different block sizes, different database platforms (e.g. MySQL), or being used as to host

other applications (e.g. web server) we cannot always make this adjustment at the hostlevel. But generally we can make it for each of the Oracle database clients.

Lets assume we have a Linux host running a Linux based Oracle database client. Letsexamine Scenario #2 from the chart below. Start by assuming that we create the host file

system using the default block size: 2K. Lets further assume that we do the same thing onthe client but that we size the database blocks at 4K. Thus each Oracle physical IOrequests asks the client OS for two IOs, and each client IO asks the host for one IO. Thats

a total offour logical IO requests (although only two physical IOs in reality but note

that there is overhead for each logical IO request, so larger numbers are worse). Now if thedatabase block size had instead been 8K while the other factors had remained at 2K (i.e.

Scenario #3), then the logical IO would instead now be DB -> Client = 4 and Client ->

Host = 4 for a grand total ofeight logical IO requests. Heres a basic chart for some

common combinations and their sometimes surprising overheads:


8/10

Obviously there are some very bad choices in the chart above. So a seemingly good 16Kdatabase block size might actually result in a total of 12 to 16 logical IO requests across the

virtualization infrastructure under the worst case scenario. While its still effectively just 16

K being read, the extra overheads added by this mismatch only serve to multiply the badeffects. So choose wisely.

The Results & Conclusion

Look again back above at Figure 1 above where response time for 300 concurrent usersdecreased from 1080 ms to 200. With those initial default settings, the server could only

scale to some 500+ users and still meet the under 2 second SLA. I was able to scale well

above 2000 users on the very same hardware and with better average response times.

The first question people ask is how did the incremental improvements pan out? That iswhat measurable percentage improvement went with each of colored line in Figure 1? I

have included Figure 2 below to show that the tests (and their resulting graph lines) do infact have a one to one correspondence to the recommended 10 steps or tricks to try. But I

very purposefully left the graph color versus performance ramifications results key off

Figure 1 so that you would be inclined to try them all. Theyre too easy to skip even one ofthem and theyre all worth it.


9/10

Figure 2 - Breakdown of the Steps to Achieve the Results

This paper attempts to demonstrate that a database can perform up to 440% better simplyby adjusting various virtualized infrastructure default settings. Of course your mileage

will vary, but the key point is that by simply taking the time to properly layer your virtual

infrastructure settings can have substantial impacts which are magnified by the net orcumulative effect of a virtual server hosting multiple database servers. And while my

example was specific to Oracle, these same principles generally apply to any database

being deployed on a virtual server.

So dont hesitate to deploy your database servers on virtualized platforms it works.

Lessons Learned: (i.e. Best Practices)

Tons of low hanging fruit (i.e. easy no-brainer stuff)

Optimize each of the four major VM platform stacks

o Optimize the Host Machine (BIOS too)

o Optimize the Host VMware / OS Setup

o Optimize each Guest VM Configuration

o Optimize each Guest Operating System

Remember: 440% improvement for practically free really


10/10

About the Author

Bert Scalzo is a Database Expert for Quest Software and a member of the TOAD dev team.

He has worked with Oracle databases for over two decades, including time spent working

at both Oracle Education & Oracle Consulting. Mr. Scalzo is an Oracle ACE, holdsseveral Oracle Masters certifications, plus an extensive academic background - including a

BS, MS and PhD in Computer Science, an MBA and several insurance industry

designations. He is an accomplished speaker and has presented at numerous Oracleconferences - including OOW, ODTUG, IOUGA, OAUG, RMOUG and Hotsos. Mr.

Scalzo's key areas of DBA interest are Data Modeling, Database Benchmarking, Database

Tuning & Optimization, "Star Schema" Data Warehouses, Linux and VMware. He has also

written many articles, papers and blogs - including for Oracle Technology Network (OTN),Oracle Magazine, Oracle Informant, PC Week (eWeek), Dell Power Solutions Magazine,

The LINUX Journal, linux.com, Oracle FAQ and Toad World. Plus he has written six

books: Oracle DBA Guide to Data Warehousing and Star Schemas, TOAD Handbook,TOAD Pocket Reference (2nd Edition),Database Benchmarking: Practical Methods for

Oracle & SQL Server, Oracle on VMware: Expert tips for Database Virtualization, and

Advanced Oracle Utilities: The Definitive Reference. Mr. Scalzos email addresses are

[email protected] [email protected] .

Optimizing Oracle on VMware

Documents

Transcript of Optimizing Oracle on VMware