Monitoring(a(Virtualized(Database(Server( · » Virtualizaon’turns’...
Transcript of Monitoring(a(Virtualized(Database(Server( · » Virtualizaon’turns’...
![Page 1: Monitoring(a(Virtualized(Database(Server( · » Virtualizaon’turns’ hardware’into’aresource’ queue’ » Started’with’mainframes,’ fell’outof’favor’when’PC’](https://reader034.fdocuments.net/reader034/viewer/2022050305/5f6e291d7954792a961b4081/html5/thumbnails/1.jpg)
Monitoring a Virtualized Database Server Thomas LaRock
Head Geek SolarWinds
![Page 2: Monitoring(a(Virtualized(Database(Server( · » Virtualizaon’turns’ hardware’into’aresource’ queue’ » Started’with’mainframes,’ fell’outof’favor’when’PC’](https://reader034.fdocuments.net/reader034/viewer/2022050305/5f6e291d7954792a961b4081/html5/thumbnails/2.jpg)
Why Are You Here?
» You have virtualized database servers » You are going to have virtualized database servers
![Page 3: Monitoring(a(Virtualized(Database(Server( · » Virtualizaon’turns’ hardware’into’aresource’ queue’ » Started’with’mainframes,’ fell’outof’favor’when’PC’](https://reader034.fdocuments.net/reader034/viewer/2022050305/5f6e291d7954792a961b4081/html5/thumbnails/3.jpg)
Why Are You Here?
» You have virtualized database servers » You are going to have virtualized database servers
» You want/need to troubleshoot » You want to avoid issues
![Page 4: Monitoring(a(Virtualized(Database(Server( · » Virtualizaon’turns’ hardware’into’aresource’ queue’ » Started’with’mainframes,’ fell’outof’favor’when’PC’](https://reader034.fdocuments.net/reader034/viewer/2022050305/5f6e291d7954792a961b4081/html5/thumbnails/4.jpg)
A Li=le About Me…
@sqlrockstar
sqlrockstar@ thomaslarock.com
linkedin.com/in/sqlrockstar
![Page 5: Monitoring(a(Virtualized(Database(Server( · » Virtualizaon’turns’ hardware’into’aresource’ queue’ » Started’with’mainframes,’ fell’outof’favor’when’PC’](https://reader034.fdocuments.net/reader034/viewer/2022050305/5f6e291d7954792a961b4081/html5/thumbnails/5.jpg)
» Virtualiza@on turns hardware into a resource queue » Started with mainframes, fell out of favor when PC became popular, but making a comeback
hGp://www.dan-‐dare.org/FreeFun/Images/TheMatrixWallpaper800.jpg
What Is Virtualiza@on?
5
![Page 6: Monitoring(a(Virtualized(Database(Server( · » Virtualizaon’turns’ hardware’into’aresource’ queue’ » Started’with’mainframes,’ fell’outof’favor’when’PC’](https://reader034.fdocuments.net/reader034/viewer/2022050305/5f6e291d7954792a961b4081/html5/thumbnails/6.jpg)
Virtualiza@on Vendors
6
![Page 7: Monitoring(a(Virtualized(Database(Server( · » Virtualizaon’turns’ hardware’into’aresource’ queue’ » Started’with’mainframes,’ fell’outof’favor’when’PC’](https://reader034.fdocuments.net/reader034/viewer/2022050305/5f6e291d7954792a961b4081/html5/thumbnails/7.jpg)
Hypervisor Layer
7
hGp://www.vmware.com/files/images/thumbnails/vmw-‐dgrm-‐vsphere-‐most-‐secure-‐hypervisor-‐lg.gif
![Page 8: Monitoring(a(Virtualized(Database(Server( · » Virtualizaon’turns’ hardware’into’aresource’ queue’ » Started’with’mainframes,’ fell’outof’favor’when’PC’](https://reader034.fdocuments.net/reader034/viewer/2022050305/5f6e291d7954792a961b4081/html5/thumbnails/8.jpg)
Physical vs. Virtual
8
hGp://cdn.Ggtmedia.com/ITKE/uploads/blogs.dir/28/files/2009/10/[email protected]
![Page 9: Monitoring(a(Virtualized(Database(Server( · » Virtualizaon’turns’ hardware’into’aresource’ queue’ » Started’with’mainframes,’ fell’outof’favor’when’PC’](https://reader034.fdocuments.net/reader034/viewer/2022050305/5f6e291d7954792a961b4081/html5/thumbnails/9.jpg)
» Maximize resource usage § Finance loves you
Why Virtualize?
9
![Page 10: Monitoring(a(Virtualized(Database(Server( · » Virtualizaon’turns’ hardware’into’aresource’ queue’ » Started’with’mainframes,’ fell’outof’favor’when’PC’](https://reader034.fdocuments.net/reader034/viewer/2022050305/5f6e291d7954792a961b4081/html5/thumbnails/10.jpg)
» Maximize resource usage § Finance loves you
» Biggest strength – shared resources » Biggest weakness – shared resources!
Why Virtualize?
10
![Page 11: Monitoring(a(Virtualized(Database(Server( · » Virtualizaon’turns’ hardware’into’aresource’ queue’ » Started’with’mainframes,’ fell’outof’favor’when’PC’](https://reader034.fdocuments.net/reader034/viewer/2022050305/5f6e291d7954792a961b4081/html5/thumbnails/11.jpg)
» Maximize resource usage § Finance loves you
» Biggest strength – shared resources » Biggest weakness – shared resources! » Database servers o`en last to go
Why Virtualize?
11
![Page 12: Monitoring(a(Virtualized(Database(Server( · » Virtualizaon’turns’ hardware’into’aresource’ queue’ » Started’with’mainframes,’ fell’outof’favor’when’PC’](https://reader034.fdocuments.net/reader034/viewer/2022050305/5f6e291d7954792a961b4081/html5/thumbnails/12.jpg)
» They don’t change! § CPU § Memory § Disk I/O § Network § Locking/blocking
hGp://www.econtech.com/newsleGer/img/delays.jpg
Resource Bo=lenecks
12
![Page 13: Monitoring(a(Virtualized(Database(Server( · » Virtualizaon’turns’ hardware’into’aresource’ queue’ » Started’with’mainframes,’ fell’outof’favor’when’PC’](https://reader034.fdocuments.net/reader034/viewer/2022050305/5f6e291d7954792a961b4081/html5/thumbnails/13.jpg)
» Standard tools may include: § Perfmon § MDW § SQL Server DMVs
Database Monitoring
13
![Page 14: Monitoring(a(Virtualized(Database(Server( · » Virtualizaon’turns’ hardware’into’aresource’ queue’ » Started’with’mainframes,’ fell’outof’favor’when’PC’](https://reader034.fdocuments.net/reader034/viewer/2022050305/5f6e291d7954792a961b4081/html5/thumbnails/14.jpg)
» Standard tools may include: § Perfmon § MDW § SQL Server DMVs
» Waits and queues are vital
Database Monitoring
14
![Page 15: Monitoring(a(Virtualized(Database(Server( · » Virtualizaon’turns’ hardware’into’aresource’ queue’ » Started’with’mainframes,’ fell’outof’favor’when’PC’](https://reader034.fdocuments.net/reader034/viewer/2022050305/5f6e291d7954792a961b4081/html5/thumbnails/15.jpg)
» Standard tools may include: § Perfmon § MDW § SQL Server DMVs
» Waits and queues are vital » Must examine hosts
§ vSphere, vCenter § System Center VMM
Database Monitoring
15
![Page 16: Monitoring(a(Virtualized(Database(Server( · » Virtualizaon’turns’ hardware’into’aresource’ queue’ » Started’with’mainframes,’ fell’outof’favor’when’PC’](https://reader034.fdocuments.net/reader034/viewer/2022050305/5f6e291d7954792a961b4081/html5/thumbnails/16.jpg)
» Standard tools may include: § Perfmon § MDW § SQL Server DMVs
» Waits and queues are vital » Must examine hosts
§ vSphere, vCenter § System Center VMM
» I want to eliminate host/guest as culprit § BEFORE trying to tune a query!
Database Monitoring
16
![Page 17: Monitoring(a(Virtualized(Database(Server( · » Virtualizaon’turns’ hardware’into’aresource’ queue’ » Started’with’mainframes,’ fell’outof’favor’when’PC’](https://reader034.fdocuments.net/reader034/viewer/2022050305/5f6e291d7954792a961b4081/html5/thumbnails/17.jpg)
Which Layer of Cake?
17
![Page 18: Monitoring(a(Virtualized(Database(Server( · » Virtualizaon’turns’ hardware’into’aresource’ queue’ » Started’with’mainframes,’ fell’outof’favor’when’PC’](https://reader034.fdocuments.net/reader034/viewer/2022050305/5f6e291d7954792a961b4081/html5/thumbnails/18.jpg)
» CPU » Memory » Disk » Network » Tips/tricks » Ques@ons/comments/concerns
Agenda
18
![Page 19: Monitoring(a(Virtualized(Database(Server( · » Virtualizaon’turns’ hardware’into’aresource’ queue’ » Started’with’mainframes,’ fell’outof’favor’when’PC’](https://reader034.fdocuments.net/reader034/viewer/2022050305/5f6e291d7954792a961b4081/html5/thumbnails/19.jpg)
» vCPU § Start at 1.5:1 ra@o of vCPU to logical cores § vNUMA and hot-‐plug § Idle vCPUs can actually hinder peformance
CPU Configura@on
19
![Page 20: Monitoring(a(Virtualized(Database(Server( · » Virtualizaon’turns’ hardware’into’aresource’ queue’ » Started’with’mainframes,’ fell’outof’favor’when’PC’](https://reader034.fdocuments.net/reader034/viewer/2022050305/5f6e291d7954792a961b4081/html5/thumbnails/20.jpg)
» vCPU § Start at 1.5:1 ra@o of vCPU to logical cores § vNUMA and hot-‐plug § Idle vCPUs can actually hinder peformance
» Know how many vCPUs you need? § Probably not, so start with 1 and go up as needed § Remember vCPU * GHz formula
CPU Configura@on
20
![Page 21: Monitoring(a(Virtualized(Database(Server( · » Virtualizaon’turns’ hardware’into’aresource’ queue’ » Started’with’mainframes,’ fell’outof’favor’when’PC’](https://reader034.fdocuments.net/reader034/viewer/2022050305/5f6e291d7954792a961b4081/html5/thumbnails/21.jpg)
» vCPU § Start at 1.5:1 ra@o of vCPU to logical cores § vNUMA and hot-‐plug § Idle vCPUs can actually hinder peformance
» Know how many vCPUs you need? § Probably not, so start with 1 and go up as needed § Remember vCPU * GHz formula
» CPU and NUMA § Set MAXDOP = vNUMA core code count
CPU Configura@on
21
![Page 22: Monitoring(a(Virtualized(Database(Server( · » Virtualizaon’turns’ hardware’into’aresource’ queue’ » Started’with’mainframes,’ fell’outof’favor’when’PC’](https://reader034.fdocuments.net/reader034/viewer/2022050305/5f6e291d7954792a961b4081/html5/thumbnails/22.jpg)
CPU Monitoring
» Databases are not typically CPU bound
![Page 23: Monitoring(a(Virtualized(Database(Server( · » Virtualizaon’turns’ hardware’into’aresource’ queue’ » Started’with’mainframes,’ fell’outof’favor’when’PC’](https://reader034.fdocuments.net/reader034/viewer/2022050305/5f6e291d7954792a961b4081/html5/thumbnails/23.jpg)
CPU Monitoring
» Databases are not typically CPU bound » BMFF metric – VM Ready Time
§ Similar to ‘runnable’ queue § Amount of @me VM was ‘ready’ to run, but needed to wait for CPU resource § Hyper-‐V Hypervisor Virtual Processor\CPU Wait Time Per Dispatch
hGp://2.bp.blogspot.com/_jIPNvBuH41s/TLgMSVTKyQI/AAAAAAAAFeU/K_atz9lM5QQ/s1600/Intel-‐718028.jpg
![Page 24: Monitoring(a(Virtualized(Database(Server( · » Virtualizaon’turns’ hardware’into’aresource’ queue’ » Started’with’mainframes,’ fell’outof’favor’when’PC’](https://reader034.fdocuments.net/reader034/viewer/2022050305/5f6e291d7954792a961b4081/html5/thumbnails/24.jpg)
CPU Monitoring
» Databases are not typically CPU bound » BMFF metric – VM Ready Time
§ Similar to ‘runnable’ queue § Amount of @me VM was ‘ready’ to run, but needed to wait for CPU resource § Hyper-‐V Hypervisor Virtual Processor\CPU Wait Time Per Dispatch
» Other metrics include: § VM CPU u@liza@on § Host CPU u@liza@on
hGp://2.bp.blogspot.com/_jIPNvBuH41s/TLgMSVTKyQI/AAAAAAAAFeU/K_atz9lM5QQ/s1600/Intel-‐718028.jpg
![Page 25: Monitoring(a(Virtualized(Database(Server( · » Virtualizaon’turns’ hardware’into’aresource’ queue’ » Started’with’mainframes,’ fell’outof’favor’when’PC’](https://reader034.fdocuments.net/reader034/viewer/2022050305/5f6e291d7954792a961b4081/html5/thumbnails/25.jpg)
Is There a CPU Problem?
» Where does it hurt? § DB performance is “slow”, for no apparent reason § Host server has high CPU u@liza@on
![Page 26: Monitoring(a(Virtualized(Database(Server( · » Virtualizaon’turns’ hardware’into’aresource’ queue’ » Started’with’mainframes,’ fell’outof’favor’when’PC’](https://reader034.fdocuments.net/reader034/viewer/2022050305/5f6e291d7954792a961b4081/html5/thumbnails/26.jpg)
Is There a CPU Problem?
» Where does it hurt? § DB performance is “slow”, for no apparent reason § Host server has high CPU u@liza@on
» Possible causes § CPU over commiGed § Addi@onal VM guests (planned or unplanned)
![Page 27: Monitoring(a(Virtualized(Database(Server( · » Virtualizaon’turns’ hardware’into’aresource’ queue’ » Started’with’mainframes,’ fell’outof’favor’when’PC’](https://reader034.fdocuments.net/reader034/viewer/2022050305/5f6e291d7954792a961b4081/html5/thumbnails/27.jpg)
Is There a CPU Problem?
» Where does it hurt? § DB performance is “slow”, for no apparent reason § Host server has high CPU u@liza@on
» Possible causes § CPU over commiGed § Addi@onal VM guests (planned or unplanned)
» Poten@al remedies § Move VM guests back to where they belong § Reconfigure vCPU setngs § Purchase more CPUs
![Page 28: Monitoring(a(Virtualized(Database(Server( · » Virtualizaon’turns’ hardware’into’aresource’ queue’ » Started’with’mainframes,’ fell’outof’favor’when’PC’](https://reader034.fdocuments.net/reader034/viewer/2022050305/5f6e291d7954792a961b4081/html5/thumbnails/28.jpg)
CPU Scenario
» Measure host CPU usage § AVG > 75% -‐or-‐ Peak > 90%
» Check guest VM ready @me § If any vCPU > 1000ms (or 10%)
» Host CPU satura@on exists § Balance guest resources § Get more CPUs for the host
![Page 29: Monitoring(a(Virtualized(Database(Server( · » Virtualizaon’turns’ hardware’into’aresource’ queue’ » Started’with’mainframes,’ fell’outof’favor’when’PC’](https://reader034.fdocuments.net/reader034/viewer/2022050305/5f6e291d7954792a961b4081/html5/thumbnails/29.jpg)
Memory Configura@on
» vRAM assigned to VM guest § Ballooning – unused memory that was given back § Swapping – memory given back forcibly
![Page 30: Monitoring(a(Virtualized(Database(Server( · » Virtualizaon’turns’ hardware’into’aresource’ queue’ » Started’with’mainframes,’ fell’outof’favor’when’PC’](https://reader034.fdocuments.net/reader034/viewer/2022050305/5f6e291d7954792a961b4081/html5/thumbnails/30.jpg)
Memory Configura@on
» vRAM assigned to VM guest § Ballooning – unused memory that was given back § Swapping – memory given back forcibly
» Plan for 80% host alloca@on
![Page 31: Monitoring(a(Virtualized(Database(Server( · » Virtualizaon’turns’ hardware’into’aresource’ queue’ » Started’with’mainframes,’ fell’outof’favor’when’PC’](https://reader034.fdocuments.net/reader034/viewer/2022050305/5f6e291d7954792a961b4081/html5/thumbnails/31.jpg)
Memory Configura@on
» vRAM assigned to VM guest § Ballooning – unused memory that was given back § Swapping – memory given back forcibly
» Plan for 80% host alloca@on » Set Memory Reserva@on >= DB Memory
![Page 32: Monitoring(a(Virtualized(Database(Server( · » Virtualizaon’turns’ hardware’into’aresource’ queue’ » Started’with’mainframes,’ fell’outof’favor’when’PC’](https://reader034.fdocuments.net/reader034/viewer/2022050305/5f6e291d7954792a961b4081/html5/thumbnails/32.jpg)
Memory Configura@on
» vRAM assigned to VM guest § Ballooning – unused memory that was given back § Swapping – memory given back forcibly
» Plan for 80% host alloca@on » Set Memory Reserva@on >= DB Memory » Configure min/max SQL memory
![Page 33: Monitoring(a(Virtualized(Database(Server( · » Virtualizaon’turns’ hardware’into’aresource’ queue’ » Started’with’mainframes,’ fell’outof’favor’when’PC’](https://reader034.fdocuments.net/reader034/viewer/2022050305/5f6e291d7954792a961b4081/html5/thumbnails/33.jpg)
Memory Configura@on
» vRAM assigned to VM guest § Ballooning – unused memory that was given back § Swapping – memory given back forcibly
» Plan for 80% host alloca@on » Set Memory Reserva@on >= DB Memory » Configure min/max SQL memory » Lock Pages in Memory granted
![Page 34: Monitoring(a(Virtualized(Database(Server( · » Virtualizaon’turns’ hardware’into’aresource’ queue’ » Started’with’mainframes,’ fell’outof’favor’when’PC’](https://reader034.fdocuments.net/reader034/viewer/2022050305/5f6e291d7954792a961b4081/html5/thumbnails/34.jpg)
Memory Monitoring
» Databases are oRen memory bound § O`en due to bad configura@ons
34
![Page 35: Monitoring(a(Virtualized(Database(Server( · » Virtualizaon’turns’ hardware’into’aresource’ queue’ » Started’with’mainframes,’ fell’outof’favor’when’PC’](https://reader034.fdocuments.net/reader034/viewer/2022050305/5f6e291d7954792a961b4081/html5/thumbnails/35.jpg)
Memory Monitoring
» Databases are oRen memory bound § O`en due to bad configura@ons
» BMFF – Swapping, Ballooning § If you got the memory, you want to keep it
35
hGp://[email protected]/wp-‐content/uploads/2013/07/ram.jpg
![Page 36: Monitoring(a(Virtualized(Database(Server( · » Virtualizaon’turns’ hardware’into’aresource’ queue’ » Started’with’mainframes,’ fell’outof’favor’when’PC’](https://reader034.fdocuments.net/reader034/viewer/2022050305/5f6e291d7954792a961b4081/html5/thumbnails/36.jpg)
Memory Monitoring
» Databases are oRen memory bound § O`en due to bad configura@ons
» BMFF – Swapping, Ballooning § If you got the memory, you want to keep it
» Other Metrics include § Guest and Host Memory U@liza@on § Available Mbytes
36
hGp://[email protected]/wp-‐content/uploads/2013/07/ram.jpg
![Page 37: Monitoring(a(Virtualized(Database(Server( · » Virtualizaon’turns’ hardware’into’aresource’ queue’ » Started’with’mainframes,’ fell’outof’favor’when’PC’](https://reader034.fdocuments.net/reader034/viewer/2022050305/5f6e291d7954792a961b4081/html5/thumbnails/37.jpg)
Is There a Memory Problem?
» Where does it hurt? § Low memory counters § Host ballooning > 0
![Page 38: Monitoring(a(Virtualized(Database(Server( · » Virtualizaon’turns’ hardware’into’aresource’ queue’ » Started’with’mainframes,’ fell’outof’favor’when’PC’](https://reader034.fdocuments.net/reader034/viewer/2022050305/5f6e291d7954792a961b4081/html5/thumbnails/38.jpg)
Is There a Memory Problem?
» Where does it hurt? § Low memory counters § Host ballooning > 0
» Possible causes § Host/VM/SQL memory misconfigured
![Page 39: Monitoring(a(Virtualized(Database(Server( · » Virtualizaon’turns’ hardware’into’aresource’ queue’ » Started’with’mainframes,’ fell’outof’favor’when’PC’](https://reader034.fdocuments.net/reader034/viewer/2022050305/5f6e291d7954792a961b4081/html5/thumbnails/39.jpg)
Is There a Memory Problem?
» Where does it hurt? § Low memory counters § Host ballooning > 0
» Possible causes § Host/VM/SQL memory misconfigured
» Poten@al remedies § Set VM memory reserva@on = memory provisioned § Check memory allowed per NUMA node
![Page 40: Monitoring(a(Virtualized(Database(Server( · » Virtualizaon’turns’ hardware’into’aresource’ queue’ » Started’with’mainframes,’ fell’outof’favor’when’PC’](https://reader034.fdocuments.net/reader034/viewer/2022050305/5f6e291d7954792a961b4081/html5/thumbnails/40.jpg)
Memory Scenario
» Measure Available Mbytes § Is it < 300, or < 10% (which do you prefer?)
» Check VM Ballooning § Is it > 0? § Set min/max to non-‐default values
» Assume min/max is set, check for ‘lock pages in memory’ granted to service account § If not, then DO IT! § Use of ‘large pages’
![Page 41: Monitoring(a(Virtualized(Database(Server( · » Virtualizaon’turns’ hardware’into’aresource’ queue’ » Started’with’mainframes,’ fell’outof’favor’when’PC’](https://reader034.fdocuments.net/reader034/viewer/2022050305/5f6e291d7954792a961b4081/html5/thumbnails/41.jpg)
Disk Configura@on
» You are sharing your storage! § Storage issues are usually related to configura@on and not capabili@es of ESX § Follow best prac@ces from storage vendor
![Page 42: Monitoring(a(Virtualized(Database(Server( · » Virtualizaon’turns’ hardware’into’aresource’ queue’ » Started’with’mainframes,’ fell’outof’favor’when’PC’](https://reader034.fdocuments.net/reader034/viewer/2022050305/5f6e291d7954792a961b4081/html5/thumbnails/42.jpg)
Disk Configura@on
» You are sharing your storage! § Storage issues are usually related to configura@on and not capabili@es of ESX § Follow best prac@ces from storage vendor
» Create dedicated datastores for databases
![Page 43: Monitoring(a(Virtualized(Database(Server( · » Virtualizaon’turns’ hardware’into’aresource’ queue’ » Started’with’mainframes,’ fell’outof’favor’when’PC’](https://reader034.fdocuments.net/reader034/viewer/2022050305/5f6e291d7954792a961b4081/html5/thumbnails/43.jpg)
Disk Configura@on
» You are sharing your storage! § Storage issues are usually related to configura@on and not capabili@es of ESX § Follow best prac@ces from storage vendor
» Create dedicated datastores for databases » Isolate data and log files
![Page 44: Monitoring(a(Virtualized(Database(Server( · » Virtualizaon’turns’ hardware’into’aresource’ queue’ » Started’with’mainframes,’ fell’outof’favor’when’PC’](https://reader034.fdocuments.net/reader034/viewer/2022050305/5f6e291d7954792a961b4081/html5/thumbnails/44.jpg)
Disk Monitoring
» BMFF – Latency § Host maxTotalLatency § Host Device Latency (by device) § VM Command Latency (for all VMs)
hGp://@nyurl.com/m5oxpdj
![Page 45: Monitoring(a(Virtualized(Database(Server( · » Virtualizaon’turns’ hardware’into’aresource’ queue’ » Started’with’mainframes,’ fell’outof’favor’when’PC’](https://reader034.fdocuments.net/reader034/viewer/2022050305/5f6e291d7954792a961b4081/html5/thumbnails/45.jpg)
Disk Monitoring
» BMFF – Latency § Host maxTotalLatency § Host Device Latency (by device) § VM Command Latency (for all VMs)
» Other metrics include § Disk Read Rate § Disk Write Rate
hGp://@nyurl.com/m5oxpdj
![Page 46: Monitoring(a(Virtualized(Database(Server( · » Virtualizaon’turns’ hardware’into’aresource’ queue’ » Started’with’mainframes,’ fell’outof’favor’when’PC’](https://reader034.fdocuments.net/reader034/viewer/2022050305/5f6e291d7954792a961b4081/html5/thumbnails/46.jpg)
Is There a Disk Problem?
» Where does it hurt? § High latencies
![Page 47: Monitoring(a(Virtualized(Database(Server( · » Virtualizaon’turns’ hardware’into’aresource’ queue’ » Started’with’mainframes,’ fell’outof’favor’when’PC’](https://reader034.fdocuments.net/reader034/viewer/2022050305/5f6e291d7954792a961b4081/html5/thumbnails/47.jpg)
Is There a Disk Problem?
» Where does it hurt? § High latencies
» Possible causes § Overloaded storage § Overloaded network § I/O intensive queries § Memory/CPU issues
![Page 48: Monitoring(a(Virtualized(Database(Server( · » Virtualizaon’turns’ hardware’into’aresource’ queue’ » Started’with’mainframes,’ fell’outof’favor’when’PC’](https://reader034.fdocuments.net/reader034/viewer/2022050305/5f6e291d7954792a961b4081/html5/thumbnails/48.jpg)
Is There a Disk Problem?
» Where does it hurt? § High latencies
» Possible causes § Overloaded storage § Overloaded network § I/O intensive queries § Memory/CPU issues
» Poten@al remedies § Reconfigure storage and/or network § Rewrite queries
![Page 49: Monitoring(a(Virtualized(Database(Server( · » Virtualizaon’turns’ hardware’into’aresource’ queue’ » Started’with’mainframes,’ fell’outof’favor’when’PC’](https://reader034.fdocuments.net/reader034/viewer/2022050305/5f6e291d7954792a961b4081/html5/thumbnails/49.jpg)
Disk Scenario
» Monitor disk latency § Is Host maxTotalLatency > 30ms ?
» Is VM Command Latency >= 30ms for your VM? § Look for PAGEIOLATCH_XX waits § Tune Disk I/O intensive processes on database § Are Memory / CPU issues causing I/O problems
» Review device latency § Review Disk Read/Write rates for that device
![Page 50: Monitoring(a(Virtualized(Database(Server( · » Virtualizaon’turns’ hardware’into’aresource’ queue’ » Started’with’mainframes,’ fell’outof’favor’when’PC’](https://reader034.fdocuments.net/reader034/viewer/2022050305/5f6e291d7954792a961b4081/html5/thumbnails/50.jpg)
Network Configura@on
» How big is that pipe? § vSwitch – so`ware switch inside of VMKernel § vSwitch can be @ed to 1 or more NICs § VMWare claims to handle over 30GB/sec § Most db’s don’t need that much
![Page 51: Monitoring(a(Virtualized(Database(Server( · » Virtualizaon’turns’ hardware’into’aresource’ queue’ » Started’with’mainframes,’ fell’outof’favor’when’PC’](https://reader034.fdocuments.net/reader034/viewer/2022050305/5f6e291d7954792a961b4081/html5/thumbnails/51.jpg)
Network Configura@on
» How big is that pipe? § vSwitch – so`ware switch inside of VMKernel § vSwitch can be @ed to 1 or more NICs § VMWare claims to handle over 30GB/sec § Most db’s don’t need that much
» VMXNET paravirtualized network adapter § Installed into guest O/S capable of 1Gbps § Minimizes overhead between VM and Host § Requires VMware Tools
![Page 52: Monitoring(a(Virtualized(Database(Server( · » Virtualizaon’turns’ hardware’into’aresource’ queue’ » Started’with’mainframes,’ fell’outof’favor’when’PC’](https://reader034.fdocuments.net/reader034/viewer/2022050305/5f6e291d7954792a961b4081/html5/thumbnails/52.jpg)
Network Configura@on
» How big is that pipe? § vSwitch – so`ware switch inside of VMKernel § vSwitch can be @ed to 1 or more NICs § VMWare claims to handle over 30GB/sec § Most db’s don’t need that much
» VMXNET paravirtualized network adapter § Installed into guest O/S capable of 1Gbps § Minimizes overhead between VM and Host § Requires VMware Tools
» Co-‐locate VMs when possible
![Page 53: Monitoring(a(Virtualized(Database(Server( · » Virtualizaon’turns’ hardware’into’aresource’ queue’ » Started’with’mainframes,’ fell’outof’favor’when’PC’](https://reader034.fdocuments.net/reader034/viewer/2022050305/5f6e291d7954792a961b4081/html5/thumbnails/53.jpg)
Network Monitoring
» BMFF – Dropped Packets » Other Metrics Include
§ Network Rate § Bytes Total/sec § Output Queue Length
hGp://@nyurl.com/n8nqwuu
![Page 54: Monitoring(a(Virtualized(Database(Server( · » Virtualizaon’turns’ hardware’into’aresource’ queue’ » Started’with’mainframes,’ fell’outof’favor’when’PC’](https://reader034.fdocuments.net/reader034/viewer/2022050305/5f6e291d7954792a961b4081/html5/thumbnails/54.jpg)
Network Monitoring
» BMFF – Dropped Packets » Other Metrics Include
§ Network Rate § Bytes Total/sec § Output Queue Length
» Look for dropped packets § Any errors on host NICs? § One NIC getng more traffic than another?
hGp://@nyurl.com/n8nqwuu
![Page 55: Monitoring(a(Virtualized(Database(Server( · » Virtualizaon’turns’ hardware’into’aresource’ queue’ » Started’with’mainframes,’ fell’outof’favor’when’PC’](https://reader034.fdocuments.net/reader034/viewer/2022050305/5f6e291d7954792a961b4081/html5/thumbnails/55.jpg)
Network Monitoring
» BMFF – Dropped Packets » Other Metrics Include
§ Network Rate § Bytes Total/sec § Output Queue Length
» Look for dropped packets § Any errors on host NICs? § One NIC getng more traffic than another?
» Is Network Rate is getng close to max?
hGp://@nyurl.com/n8nqwuu
![Page 56: Monitoring(a(Virtualized(Database(Server( · » Virtualizaon’turns’ hardware’into’aresource’ queue’ » Started’with’mainframes,’ fell’outof’favor’when’PC’](https://reader034.fdocuments.net/reader034/viewer/2022050305/5f6e291d7954792a961b4081/html5/thumbnails/56.jpg)
Is There a Network Problem?
» Where does it hurt? § Network performance is slow or intermiGent
![Page 57: Monitoring(a(Virtualized(Database(Server( · » Virtualizaon’turns’ hardware’into’aresource’ queue’ » Started’with’mainframes,’ fell’outof’favor’when’PC’](https://reader034.fdocuments.net/reader034/viewer/2022050305/5f6e291d7954792a961b4081/html5/thumbnails/57.jpg)
Is There a Network Problem?
» Where does it hurt? § Network performance is slow or intermiGent
» Possible causes § Overloaded host CPU § Single NIC § Wrong network adapter drivers
![Page 58: Monitoring(a(Virtualized(Database(Server( · » Virtualizaon’turns’ hardware’into’aresource’ queue’ » Started’with’mainframes,’ fell’outof’favor’when’PC’](https://reader034.fdocuments.net/reader034/viewer/2022050305/5f6e291d7954792a961b4081/html5/thumbnails/58.jpg)
Is There a Network Problem?
» Where does it hurt? § Network performance is slow or intermiGent
» Possible causes § Overloaded host CPU § Single NIC § Wrong network adapter drivers
» Poten@al remedies § Reduce CPU pressure on host § NIC teaming
![Page 59: Monitoring(a(Virtualized(Database(Server( · » Virtualizaon’turns’ hardware’into’aresource’ queue’ » Started’with’mainframes,’ fell’outof’favor’when’PC’](https://reader034.fdocuments.net/reader034/viewer/2022050305/5f6e291d7954792a961b4081/html5/thumbnails/59.jpg)
Network Scenario
» Monitor droppedTx, droppedRx for all vmnic objects § Are they > 0?
» Check for overloaded host CPU § Add addi@onal vCPU § Add addi@onal vNIC
» Check VM driver configura@on for network devices § Bad config op@ons lead to bad network perf
![Page 60: Monitoring(a(Virtualized(Database(Server( · » Virtualizaon’turns’ hardware’into’aresource’ queue’ » Started’with’mainframes,’ fell’outof’favor’when’PC’](https://reader034.fdocuments.net/reader034/viewer/2022050305/5f6e291d7954792a961b4081/html5/thumbnails/60.jpg)
1. Don’t build your own host from spare parts § Unless your name is “MacGyver”
Tips and Tricks
60
hGp://www.unclebobs.com/getstorganized/wp-‐content/uploads/2012/11/mcgyver.jpeg
![Page 61: Monitoring(a(Virtualized(Database(Server( · » Virtualizaon’turns’ hardware’into’aresource’ queue’ » Started’with’mainframes,’ fell’outof’favor’when’PC’](https://reader034.fdocuments.net/reader034/viewer/2022050305/5f6e291d7954792a961b4081/html5/thumbnails/61.jpg)
2. Baseline/benchmark for performance § Otherwise you have no idea what is “good” or “bad”
Tips and Tricks
61
hGp://www.gulfcoastnews.com/2010-‐Images/KatrinaFloodMarkerRodenburg.jpg
![Page 62: Monitoring(a(Virtualized(Database(Server( · » Virtualizaon’turns’ hardware’into’aresource’ queue’ » Started’with’mainframes,’ fell’outof’favor’when’PC’](https://reader034.fdocuments.net/reader034/viewer/2022050305/5f6e291d7954792a961b4081/html5/thumbnails/62.jpg)
3. Know datastore op@ons § VMFS versus RDM, which one is right for you
Tips and Tricks
62
hGp://forza-‐it.co.uk/wp-‐content/uploads/Proper@es-‐of-‐Disk-‐4-‐RDM.jpg
![Page 63: Monitoring(a(Virtualized(Database(Server( · » Virtualizaon’turns’ hardware’into’aresource’ queue’ » Started’with’mainframes,’ fell’outof’favor’when’PC’](https://reader034.fdocuments.net/reader034/viewer/2022050305/5f6e291d7954792a961b4081/html5/thumbnails/63.jpg)
4. Avoid thin provisioning § Unless you enjoy headaches
Tips and Tricks
63
hGps://[email protected]/servlet/JiveServlet/showImage/38-‐8422-‐15789/ThinProvisioning+1.png
![Page 64: Monitoring(a(Virtualized(Database(Server( · » Virtualizaon’turns’ hardware’into’aresource’ queue’ » Started’with’mainframes,’ fell’outof’favor’when’PC’](https://reader034.fdocuments.net/reader034/viewer/2022050305/5f6e291d7954792a961b4081/html5/thumbnails/64.jpg)
5. Avoid over-‐alloca@on of CPU and memory § Over-‐alloca@on leads to over-‐commit
Tips and Tricks
64
hGp://www.endlesssimmer.com/wp-‐content/uploads/2009/10/3590888947_7e80ca52df.jpg
![Page 65: Monitoring(a(Virtualized(Database(Server( · » Virtualizaon’turns’ hardware’into’aresource’ queue’ » Started’with’mainframes,’ fell’outof’favor’when’PC’](https://reader034.fdocuments.net/reader034/viewer/2022050305/5f6e291d7954792a961b4081/html5/thumbnails/65.jpg)
6. Don’t trust O/S counters § O/S doesn’t know it is virtualized
Tips and Tricks
65
hGp://www.bryaneisenberg.com/wp-‐content/uploads/2013/11/bigstock-‐Mousetrap-‐35879422.jpg
![Page 66: Monitoring(a(Virtualized(Database(Server( · » Virtualizaon’turns’ hardware’into’aresource’ queue’ » Started’with’mainframes,’ fell’outof’favor’when’PC’](https://reader034.fdocuments.net/reader034/viewer/2022050305/5f6e291d7954792a961b4081/html5/thumbnails/66.jpg)
7. Running it all at once § Know your workloads
Tips and Tricks
66
hGp://epicswag.net/wp-‐content/uploads/2012/02/816.jpg
![Page 67: Monitoring(a(Virtualized(Database(Server( · » Virtualizaon’turns’ hardware’into’aresource’ queue’ » Started’with’mainframes,’ fell’outof’favor’when’PC’](https://reader034.fdocuments.net/reader034/viewer/2022050305/5f6e291d7954792a961b4081/html5/thumbnails/67.jpg)
8. Capacity planning § Leave room for growth, failovers
Tips and Tricks
67
hGp://www.paleycenter.org/assets/public-‐programs/PP-‐spring-‐2013/DrWhoFridge-‐web.jpg
![Page 68: Monitoring(a(Virtualized(Database(Server( · » Virtualizaon’turns’ hardware’into’aresource’ queue’ » Started’with’mainframes,’ fell’outof’favor’when’PC’](https://reader034.fdocuments.net/reader034/viewer/2022050305/5f6e291d7954792a961b4081/html5/thumbnails/68.jpg)
Resource Metric Host / VM Descrip@on CPU Ready VM CPU @me spent in ready state
Usage Both CPU usage as a percentage during a defined interval
Memory Swapin, Swapout Both Memory the host swaps in/out from/to disk (per VM, or cumula@ve over host)
Vmmemctl Both Amount of memory reclaimed from resource pool by way of ballooning
Disk maxtotallatency Host Highest latency value across all disks used by the host.
deviceLatency Host Average @me to complete a command from the physical device.
totalLatency Host Average latency in all guests.
Network droppedTx, dropped Rx Both Drop packets per second usage Both Sum of data transmiGed and received
Quick BMFF Sheet
68
![Page 69: Monitoring(a(Virtualized(Database(Server( · » Virtualizaon’turns’ hardware’into’aresource’ queue’ » Started’with’mainframes,’ fell’outof’favor’when’PC’](https://reader034.fdocuments.net/reader034/viewer/2022050305/5f6e291d7954792a961b4081/html5/thumbnails/69.jpg)
Ques@ons?
69
![Page 70: Monitoring(a(Virtualized(Database(Server( · » Virtualizaon’turns’ hardware’into’aresource’ queue’ » Started’with’mainframes,’ fell’outof’favor’when’PC’](https://reader034.fdocuments.net/reader034/viewer/2022050305/5f6e291d7954792a961b4081/html5/thumbnails/70.jpg)
» hGp://@nyurl.com/vm-‐perf-‐counters » hGp://@nyurl.com/common-‐vm-‐network-‐issues » hGp://@nyurl.com/perf-‐whitepaper » hGp://@nyurl.com/vmware-‐sql-‐bp-‐guide » hGp://@nyurl.com/vmdk-‐or-‐rdm » hGp://@nyurl.com/large-‐pages » hGp://@nyurl.com/klee-‐vnuma » hGp://@nyurl.com/measure-‐hyper-‐v-‐perf
For More Informa@on
70