OTN tour 2015 benchmarking oracle io performance with Orion by Alex Gorbachev
74
Benchmarking Oracle I/O Performance with ORION Alex Gorbachev November 2015
-
Upload
andrejs-vorobjovs -
Category
Education
-
view
250 -
download
1
Transcript of OTN tour 2015 benchmarking oracle io performance with Orion by Alex Gorbachev
Benchmarking Oracle I/O Performance with ORIONAlex Gorbachev
November 2015
Alex Gorbachev• Chief Technology Officer at Pythian• Blogger• Cloudera Champion of Big Data• OakTable Network member• Oracle ACE Director• Founder of BattleAgainstAnyGuess.com• Founder of Sydney Oracle Meetup• EVP, IOUG• EVP, Ottawa Oracle User Group
© 2014 Pythian
3
ABOUT PYTHIAN
Pythian is a global data outsourcing and consulting company that specializes in optimizing and managing mission-critical data systems.
Pythian blends the world’s leading data experts with advanced, secure service delivery processes to create the industry’s best standard of care for its clients.
Since its inception, Pythian has managed some of the world’s largest, most business-critical data infrastructures.
4
11,500Pythian currently manages over 11,500 systems.
385Pythian currently employs more than 385 people in 30 countries worldwide.
1997Pythian was founded in 1997
New Q2 2015
© 2011-2012 Pythian5
© 2011-2012 Pythian6
ORION - ORacle I/O Numbers
Generate I/O workload similar to database patterns
&measure I/O performance
© 2011-2012 Pythian7
Orion is designed to
stress testthe I/O subsystem
© 2011-2012 Pythian8
Orion is not perfect for simulation but
good enough
© 2011-2012 Pythian9
Use Orion before moving/deploying databases to the
new platform
© 2011-2012 Pythian10
Orion is used intwo scenarios
© 2011-2012 Pythian11
You have no idea what you need
and want to ensure you get
You know what you needand want toensure you have it
or
© 2011-2012 Pythian
Infrastructure tuning - what’s the goal?
• When you don’t know how much you need you try at least to ensure you take all you can
• Assess what’s your possible bottlenecks • 1 Gbit Ethernet => 100+ MBPS or 10,000+ IOPS (8K)
• 15K RPM disk
• will easily serve 100-150 IOPS with average resp. time <10ms
• can get to 200-250 IOPS but response time increase to 20 ms
• SSD - see vendors specs
• reads: random vs sequential... small vs large... no matter
• writes: pattern matters
12
© 2011-2012 Pythian
Orion
• Uses code-base similar to Oracle database kernel • Standalone binary or part of Oracle home since 11.2.0.1
• Standalone Orion downloadable version is only 11.1
• Tests only I/O subsystem • Minimal CPU consumption
• Async I/O is used to submit concurrent I/O requests • Each run includes multiple data points / tests
• Scaling concurrency of small and large I/Os
13
© 2011-2012 Pythian
Controlling Orion
• Workload patterns • Small random I/O size and scale
• Large I/O size, scale and pattern (random vs sequential)
• Write percentage • Cache warming • Duration of each test (data point) • Data layout (concatenation vs striping)
14
© 2011-2012 Pythian
Data Points Matrix
Large/Small, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9... 0, x, x, x, x, x, x, x, x, x, x... 1, x, x, x, x, x, x, x, x, x, x... 2, x, x, x, x, x, x, x, x, x, x... 3, x, x, x, x, x, x, x, x, x, x... 4, x, x, x, x, x, x, x, x, x, x... 5, x, x, x, x, x, x, x, x, x, x... 6, x, x, x, x, x, x, x, x, x, x... 7, x, x, x, x, x, x, x, x, x, x... 8, x, x, x, x, x, x, x, x, x, x... 9, x, x, x, x, x, x, x, x, x, x... 10, x, x, x, x, x, x, x, x, x, x... 11, x, x, x, x, x, x, x, x, x, x... .............................. .............................. ..............................
15
© 2011-2012 Pythian
Data Points Matrix
Large/Small, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9... 0, x, x, x, x, x, x, x, x, x, x... 1, x, x, x, x, x, x, x, x, x, x... 2, x, x, x, x, x, x, x, x, x, x... 3, x, x, x, x, x, x, x, x, x, x... 4, x, x, x, x, x, x, x, x, x, x... 5, x, x, x, x, x, x, x, x, x, x... 6, x, x, x, x, x, x, x, x, x, x... 7, x, x, x, x, x, x, x, x, x, x... 8, x, x, x, x, x, x, x, x, x, x... 9, x, x, x, x, x, x, x, x, x, x... 10, x, x, x, x, x, x, x, x, x, x... 11, x, x, x, x, x, x, x, x, x, x...
16
-run advanced -matrix detailed# of tests = (Xlarge + 1) * (Xsmall + 1)
© 2011-2012 Pythian
Data Points Matrix
Large/Small, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9... 0, x, x, x, x, x, x, x, x, x, x... 1, x, x, x, x, x, x, x, x, x, x... 2, x, x, x, x, x, x, x, x, x, x... 3, x, x, x, x, x, x, x, x, x, x... 4, x, x, x, x, x, x, x, x, x, x... 5, x, x, x, x, x, x, x, x, x, x... 6, x, x, x, x, x, x, x, x, x, x... 7, x, x, x, x, x, x, x, x, x, x... 8, x, x, x, x, x, x, x, x, x, x... 9, x, x, x, x, x, x, x, x, x, x... 10, x, x, x, x, x, x, x, x, x, x... 11, x, x, x, x, x, x, x, x, x, x...
17
-run advanced -matrix row -num_large 2# of tests = Xsmall + 1
© 2011-2012 Pythian
Data Points Matrix
Large/Small, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9... 0, x, x, x, x, x, x, x, x, x, x... 1, x, x, x, x, x, x, x, x, x, x... 2, x, x, x, x, x, x, x, x, x, x... 3, x, x, x, x, x, x, x, x, x, x... 4, x, x, x, x, x, x, x, x, x, x... 5, x, x, x, x, x, x, x, x, x, x... 6, x, x, x, x, x, x, x, x, x, x... 7, x, x, x, x, x, x, x, x, x, x... 8, x, x, x, x, x, x, x, x, x, x... 9, x, x, x, x, x, x, x, x, x, x... 10, x, x, x, x, x, x, x, x, x, x... 11, x, x, x, x, x, x, x, x, x, x...
18
-run advanced -matrix col -num_small 3# of tests = Xlarge + 1
© 2011-2012 Pythian
Data Points Matrix
Large/Small, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9... 0, x, x, x, x, x, x, x, x, x, x... 1, x, x, x, x, x, x, x, x, x, x... 2, x, x, x, x, x, x, x, x, x, x... 3, x, x, x, x, x, x, x, x, x, x... 4, x, x, x, x, x, x, x, x, x, x... 5, x, x, x, x, x, x, x, x, x, x... 6, x, x, x, x, x, x, x, x, x, x... 7, x, x, x, x, x, x, x, x, x, x... 8, x, x, x, x, x, x, x, x, x, x... 9, x, x, x, x, x, x, x, x, x, x... 10, x, x, x, x, x, x, x, x, x, x... 11, x, x, x, x, x, x, x, x, x, x...
19
-run advanced -matrix basic# of tests = Xlarge + Xsmall + 1
© 2011-2012 Pythian
Data Points Matrix
Large/Small, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 0, x, x, x, x, x, x, x, x, x x 1, x, x, x, x, x, x, x, x, x x 2, x, x, x, x, x, x, x, x, x x 3, x, x, x, x, x, x, x, x, x x 4, x, x, x, x, x, x, x, x, x x 5, x, x, x, x, x, x, x, x, x x 6, x, x, x, x, x, x, x, x, x x 7, x, x, x, x, x, x, x, x, x x 8, x, x, x, x, x, x, x, x, x x 9, x, x, x, x, x, x, x, x, x x 10, x, x, x, x, x, x, x, x, x x 11, x, x, x, x, x, x, x, x, x, x
20
-run advanced -matrix max# of tests = Xlarge + Xsmall + 1
© 2011-2012 Pythian
Data Points Matrix
Large/Small, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9... 0, x, x, x, x, x, x, x, x, x, x... 1, x, x, x, x, x, x, x, x, x, x... 2, x, x, x, x, x, x, x, x, x, x... 3, x, x, x, x, x, x, x, x, x, x... 4, x, x, x, x, x, x, x, x, x, x... 5, x, x, x, x, x, x, x, x, x, x... 6, x, x, x, x, x, x, x, x, x, x... 7, x, x, x, x, x, x, x, x, x, x... 8, x, x, x, x, x, x, x, x, x, x... 9, x, x, x, x, x, x, x, x, x, x... 10, x, x, x, x, x, x, x, x, x, x... 11, x, x, x, x, x, x, x, x, x, x...
21
-run advanced -matrix point -num_large 2 -num_small 3# of tests = 1
© 2011-2012 Pythian
Data Points Matrix
Large/Small, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9... 0, x, x, x, x, x, x, x, x, x, x... 1, x, x, x, x, x, x, x, x, x, x... 2, x, x, x, x, x, x, x, x, x, x... 3, x, x, x, x, x, x, x, x, x, x... 4, x, x, x, x, x, x, x, x, x, x... 5, x, x, x, x, x, x, x, x, x, x... 6, x, x, x, x, x, x, x, x, x, x... 7, x, x, x, x, x, x, x, x, x, x... 8, x, x, x, x, x, x, x, x, x, x... 9, x, x, x, x, x, x, x, x, x, x... 10, x, x, x, x, x, x, x, x, x, x... 11, x, x, x, x, x, x, x, x, x, x...
22
-run simple
© 2011-2012 Pythian
Data Points Matrix
Large/Small, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9... 0, x, x, x, x, x, x, x, x, x, x... 1, x, x, x, x, x, x, x, x, x, x... 2, x, x, x, x, x, x, x, x, x, x... 3, x, x, x, x, x, x, x, x, x, x... 4, x, x, x, x, x, x, x, x, x, x... 5, x, x, x, x, x, x, x, x, x, x... 6, x, x, x, x, x, x, x, x, x, x... 7, x, x, x, x, x, x, x, x, x, x... 8, x, x, x, x, x, x, x, x, x, x... 9, x, x, x, x, x, x, x, x, x, x... 10, x, x, x, x, x, x, x, x, x, x... 11, x, x, x, x, x, x, x, x, x, x...
23
-run normal
© 2011-2012 Pythian
Data Points Matrix
Large/Small, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9... 0, x, x, x, x, x, x, x, x, x, x... 1, x, x, x, x, x, x, x, x, x, x... 2, x, x, x, x, x, x, x, x, x, x... 3, x, x, x, x, x, x, x, x, x, x... 4, x, x, x, x, x, x, x, x, x, x... 5, x, x, x, x, x, x, x, x, x, x... 6, x, x, x, x, x, x, x, x, x, x... 7, x, x, x, x, x, x, x, x, x, x... 8, x, x, x, x, x, x, x, x, x, x... 9, x, x, x, x, x, x, x, x, x, x... 10, x, x, x, x, x, x, x, x, x, x... 11, x, x, x, x, x, x, x, x, x, x...
24
-run oltp
© 2011-2012 Pythian
Data Points Matrix
Large/Small, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9... 0, x, x, x, x, x, x, x, x, x, x... 1, x, x, x, x, x, x, x, x, x, x... 2, x, x, x, x, x, x, x, x, x, x... 3, x, x, x, x, x, x, x, x, x, x... 4, x, x, x, x, x, x, x, x, x, x... 5, x, x, x, x, x, x, x, x, x, x... 6, x, x, x, x, x, x, x, x, x, x... 7, x, x, x, x, x, x, x, x, x, x... 8, x, x, x, x, x, x, x, x, x, x... 9, x, x, x, x, x, x, x, x, x, x... 10, x, x, x, x, x, x, x, x, x, x... 11, x, x, x, x, x, x, x, x, x, x...
25
-run dss
© 2011-2012 Pythian
Orion I/O Performance Metrics
• Small IOs • iops - average number of IOs per second
• {test name}_{date}_{time}_iops.csv
• lat - average IO response time
• {test name}_{date}_{time}_lat.csv
• Large IOs • mbps - throughput MB per second
• {test name}_{date}_{time}_mbps.csv
26
© 2011-2012 Pythian
Sample for -matrix detailed
27
Large/Small, 1, 2, 3, 4, 5 0, 58, 114, 117, 127, 84 1, 11, 29, 49, 63, 81 2, 12, 23, 30, 24, 31
iops
Large/Small, 1, 2, 3, 4, 5 0, 17184.84, 17487.14, 25594.11, 31505.73, 59205.26 1, 88272.75, 66781.92, 60642.59, 62514.76, 61699.40 2, 80854.55, 83085.06, 99019.72, 155528.65, 156500.44
lat (us)
Large/Small, 0, 1, 2, 3, 4, 5 1, 18.35, 12.14, 15.99, 16.99, 16.48, 16.37 2, 29.74, 27.07, 25.19, 21.18, 13.04, 13.33
mbps
© 2011-2012 Pythian
Sample for -matrix basic
28
Large/Small, 1, 2, 3, 4, 5 0, 80, 153, 165, 163, 197 1 2
iops
Large/Small, 1, 2, 3, 4, 5 0, 12370.09, 13060.23, 18112.16, 24448.27, 25250.33 1 2
lat (us)
Large/Small, 0, 1, 2, 3, 4, 5 1, 31.84 2, 29.87
mbps
© 2011-2012 Pythian
Trace file contentran (small): VLun = 0 Size = 10737418240 ran (small): Index = 0 Avg Lat = 22996.61 us Count = 431 ran (small): Index = 1 Avg Lat = 23825.39 us Count = 417 ran (small): nio=848 nior=652 niow=196 req w%=25 act w%=23 ran (small): my 2 oth 1 iops 65 lat 26081 us, bw = 0.51 MBps dur 9.96 s size 8 K, min lat 932 us, max lat 227524 us READ ran (small): my 2 oth 1 iops 19 lat 14499 us, bw = 0.15 MBps dur 9.96 s size 8 K, min lat 1422 us, max lat 120529 us WRITE ran (small): my 2 oth 1 iops 85 lat 23404 us, bw = 0.66 MBps dur 9.96 s size 8 K, min lat 932 us, max lat 227524 us TOTAL
seq (large): VLun = 0 Size = 10737418240 seq (large): Index = 0 Avg Lat = 22038.99 us Count = 450 seq (large): Stream = 0 VLun = 0 Start = 2675965952 End = 3152019456 seq (large): Stream = 0 Avg Lat = 22038.99 us CIO = 1 NIO Count = 450 seq (large): nio=450 nior=450 niow=0 req w%=25 act w%=0 seq (large): my 1 oth 2 iops 45 lat 22039 us, bw = 45.22 MBps dur 9.95 s size 1024 K, min lat 9976 us, max lat 223534 us READ seq (large): my 1 oth 2 iops 0 lat 0 us, bw = 0.00 MBps dur 9.95 s size 1024 K, min lat 18446744073709551614 us, max lat 0 us WRITE seq (large): my 1 oth 2 iops 45 lat 22039 us, bw = 45.22 MBps dur 9.95 s size 1024 K, min lat 9976 us, max lat 223534 us TOTAL
29
© 2011-2012 Pythian
Concurrent I/O requests
= number of
outstanding I/
30
Separate process for
large and small
© 2011-2012 Pythian
Setting Scale of Concurrent I/Os
• Range of concurrency is {0..max} • unless specified with -num_small or -num_large or fixed by run type
• max for small IOs • num_disks * 5 for advanced, simple and normal runs
• num_disks * 20 for OLTP run
• max for large IOs • num_disks * 2 for advanced, simple and normal runs
• num_disks * 15 for DSS run
31
© 2011-2012 Pythian
OLTP and DSS runs are impractical*
• Range of concurrency is {0..max} • unless specified with -num_small or -num_large or fixed by run type
• max for small IOs • num_disks * 5 for advanced, simple and normal runs
• num_disks * 20 for oltp run
• max for large IOs • num_disks * 2 for advanced, simple and normal runs
• num_disks * 15 for dss run
32
20 steps with interval num_disks{num_disks..num_disks*20)
To much concurrency
15 steps with interval num_disks{num_disks..num_disks*15)
* 11.2.0.3 behavior
© 2011-2012 Pythian
Orion command-line syntax required arguments: -testname & -run
orion -testname {testname} \ -run advanced | normal | simple | oltp | dss \ -matrix detailed | col | row | basic | max | point \ -duration {seconds} \ -num_disks {disks} \ -num_large {num} \ -num_streamIO {num} \ -size_large {Kb} \ -type rand|seq \ -num_small {num} \ -size_small {Kb} \ -simulate concat|raid0 \ -stripe {Mb} \ -write {%} \ -cache_size {MB} \ -verbose
33
Defines input file with the list of disks {testname}.lun in the current directory# cat mytest.lun /dev/sdc /dev/sdd /dev/sde
© 2011-2012 Pythian
Orion command-line syntax -run normal
orion -testname {testname} \ -run advanced | normal | simple | oltp | dss \ -matrix detailed | col | row | basic | max | point \ -duration 60 \ -num_disks {disks} \ -num_large {num} \ -type rand \ -num_streamIO {num} \ -size_large 1024 \ -num_small {num} \ -size_small 8 \ -simulate concat \ -stripe 1 \ -write 0 \ -cache_size {MB} \ -verbose
34
this is preset this can’t be this can be set
© 2011-2012 Pythian
Orion command-line syntax -run simple
orion -testname {testname} \ -run advanced | normal | simple | oltp | dss \ -matrix detailed | col | row | basic | max | point \ -duration 60 \ -num_disks {disks} \ -num_large {num} \ -type rand \ -num_streamIO {num} \ -size_large 1024 \ -num_small {num} \ -size_small 8 \ -simulate concat \ -stripe 1 \ -write 0 \ -cache_size {MB} \ -verbose
35
this is preset this can’t be this can be set
© 2011-2012 Pythian
Orion command-line syntax -run oltp
orion -testname {testname} \ -run advanced | normal | simple | oltp | dss \ -matrix detailed | col | row | basic | max | point \ -duration {seconds} \ -num_disks {disks} \ -num_large 0 \ -type rand|seq \ -num_streamIO {num} \ -size_large {Kb} \ -num_small {num} \ -size_small {Kb} \ -simulate concat|raid0 \ -stripe {Mb} \ -write {%} \ -cache_size {MB} \ -verbose
36
this is preset this can’t be this can be set
© 2011-2012 Pythian
Orion command-line syntax -run dss
orion -testname {testname} \ -run advanced | normal | simple | oltp | dss \ -matrix detailed | col | row | basic | max | point \ -duration {seconds} \ -num_disks {disks} \ -num_large {num} \ -type rand|seq \ -num_streamIO {num} \ -size_large {Kb} \ -num_small 0 \ -size_small {Kb} \ -simulate concat|raid0 \ -stripe {Mb} \ -write {%} \ -cache_size {MB} \ -verbose
37
this is preset this can’t be this can be set
© 2011-2012 Pythian
Orion command-line syntax -run advanced -matrix detailed | basic | max
orion -testname {testname} \ -run advanced | normal | simple | oltp | dss \ -matrix detailed | col | row | basic | max | point \ -duration {seconds} \ -num_disks {disks} \ -num_large {num} \ -type rand|seq \ -num_streamIO {num} \ -size_large {Kb} \ -num_small {num} \ -size_small {Kb} \ -simulate concat|raid0 \ -stripe {Mb} \ -write {%} \ -cache_size {MB} \ -verbose
38
this can’t be this can be set
© 2011-2012 Pythian
Orion command-line syntax -run advanced -matrix col
orion -testname {testname} \ -run advanced | normal | simple | oltp | dss \ -matrix detailed | col | row | basic | max | point \ -duration {seconds} \ -num_disks {disks} \ -num_large {num} \ -type rand|seq \ -num_streamIO {num} \ -size_large {Kb} \ -num_small {num} \ -size_small {Kb} \ -simulate concat|raid0 \ -stripe {Mb} \ -write {%} \ -cache_size {MB} \ -verbose
39
this must be set this can’t be this can be set
© 2011-2012 Pythian
Orion command-line syntax -run advanced -matrix row
orion -testname {testname} \ -run advanced | normal | simple | oltp | dss \ -matrix detailed | col | row | basic | max | point \ -duration {seconds} \ -num_disks {disks} \ -num_large {num} \ -type rand|seq \ -num_streamIO {num} \ -size_large {Kb} \ -num_small {num} \ -size_small {Kb} \ -simulate concat|raid0 \ -stripe {Mb} \ -write {%} \ -cache_size {MB} \ -verbose
40
this must be set this can’t be this can be set
© 2011-2012 Pythian
Orion command-line syntax -run advanced -matrix point
orion -testname {testname} \ -run advanced | normal | simple | oltp | dss \ -matrix detailed | col | row | basic | max | point \ -duration {seconds} \ -num_disks {disks} \ -num_large {num} \ -type rand|seq \ -num_streamIO {num} \ -size_large {Kb} \ -num_small {num} \ -size_small {Kb} \ -simulate concat|raid0 \ -stripe {Mb} \ -write {%} \ -cache_size {MB} \ -verbose
41
this must be set this can’t be this can be set
© 2011-2012 Pythian
Orion command-line syntax -simulate raid0
orion -testname {testname} \ -run advanced | normal | simple | oltp | dss \ -matrix detailed | col | row | basic | max | point \ -duration {seconds} \ -num_disks {disks} \ -num_large {num} \ -type rand|seq \ -num_streamIO {num} \ -size_large {Kb} \ -num_small {num} \ -size_small {Kb} \ -simulate concat|raid0 \ -stripe {Mb} \ -write {%} \ -cache_size {MB} \ -verbose
42
Great way to simulate ASM
striping
© 2011-2012 Pythian
Orion command-line syntax -type seq
orion -testname {testname} \ -run advanced | normal | simple | oltp | dss \ -matrix detailed | col | row | basic | max | point \ -duration {seconds} \ -num_disks {disks} \ -num_large {num} \ -type rand|seq \ -num_streamIO {num default 4} \ -size_large {Kb} \ -num_small {num} \ -size_small {Kb} \ -simulate concat|raid0 \ -stripe {Mb} \ -write {%} \ -cache_size {MB} \ -verbose
43
© 2011-2012 Pythian
Orion Sequential I/O
44
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
1 1 1 1 2 2 2 2 3 3 3 3 4 4 4 4 5 5 5
-num_streamIO 1
-num_streamIO 4
Schedule one
IO request
and wait
Schedule o
ne
IO request
and wait
Schedule one
IO request and wait
Schedule fourIO requests and wait
Schedule fourIO requests and wait
Schedule fourIO requests and wait
© 2011-2012 Pythian
What I/O in Oracle behaves like -num_streamIO 4?
• Some examples: • serial direct parallel read
• ARCH reads of redo logs
• some operations with temporary segments
• How do you verify/know? • Enable 10046 trace and OS trace (strace/truss/tusc)
45
* needs verification *
© 2011-2012 Pythian
Orion Flexibility (Inflexibility?)
46
• Single Orion run is enough to assess scalability at defined settings
• Need several separate Orion runs to vary • write %
• large IO pattern
• IO size
• striping
• Need multiple concurrent runs to • simulate more complex IO patterns
• simulate RAC
© 2011-2012 Pythian
Scenarios: OLTP traffic
• -run advanced -matrix row -large_num 0 • Shadow processes’ “db file sequential reads”
• DBWR’s “db file parallel write”
• Optionally several runs with different settings like -write %
• Analyze IOPS & response time
47
© 2011-2012 Pythian
Scenarios: OLTP traffic visualization Oracle Database Appliance example
48
ODA: Small IOPS scalability / HDDs
IO R
espo
nse
Tim
e, m
s
0
5
10
15
20
25
Thro
ughp
ut, I
OPS
0
1 000
2 000
3 000
4 000
5 000
1 2 3 4 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100
IOPS Response Time
© 2011-2012 Pythian
Scenarios: OLTP traffic variation analysis Varying write percentage in ODA
49
Small IOPS by writes percentage Oracle Database Appliance / OLPT / whole HDDs
IO R
espo
nse
Tim
e, m
s
0
10
20
30
40
50
60
70
80
Thro
ughp
ut, I
OPS
0
1 000
2 000
3 000
4 000
5 000
6 000
7 000
Concurrent IO requests20 40 60 80 100 120 140 160 180 200 220 240 260 280 300 320 340 360 380 400
IOPS wrt 0% IOPS wrt 10% IOPS wrt 20% IOPS wrt 40% IOPS wrt 60%Latency wrt 0% Latency wrt 10% Latency wrt 20% Latency wrt 40% Latency wrt 60%
© 2011-2012 Pythian
Scenarios: OLTP traffic variation analysis Write percentage adjusted for ASM mirroring
50
Small IOPS by writes percentage Oracle Database Appliance / OLPT / whole HDDs
IO R
espo
nse
Tim
e, m
s
0
10
20
30
40
50
60
70
80
Thro
ughp
ut, I
OPS
0
1 000
2 000
3 000
4 000
5 000
6 000
7 000
Concurrent IO requests20 40 60 80 100 120 140 160 180 200 220 240 260 280 300 320 340 360 380 400
IOPS wrt 0% IOPS wrt 4% IOPS wrt 8% IOPS wrt 18% IOPS wrt 33%Latency wrt 0% Latency wrt 4% Latency wrt 8% Latency wrt 18% Latency wrt 33%
© 2011-2012 Pythian
Impact of writes on RAID5 is huge 40% writes => 4 times lower IOPS
51
© 2011-2012 Pythian
Same disks reconfigured as RAID1+0 40% writes => less than 50% hit
52
© 2011-2012 Pythian
Scenarios: Data Warehouse queries
• -run advanced -matrix col -small_num 0 • Keep read only (-write 0) • Concurrent users environment
• -type rand
• Single dedicated user performance • -type seq
• -num_streamIO 1 • Most reads in the DB are synchronous
• Analyze MBPS
53
© 2011-2012 Pythian
Scenarios: Data Warehouse IO visualization
54
Large IOs throughput
Thro
ughp
ut, M
BPS
0
75
150
225
300
Concurrent threads1 2 4 6 8 10 12 14 18 20 22 24 26 28 30 32
© 2011-2012 Pythian
Scenarios: RMAN backup
• -run advanced -matrix col -small_num 0 -type seq -num_streamIO 1
• Backup source only => -write 0 • Backup destination only => -write 100 • Database and backup destination combined => -write 50
• Watch for actual write percentage
• 1 thread => 0% actual writes
• 2 threads => 50% actual writes
• 3 threads => 33% actual writes
• 4 threads => 50% actual writes and etc...
• Analyze MBPS
55
© 2011-2012 Pythian
Scenarios: LGWR writes
• -run advanced -matrix point -small_num 0 -type seq -num_streamIO 1 -write 100 -num_large 1 -size_large 5 • -size_large should be set to average LGWR write size which is often
about 5-20k for OLTP systems
• -num_large n • multiple instances
• multiple LGWR threads in RAC
• redo logs multiplexing
• Analyze IOPS and response time • Gather from Orion run’s trace file
56
© 2011-2012 Pythian
Scenarios: LGWR writes visualization
57
ODA SSD sequential 32K IO streams (tripple mirroring)
Aver
age
Resp
onse
tim
e, m
s
0,00
0,20
0,40
0,60
0,80
1,00
Writ
es p
er s
econ
d
0
1600
3200
4800
6400
8000
Concurrent Threads2 4 6 8 10 12 14 16
IOPS Response Time, ms
© 2011-2012 Pythian
Combining different workloads
• Start multiple parallel Orion runs • OLTP -matrix point -num_large 0 -num_small X • LGWR -matrix point -num_large 1 -num_small 0 -write 100 • ARCH -matrix point -num_small 0 -write {0 | 50} • RMAN - matrix point -num_small 0 -write {0 | 50} • Add batch data load with large parallel writes • Add batch reporting (DW-like) with large reads
58
Cannot throttle IO other than controlling number of outstanding IOs
Cannot schedule a run with repetitive data points - must schedule multiple consecutive runs
© 2011-2012 Pythian
EC2 large 5 EBS disks: first run to test scalability
59
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25IOPS 156 326 178 411 532 729 928 1 103 1 023 1 070 964 1 202 1 285 1 232 1 204 1 245 1 352 1 338 1 360 1 149 1 379 1 327 1 334 1 362 1 363
Response time, ms
6,4 6,1 10,2 9,7 9,3 8,2 7,5 7,2 8,8 9,3 11,4 10 10,1 11,3 12,4 12,8 12,6 13,4 14 17,3 15,2 16,5 17,2 17,6 18,2
Initial OTLP test with 5 disks and 20% writes
Aver
age
resp
onse
tim
e, m
s
0
5
10
15
20
IOPS
0
350
700
1 050
1 400
Number of concurrent IOs1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
IOPS Response time, ms
© 2011-2012 Pythian
Let’s mix in additional I/O workloads
DUR=60
# OLTP test of scalability - original first run # /root/orion11203/orion -testname baseoltp -run advanced -duration $DUR \ -matrix row -num_large 0 -write 20
# OLTP point /root/orion11203/orion -testname oltp -run advanced -duration $DUR -matrix point \ -num_large 0 -num_small 10 -write 20 & # Adding LGWR /root/orion11203/orion -testname lgwr -run advanced -duration $DUR -matrix point \ -num_large 1 -num_small 0 -type seq -num_streamIO 1 -size_large 5 -write 100 & # Adding ARCH /root/orion11203/orion -testname arch -run advanced -duration $DUR -matrix point \ -num_large 2 -num_small 0 -type seq -num_streamIO 1 -size_large 1024 -write 50 & # Backup in 1 channel # /root/orion11203/orion -testname backup -run advanced -duration $DUR -matrix point \ -num_large 1 -num_small 0 -type seq -num_streamIO 1 -size_large 1024 -write 0 & # Backup in 4 channels # /root/orion11203/orion -testname backup -run advanced -duration $DUR -matrix point \ -num_large 4 -num_small 0 -type seq -num_streamIO 1 -size_large 1024 -write 0 &
wait
60
© 2011-2012 Pythian61
OLTP IOPS Response time, ms
LGWR writes LGWR write, ms
OLTP only 1306 7,7OLTP +LGWR 1239 8,1 139 7,1
OLTP+LGWR+ARCH 576 17,4 17 56OLTP+LGWR+RMAN1 778 12,8 38 26,1OLTP+LGWR+RMAN4 571 17,5 49 20,3
Resp
onse
tim
e, m
s
0
4
8
12
16
20
IOPS
0
300
600
900
1200
1500
OLTP onlyOLTP +LGWR
OLTP+LGWR+ARCH
OLTP+LGWR+RMAN1
OLTP+LGWR+RMAN4
OLTP IOPS Response time, ms
LGW
R w
rite,
ms
0
15
30
45
60
75
LGW
R w
rites
per
sec
ond
0
30
60
90
120
150
LGWR writes LGWR write, ms
EC2... visualizing combined workload impact
© 2011-2012 Pythian
The best Orion 11.2 new feature
62
Bucket LGWR no ARCH
LGWR with ARCH
0 - 128 0 0 128 - 256 0 0 256 - 512 0 0 512 - 1024 1085 1 1024 - 2048 3376 8 2048 - 4096 395 1 4096 - 8192 845 0 8192 - 16384 1406 2 16384 - 32768 1115 161 32768 - 65536 161 699 65536 - 131072 4 169 131072 - 262144 0 17 262144 - 524288 1 10 524288 - 1048576 0 21048576 - 2097152 0 1
no ARCH
0 -
128
128
- 256
256
- 512
512
- 102
4
1
024
- 204
8
2
048
- 409
6
4
096
- 819
2
8
192
- 163
84
1
6384
- 32
768
3
2768
- 65
536
6
5536
- 13
1072
13
1072
- 26
2144
26
2144
- 52
4288
52
4288
- 10
4857
6
1048
576
- 209
7152
with ARCH
Histograms!
© 2011-2012 Pythian
Got RAC? Schedule parallel runs on each node
63
HP bladesHP Virtual ConnectFlex10Big NetApp box100 disks
© 2011-2012 Pythian
Example of Failed Expectations NetApp NAS, 1 Gbit Ethernet, 42 disks
64
Late
ncy,
ms
0,0
7,5
15,0
22,5
30,0
IOPS
0
1000
2000
3000
4000
5000
1 2 3 4 5 10 20 30 40 50 60 70 80 90 100
IOPS Latency
Read only
Late
ncy,
ms
0
10
20
30
40
50
IOPS
0
1000
2000
3000
4000
5000
1 2 3 4 5 10 20 30 40 50 60 70 80 90 100
Read write
© 2011-2012 Pythian
Tune-Up Results Switched from Intel to Broadcom NICs and disabled
65
Late
ncy,
ms
0
2
4
6
8
10
12
IOPS
0
2000
4000
6000
8000
10000
1 2 3 4 5 10 20 30 40 50 60 70 80 90 100
IOPS Latency
Late
ncy,
ms
0
2
4
6
8
IOPS
0
2500
5000
7500
10000
12500
15000
1 2 3 4 5 10 20 30 40 50 60 70 80 90 100
© 2011-2012 Pythian
Possible “What-If” scenarios
66
• Impact of a failed disk in a RAID group • Different block size • Different ASM allocation unit size (-stripe) • Assess foreign workload impact (shared SAN with other servers)
• Test impact of configuration / infrastructure changes • Impact of backup or a batch job • Impact of decreased MTTR target (higher -write %) • Platform stability test (repeating the same data point for many days)
• Impact of CPU starvation
© 2011-2012 Pythian
Concurrent IOs on axis X is not always the best...
67
ODA: Small IOPS scalability and data placement / HDDs
IO R
espo
nse
Tim
e, m
s
0
5
10
15
20
25
Thro
ughp
ut, I
OPS
0
1 200
2 400
3 600
4 800
6 000
1 2 3 4 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100
IOPS whole disk IOPS outside 40% IOPS inside 60%Latency whole disk Latency outside 40% Latency inside 60%
© 2011-2012 Pythian
Smarter presentation 50% IOPS at the same response time
68
ODA: Improving IO throughput by data placement
IOPS
0
1200
2400
3600
4800
6000
IO response time0 5 10 15 20 25
whole diskoutside 40%inside 60%
© 2011-2012 Pythian
Storage types
• Anything as long as ASYNC IO is supported
• Local storage (LUNs or filesystem) • NAS via NFS • iSCSI / FC devices (any block or raw device) • Cluster filesystem should work just fine
69
© 2011-2012 Pythian
Beware of thin provisioning and other NAS magic
• Smart storage technologies play bad jokes • If in doubt - “initialize” disks with non-zeroes
70
© 2011-2012 Pythian
Orion 11.2
71
11.2.0.1 11.2.0.2 11.2.0.3
libcell11.so x x x
libclntsh.so.11.1 x x
libskgxp11.so x x
libnnz11.so x x
Included in • Database
• Grid home
• Client (tested Administrative option)
Dependencies
© 2011-2012 Pythian
Orion with SLOB (Silly Little Oracle Benchmark)
• Orion gives more control • Orion is easier to setup • Orion uses very little CPU - it doesn’t do anything with data • Easier to saturate IO subsystem without CPU starvation
• Less realistic results if you want to account database CPU use for LIO and processing the data
• Less realistic for multiprocess orchestration
• SLOB - is more realistic but more difficult to control
72
© 2011-2012 Pythian73
Visualization is the Key
© 2011-2012 Pythian
Thank you and Q&A
74
http://www.pythian.com/news/
http://www.facebook.com/pages/The-Pythian-Group/
http://twitter.com/pythian
http://www.linkedin.com/company/pythian
1-866-PYTHIAN
[email protected] or [email protected]
To contact us…
To follow us… [email protected]
