Assumptions About Human Behavior : A historical Perspective and Research Method
What Assumptions Make: Filesystem I/O from a database perspective
94
What Assumptions Make: Performance Testing with P4 Portland PostgreSQL Performance Pad Selena Deckelmann [email protected] End Point Corporation twitter: @selenamarie
-
Upload
selena-deckelmann -
Category
Technology
-
view
3.671 -
download
0
description
The Linux operating system provides a number of file systems that can be used, as well as volume management and hardware or software RAID. We are running performance benchmarks for database tuning, and are curious if the file systems really behave like we expect them to, especially when used in conjunction with RAID or volume management. Are these file systems being used in manners for which they were designed? There is also more to file systems than how fast we can read to them or how fast we can write to them. How reliable is the file system, and how do be prove it? We have collected data and will have a server available for development during the conference.
Transcript of What Assumptions Make: Filesystem I/O from a database perspective
What Assumptions Make:Performance Testing with P4
Portland PostgreSQL Performance Pad
Selena [email protected] Point Corporationtwitter: @selenamarie
www.endpoint.com
SCALE 7x Feb 21-22, 2009
Do filesystems do what we expect?
5
www.endpoint.com
SCALE 7x Feb 21-22, 2009
We are volunteers.
6
www.endpoint.com
SCALE 7x Feb 21-22, 2009
We think you should run these tests.
7
www.endpoint.com
SCALE 7x Feb 21-22, 2009
We are:DBAs
SysadminsPerformance tuners
8
www.endpoint.com
SCALE 7x Feb 21-22, 2009
How will this hardware perform?
9
www.endpoint.com
SCALE 7x Feb 21-22, 2009
How will this filesystem perform?
10
www.endpoint.com
SCALE 7x Feb 21-22, 2009
Why should you care about filesystem-specific
performance?
11
www.endpoint.com
SCALE 7x Feb 21-22, 2009
Expectations
12
www.endpoint.com
SCALE 7x Feb 21-22, 2009
PERSONAL CONFESSION
13
www.endpoint.com
SCALE 7x Feb 21-22, 2009
Where to start?
14
www.endpoint.com
SCALE 7x Feb 21-22, 2009
The Defaults.
15
www.endpoint.com
SCALE 7x Feb 21-22, 2009
17
Not addressing reliability
www.endpoint.com
SCALE 7x Feb 21-22, 2009
18
Very Narrow Use Case:A Relational Database
www.endpoint.com
SCALE 7x Feb 21-22, 2009
19
Need for periodic testing.(And we've got some
hardware!)
www.endpoint.com
SCALE 7x Feb 21-22, 2009
20
★Kernel differences★FS patch-level differences★Mount options★mkfs options
www.endpoint.com
SCALE 7x Feb 21-22, 2009
21
Focused on THROUGHPUT
(Because that’s what people who buy large systems look for)
www.endpoint.com
SCALE 7x Feb 21-22, 2009
22
Later:Response Time
Operations per second
www.endpoint.com
SCALE 7x Feb 21-22, 2009
No, we will not be testing ZFS.
23
www.endpoint.com
SCALE 7x Feb 21-22, 2009
BtrFS (nope, not yet)
24
FS
www.endpoint.com
SCALE 7x Feb 21-22, 2009
What do we expect?
25
www.endpoint.com
SCALE 7x Feb 21-22, 2009
Some conventional wisdom:
26
www.endpoint.com
SCALE 7x Feb 21-22, 2009
“RAID5 is the worst choice
for a database.”
27
www.endpoint.com
SCALE 7x Feb 21-22, 2009
“LVM incurstoo much overhead
to use.”
28
www.endpoint.com
SCALE 7x Feb 21-22, 2009
“Striping doubles performance.”
29
www.endpoint.com
SCALE 7x Feb 21-22, 2009
“Turning off 'atime'is a big
performance gain.”
30
www.endpoint.com
SCALE 7x Feb 21-22, 2009
31
“Getting rid of atime updates would give us more everyday Linux performance than all the pagecache speedups of the last 10 years, _combined_.”
www.endpoint.com
SCALE 7x Feb 21-22, 2009
“Journaling filesystems (ext3) will have worse performance than non-journaling filesystems
(ext2).”
32
www.endpoint.com
SCALE 7x Feb 21-22, 2009
“Your read-ahead buffer
is big enough.”
33
www.endpoint.com
SCALE 7x Feb 21-22, 2009
Now... on to the good stuff.
34
www.endpoint.com
SCALE 7x Feb 21-22, 2009
PostgreSQL’s Portland Performance Pad
36
Hosted by CommandPrompt, Inc.
www.endpoint.com
SCALE 7x Feb 21-22, 2009
Our machine:
HP ProLiant DL380G5Smart Array p800
72GB 15,000 RPM SAS (up to 25 disks)32GB RAM
Linux: 2.6.25-gentoo-r6*New tests being run with 2.6.28
37
www.endpoint.com
SCALE 7x Feb 21-22, 2009
Our machine:Chosen because
of it’s low, low price.
Thank you, HP.
38
www.endpoint.com
SCALE 7x Feb 21-22, 2009
Our tests:fio64 GB working set8 threadsno fadviseno direct i/o8KB blocksizeI/O elevator: deadline
39
www.endpoint.com
SCALE 7x Feb 21-22, 2009
Our stats:sarmpstatiostatvmstatreadprofile
40
www.endpoint.com
SCALE 7x Feb 21-22, 2009
Our tests:Chosen because of their relevance to PostgreSQL
41
www.endpoint.com
SCALE 7x Feb 21-22, 2009
Filesystems Tested:
ext2ext3jfsxfsreiserfsext4 (but having trouble)
42
www.endpoint.com
SCALE 7x Feb 21-22, 2009
Disk configs tested:
Single diskRAID-0RAID-1RAID-5RAID-10RAID-6
43
www.endpoint.com
SCALE 7x Feb 21-22, 2009
The Data:http://moourl.com/fsperf
44
www.endpoint.com
SCALE 7x Feb 21-22, 2009
Confessions:• May be high standard deviation with results (don’t know yet!)
•No filesystem tuning, all default create and mount options
•No software raid comparison or lvm (volume management test) for 2.6.28 tests
45
www.endpoint.com
SCALE 7x Feb 21-22, 2009
Confessions:• Some xfs runs had to be repeated and some ext4 runs did not complete successfully
• Only presenting throughput
• Interested in system performance for a specific application, not code performance
46
www.endpoint.com
SCALE 7x Feb 21-22, 2009
Confessions:•I/O profiles don’t exhibit atime or partition alignment issues
•Disk controller firmware not at the latest version in 2.6.25 tests
•Software RAID is on top of 1 disk RAID 0 devices (HP SmartArray doesn’t have JBOD option)
47
www.endpoint.com
SCALE 7x Feb 21-22, 2009
AUDIENCE PARTICIPATIONHigher throughput:
ext2 or ext3?
48
www.endpoint.com
SCALE 7x Feb 21-22, 2009
Seek bundling/batching in ext3 is better?
53
www.endpoint.com
SCALE 7x Feb 21-22, 2009
What if we add a disk?
54
www.endpoint.com
SCALE 7x Feb 21-22, 2009
AUDIENCE PARTICIPATIONRAID 0 (stripe) versus
RAID 1 (mirroring)performance?
60
www.endpoint.com
SCALE 7x Feb 21-22, 2009
What happens when we: add disks to a
RAID 0 (stripe) LUN?
64
www.endpoint.com
SCALE 7x Feb 21-22, 2009
Adding disks to a RAID 5 LUN
69
www.endpoint.com
SCALE 7x Feb 21-22, 2009
Only have 4 disks?What should you do?
73
www.endpoint.com
SCALE 7x Feb 21-22, 2009
In most cases, RAID 5 out performs on sequential writes (xlog).
Random writes is only an improvement on xfs and reiserfs.
78
www.endpoint.com
SCALE 7x Feb 21-22, 2009
Are software RAID and LVM are slow?
79
www.endpoint.com
SCALE 7x Feb 21-22, 2009
The Read-ahead buffer
82
www.endpoint.com
SCALE 7x Feb 21-22, 2009
AUDIENCE PARTICIPATIONReadahead buffer:
Default is 128 KWhat do you think it should be?
83
www.endpoint.com
SCALE 7x Feb 21-22, 2009
And is there a cost to increasing the buffer
that much?
85
www.endpoint.com
SCALE 7x Feb 21-22, 2009
http://moourl.com/readaheadconfirm
87
www.endpoint.com
SCALE 7x Feb 21-22, 2009
Future Work•OLTP system characterization, sizing
•Daily OLTP regression testing•More presentations•P5 - PostgreSQL Portland Performance Pad PRACTICE
88
www.endpoint.com
SCALE 7x Feb 21-22, 2009
MOAR Hardware?
Thanks again, HP!MSA70, DL380 in 2009 ??
89
www.endpoint.com
SCALE 7x Feb 21-22, 2009
Let’s recap...
90
www.endpoint.com
SCALE 7x Feb 21-22, 2009
“RAID5 is the worst choice for a database.” Fast for sequential writes in our tests.
“LVM incurs too much overhead to use. Software RAID is slower.” For reads – throughput is about the same, but saw higher CPU.
“Turning off 'atime' is a big performance gain.” Not in our tests. But, 2-3% for “free”.
91
www.endpoint.com
SCALE 7x Feb 21-22, 2009
“Journaling filesystems will have worse performance than non-journaling filesystems.” Turn the data journaling off on ext3, and you do see better performance, but there are edge cases and performance differences we could not explain.
“Striping doubles performance.” Performance is better, but no where near double. Why?
92
www.endpoint.com
SCALE 7x Feb 21-22, 2009
“Your read-ahead buffer is big enough.” Your read-ahead buffer IS NOT big enough. Make it 8MB. And can we make that the default?
93
www.endpoint.com
SCALE 7x Feb 21-22, 2009
Thank you!
94
Results: http://wiki.postgresql.org/wiki/
HP_ProLiant_DL380_G5_Tuning_Guide
http://moourl.com/fsperf
Selena [email protected]: @selenamarie
