Tuning ZFS on FreeBSD
description
Transcript of Tuning ZFS on FreeBSD
Tuning ZFS on FreeBSD
Martin [email protected]
EuroBSDCon 201221.10.2012
Martin Matuska [email protected]
Tuning ZFS on FreeBSD
ZFS is a modern 128-bit open-source operating system utilizingthe copy-on-write model.
This presentation is going to cover the following topics:
I How can we tune ZFS?
I When should ZFS be tuned?
Martin Matuska [email protected]
Tuning ZFS on FreeBSD
Is ZFS slow?
Help, my ZFS is slow!
I compared to ... ?
I this depends on many factors (workload, data access, ...)
I tradeoff speed vs data consistency + features
I maybe auto-tuning does not well in your case ...
Martin Matuska [email protected]
Tuning ZFS on FreeBSD
Think twice about what you do
From blogs about optimizing ZFS:
Disable the unwanted featuresBy default, ZFS enables a lot of settings for data security, such aschecksum etc. If you don’t care about the additional data security,just disable them.http://icesquare.com/wordpress/how-to-improve-zfs-performance
A note on disabling ZFS checksum: don’t.http://v-reality.info/2010/06/using-nexentastor-zfs-storage-appliance-with-vsphere
Martin Matuska [email protected]
Tuning ZFS on FreeBSD
ZFS checksum
We need the checksums to do:
I data and metadata integrity verification
I self-healing (scrub, mirror, raidz)
Martin Matuska [email protected]
Tuning ZFS on FreeBSD
General tuning tips
I System memory
I Access time
I Dataset compression
I Deduplication
I ZFS send and receive
Martin Matuska [email protected]
Tuning ZFS on FreeBSD
Random Access Memory
ZFS performance and stability depends on the amount of systemRAM.
I recommended minimum: 1GB
I 4GB is ok
I 8GB and more is good
Martin Matuska [email protected]
Tuning ZFS on FreeBSD
Access time
Due to copy-on-write enabled access time:
I reduces performance on read-intensive workloads
I increases space used by snapshots
# zfs set atime=off dataset
Martin Matuska [email protected]
Tuning ZFS on FreeBSD
Dataset compression
Dataset compression does:
I save space (LZJB less, gzip more)
I increase CPU usage (LZJB less, gzip much more)
I increase data throughput (LZJB, relative)
Therefore I recommend using compression primarily for archivingpurposes (e.g. system or webserver logfiles)
# zfs set compression=[on|off|gzip]
Martin Matuska [email protected]
Tuning ZFS on FreeBSD
Deduplication
Deduplication saves space by keeping just a single copy of identicaldata blocks. But remember, that deduplication:
I requiries even more memory (fast only if table fits to RAM)
I increases CPU usage
Command to simulate the effect of enabling deduplication:# zdb -S pool
Command for viewing detailed deduplication information:# zdb -D pool# zdb -DD pool
Martin Matuska [email protected]
Tuning ZFS on FreeBSD
ZFS send and receive
I higly recommend using a intermediate buffering solution forsending and receiving large streams:
I misc/buffer
I misc/mbuffer (network capable)
Martin Matuska [email protected]
Tuning ZFS on FreeBSD
Application Tuning Tips
We are going to look how to optimize the following applications forZFS:
I Web servers
I Database servers
I File servers
Martin Matuska [email protected]
Tuning ZFS on FreeBSD
Webservers
As of the current ZFS implementation it is recommended todisable sendfile and mmap to avoid redundant data caching.
I ApacheEnableMMAP OffEnableSendfile Off
I NginxSendfile off
I Lighttpdserver.network-backend="writev"
Martin Matuska [email protected]
Tuning ZFS on FreeBSD
Database servers
For PostgreSQL and MysSQL users recomment using a differentrecordsize than default 128k.
I PostgreSQL: 8k
I MySQL MyISAM storage: 8k
I MySQL InnoDB storage: 16k
# zfs create -o recordsize=8k tank/mysql
Martin Matuska [email protected]
Tuning ZFS on FreeBSD
File Servers
Here are some tips for file servers:
I disable access time
I keep number of snapshots low
I dedup only of you have lots of RAM
I for heavy write workloads move ZIL to separate SSD drives
I optionally disable ZIL for datasets (beware consequences)
Martin Matuska [email protected]
Tuning ZFS on FreeBSD
Cache and Prefetch Tuning
We are going to look at the following:
I Adaptive Replacement Cache (ARC)
I Level 2 Adaptive Replacement Cache (L2ARC)
I ZFS Intent Log (ZIL)
I File-level Prefetch (zfetch)
I Device-level Prefetch (vdev prefetch)
I ZFS Statistics Tools
Martin Matuska [email protected]
Tuning ZFS on FreeBSD
Adaptive Replacement Cache 1/2
ARC resides in system RAM, provides major speedup to ZFS andits size is auto-tuned.
Default values are:
I maximum: physical RAM less 1GB (or 1/2 of all memory)
I metadata limit: arc meta limit = 1/4 of arc max
I minimum: 1/2 of arc meta limit (but at least 16MB)
Martin Matuska [email protected]
Tuning ZFS on FreeBSD
Adaptive Replacement Cache 2/2
How to tune the ARC:
I you can disable ARC on per-dataset level
I maximum can be limited to reserve memory for other tasks
I increasing arc meta limit may help if working with many files
# sysctl kstat.zfs.misc.arcstats.size# sysctl vfs.zfs.arc meta used# sysctl vfs.zfs.arc meta limit
Martin Matuska [email protected]
Tuning ZFS on FreeBSD
Level 2 Adaptive Replacement Cache 1/2
Some facts about L2ARC:
I is designed to run on fast block devices (SSD)
I helps primarily read-intensive workloads
I each device can be attached to only one ZFS pool
# zpool add pool cache device# zpool remove pool device
Martin Matuska [email protected]
Tuning ZFS on FreeBSD
Level 2 Adaptive Replacement Cache 2/2
How to tune the L2ARC:
I enable prefetch for streaming or serving of large files
I configurable on per-dataset basis
I turbo warmup phase may require tuning (e.g. set to 16MB)
vfs.zfs.l2arc noprefetchvfs.zfs.l2arc write maxvfs.zfs.l2arc write boost
Martin Matuska [email protected]
Tuning ZFS on FreeBSD
ZFS Intent Log
The ZFS Intent Log (ZIL)
I guarantees data consistency on fsync() calls
I replays transactions in case of a panic or power failure
I uses small storage space on each pool by default
To speed up writes, you can deploy ZIL on a separate log device.Per-dataset synchronicity behaviour can be configured.
# zfs set sync=[standard|always|disabled]dataset
Martin Matuska [email protected]
Tuning ZFS on FreeBSD
File-level Prefetch (zfetch)
File-level prefetching
I analyses read patterns of files
I tries to predict next reads
I goal: reduce application response times
Loader tunable to enable/disable zfetch:vfs.zfs.prefetch disable
Martin Matuska [email protected]
Tuning ZFS on FreeBSD
Device-level Prefetch (vdev prefetch)
Device-level prefetching
I reads data after small reads from pool devices
I may be useful for drives with higher latency
I consumes constant RAM per vdev
I is disabled by default
Loader tunable to enable/disable vdev prefetch:vfs.zfs.vdev.cache.size=[bytes]
Martin Matuska [email protected]
Tuning ZFS on FreeBSD
ZFS Statistics Tools
ZFS statistical data is provided by# sysctl vfs.zfs# sysctl kstat.zfs
This data can help to make tuning decisions.I have prepared tools to view and analyze this data:
I zfs-stats: analyzes settings and counters since boot
I zfs-mon: real-time statistics with averages
Both tools are available in ports under sysutils/zfs-stats
Martin Matuska [email protected]
Tuning ZFS on FreeBSD
zfs-stats: overview
The zfs-stats utility is based on Ben Rockwood’s arc-summary.pland includes modifications by Jason J. Hellenthal and myself.
It provides information about:
I ARC structure and efficiency
I L2ARC structure and efficiency
I ZFETCH efficiency
I values of ZFS tunables
I system memory (overview)
Martin Matuska [email protected]
Tuning ZFS on FreeBSD
zfs-stats: sample output excerpt
ARC Size: 79.89% 25.57 GiBTarget Size: (Adaptive) 79.89% 25.57 GiBMin Size (Hard Limit): 12.50% 4.00 GiBMax Size (High Water): 8:1 32.00 GiB
ARC Efficiency: 1.25bCache Hit Ratio: 90.52% 1.13bCache Miss Ratio: 9.48% 118.08mActual Hit Ratio: 84.54% 1.05b
Data Demand Efficiency: 95.45% 356.90mData Prefetch Efficiency: 40.64% 11.36m
L2 ARC Breakdown: 118.18mHit Ratio: 62.87% 74.29mMiss Ratio: 37.13% 43.89mFeeds: 849.64k
File-Level Prefetch: (HEALTHY)
DMU Efficiency: 28.09bHit Ratio: 88.54% 24.87b
Martin Matuska [email protected]
Tuning ZFS on FreeBSD
zfs-mon: overview
The zfs-mon utility
I polls ZFS counters in real-time
I analyzes ARC, L2ARC, ZFETCH and vdev prefetch
I displays absolute and relative values
I displays output in varnishstat(1) style
Martin Matuska [email protected]
Tuning ZFS on FreeBSD
ZFS real-time cache activity monitorSeconds elapsed: 120
Cache hits and misses:1s 10s 60s tot
ARC hits: 259 431 418 466ARC misses: 51 40 49 52
ARC demand data hits: 223 417 390 437ARC demand data misses: 36 20 17 16
ARC demand metadata hits: 36 11 25 25ARC demand metadata misses: 15 19 21 25
ARC prefetch data hits: 0 4 3 4ARC prefetch data misses: 0 1 10 8
ARC prefetch metadata hits: 0 0 0 0ARC prefetch metadata misses: 0 0 1 3
L2ARC hits: 47 34 40 37L2ARC misses: 4 5 9 15ZFETCH hits: 47903 47294 48155 47138
ZFETCH misses: 272 449 1147 3593
Cache efficiency percentage:10s 60s tot
ARC: 91.51 89.51 89.96ARC demand data: 95.42 95.82 96.47
ARC demand metadata: 36.67 54.35 50.00ARC prefetch data: 80.00 23.08 33.33
ARC prefetch metadata: 0.00 0.00 0.00L2ARC: 87.18 81.63 71.15
ZFETCH: 99.06 97.67 92.92
Martin Matuska [email protected]
Tuning ZFS on FreeBSD