Performance Tuning Couchbase: Couchbase Connect 2014

Performance Tuning CouchbaseKirk Kirkconnell | Sr. Solutions Engineer | Couchbase

@cbcollect_info

©2014 Couchbase, Inc. 2

A Properly Sized ClusterTuning CouchbaseTuning the Operating SystemAdditional Resources

Outline

A Properly Sized ClusterGood PerformanceStarts with Sizing


Size server resources for ~3x your total data set on average. Take into account:

• Replicas• Compaction• Views and Indexes• XDCR• Maintenance

Number of Nodes• Distributes data and utilizes more resources across additional servers• Scale out is the preferred strategy

General Sizing


Size IO for your workload• Heavy writes, view queries = lean towards SSDs and eSSDs • Heavy reads, stable working set = HDD is good, SSD is better• Consistent performance is most important• Ensure overhead for maintenance tasks

RAM Sizing• Have sufficient RAM for your working set• Have enough RAM for your OS• XDCR needs more RAM, but depends on number of target clusters• If you use indexes/views, leave enough for OS page cache• Factor in meta data

General Sizing


Disk Considerations• Spinning disks• SSD• FusionIO

How Many Cores?• Have at least 4 cores for Couchbase• More buckets, usually means more Cores• One more core per design document• Two more cores per XDCR stream

General RAM Sizing


Bandwidth• Throughput• Rebalance• Intra-Cluster Replication• Inter-Cluster Replication (XDCR)• Backups

Latency• 1GbE• 10GbE• Infiniband

A Properly Sized Cluster


Replicas• The number of replicas impacts overall performance• For each replica there is more network, disk, RAM, etc. used.• If you need more replicas, you must size for them.

A Properly Sized Cluster


Do not over-subscribe resources, specifically:• Disk I/O• Network• CPU

Avoid VM migrations if you can (e.g. vMotion)• Manually manage migrations

Monitor the hypervisor as well as the guest OS if you canUse pass-through devices for disk to guarantee resources

Virtualization Considerations


• Think of sizing as an ongoing activity• Monitor your cluster’s performance and incorporate sizing into your

continuous improvement process

Sizing Does Not End

Tuning CouchbaseSome of the Internals


• Some like to say Couchbase is schemaless, but more so it is flexible• Prioritize your schema around the “hot path” of the app

• How does your app access data?• Which parts have to perform the fastest?• Replicating around large objects takes more resources obviously• This also contributes to scalability

• Get the document by key as much as possible for best performance• Key naming – you have 256 characters, use them wisely

• For best performance, minimize use of views• If you are using indexes/views, group similar documents in the same bucket

to avoid indexing too much unrelated data

Schema Design Matters


In 2.x, tune the number of Read/Write threads per bucket

• Best to do when you create the bucket

In 2.x, tune working balance of reader/writer threads• Default is weighted heavily towards reader threads• The other options weigh more towards writing or 50/50 mix

Tuning Thread and Resource Usage


In 3.x, GTP makes manual thread tuning is no longer necessary

Bucket Priorities• Balances I/O priorities of buckets• By default every bucket is set to low• Only kicks in when one or more bucket are set differently• So if all buckets are set to low or all set to high, all get equal resources

Couchbase 3.0


Three area to look at• Change number of vBuckets moved concurrently• Turn off index aware rebalancing• Disable view compaction during rebalance

Change before maintenance window. When complete, change settings back.Be conservative or you could end up drowning out your application

Have enough RAM to avoid disk backfills, otherwise you have lots of disk I/O

Only do this if you have the cluster resources and test these first.

Tuning Rebalances


Auto-Compaction


Look to raise the thresholds to 60% if you need it• Compaction runs less often, thus lowering I/O utilization• Increases disk space requirements. Depending on your workload this

could be dramatically more• If possible, schedule this for off-peak hours

• Also increases disk space requirements

Auto-Compaction


• Set the maximum parallel indexers• maxParallelIndexers

• Configure automated index updates• updateInterval• updateMinChanges• replicaUpdateMinChanges

• Page Cache buffers indexes• Set the maximum parallel indexers

• maxParallelIndexers

Views


XDCR

Tuning the Operating SystemSome advanced stuff


• OS Page Cache• Best Practice = 20%. More if you have large sized indexes/views

• Deprioritize swap• vm.swappiness = 0

• Turn off Transparent Huge Pages• Limit page cache dirty bytes

• vm.dirty_bytes• vm.dirty_background_bytes

• Control the rate page cache is flushed• vm.dirty_expire_centisecs• vm.dirty_writeback_centisecs

Memory


• File Systems• EXT4• XFS

• Mount Options• EXT4: noatime,barrier=0,data=writeback• XFS: noatime,nobarrier,logbufs=8,logbsize=256k,allocsize=2M

• IO Scheduler• Deadline scheduler• Queue depth increase from 128 to 1024

Disk


• Increase backlogs• net.core.netdev_max_backlog• net.core.somaxconn• net.ipv4.tcp_max_syn_backlog• net.ipv4.tcp_fin_backlog

• Increase buffer limits• net.core.rmem_max• net.core.wmem_max• net.ipv4.tcp_mem• net.ipv4.tcp_rmem• net.ipv4.tcp_wmem

Network Resilience


• Reduce TCP overhead• net.ipv4.tcp_sack• net.ipv4.tcp_fack

• Reduce connection overheat• net.ipv4.tcp_fin_timeout = decrease• net.ipv4.tcp_tw_reuse = enable

• Find failed connections faster• net.ipv4.tcp_keepalive_intvl

• Enable auto-tuning• net.ipv4.tcp_moderate_rcvbuf• net.ipv4.tcp_window_scaling

Network Efficiency

Evaluation SuccessLoad Testing Tools


• Pillowfight – based on libcouchbase (C SDK)• Roadrunner – based on Java SDK• Cbworkloadgen – Python + libcouchbase

Load Testing Tools


Again, advanced tuning only helps if your Couchbase cluster is properly sized!

http://docs.couchbase.com/couchbase-manual-2.5/cb-admin/#sizing-guidelines

http://blog.couchbase.com/how-many-nodes-part-1-introduction-sizing-couchbase-server-20-cluster

Resources

http://docs.couchbase.com/couchbase-manual-2.5/cb-admin/%23sizing-guidelines








With great power comes great responsibility.

Performance tuning is often a trade-off between safety and performance.

Understand and test before you deploy.

Advice #2

Performance Tuning Couchbase: Couchbase Connect 2014

Data & Analytics

Transcript of Performance Tuning Couchbase: Couchbase Connect 2014