Benchmarking RedisBy itself and versus other NoSQL databases

@ItamarHaber #RedisConf 2015

About Myself

A Redis Geek and Chief Developers Advocate at

Have you subscribed to the newsletter yet?

https://redislabs.com/redis-watch-archive

define:benchmark

"In computing, a benchmark is the act of running a computer program, a set of programs, or other operations, in order to assess the relative performanceof an object, normally by running a number of standard tests and trials against it."

https://en.wikipedia.org/wiki/Benchmark_(computing)

The Only Truism About Benchmarking

The chances that any benchmark's results will accurately reflect the performance of your specific use case are infinitesimal*

* single exception: benchmarks that you run on your own application <- please wait patiently for my guest speaker

So Why Benchmark Redis at all?

•Quantify Blazing Fast with some ballpark figures

•Regression testing/optimizations verifications/sanity

•Comparative benchmark(et)ing

• Fun & profit

Howto #1: redis-benchmark

• Included with Redis (build it with make)

•Built-in suite of tests and can also run arbitrary operations (the -t switch)

• The perfect tool to profile Redis (and moar) but...• Like Redis, it is single-threaded so generating load is

challenging• Does not report errors - if results are exceptionally high,

there's probably an error in the statement that you're checking

Howto #2: memtier_benchmark

https://github.com/RedisLabs/memtier_benchmark

• Open source traffic generator for Redis (and Memcached)

• We use it for baseline profiling, stress loading and benchmarking

• Notable differences (compared to redis-benchmark):

• Multi-threaded -> a single client can create considerable loads

• Executes n requests or runs for m seconds

• Generates Read:Write traffic patterns (e.g. 80:20)

• Does uniform and Gaussian distributions

• OTOH – only does GET/SET (or GETRANGE/SETRANGE)

Benchmarking With Multiple Clients – Ouch!

redisbenchmarkframework

• https://github.com/RedisLabs/redisbenchmarkframework

• Python & Fabric

• Executes redis-benchmark/memtier_benchmark/other and custom scripts from multiple servers

• Can spawn multiple threads per server

• Collects RPS results and outputs CSV

Benchmarking vs. Other NoSQLs, an i.e.g.

Aerospike vs. Redis benchmark• http://lynnlangit.com/2015/01/28/lessons-learned-benchmarking-nosql-on-

the-aws-cloud-aerospikedb-and-redis/• http://antirez.com/news/85• https://redislabs.com/blog/the-lessons-missing-from-benchmarking-nosql-

on-the-aws-cloud-aerospikedb-and-redis

To test Redis doing GET/SET is like to test a Ferrari checking how good it is at cleaning the mirror when it rains.

"To test Redis doing GET/SET is like to test a Ferrari checking how good it is at cleaning the mirror when it rains."

Salvatore Sanfilippo

Benchmarking vs. Other NoSQLs, i.e.g. #2

Google Compute Engine's How Many Servers Do You Need To Change a Lightbulb 1 Million Writes/Sec Benchmark Series• DataStax – 300 servers: http://googlecloudplatform.blogspot.com/2014/03/cassandra-

hits-one-million-writes-per-second-on-google-compute-engine.html

• Aerospike – 50 servers: http://googlecloudplatform.blogspot.com/2014/12/aerospike-hits-one-million-writes-Per-Second-with-just-50-Nodes-on-Google-Compute-Engine.html

<- We actually needed two, publication pending

"Comparative" Advertising – There Is A War

• The Open Source Redis Project is (and IMO should always remain) pure

• However,• The users want numbers and comparisons• The analysts want numbers and comparisons• The press wants numbers and comparisons• Redis' competitors are providing numbers and comparisons

• Redis is extremely popular, but it may not remain so without the public's attention

• The Redis community and ecosystem is (and IMO should continue to) provide this collateral in form of benchmarks, success stories, case studies, etc…

All Is Fair In Love And War…

And Benchmarking

An i.e.g.: Yahoo Cloud Serving Benchmark

"The goal of the Yahoo Cloud Serving Benchmark (YCSB) project is to develop a framework and common set of workloads for evaluating the performance of different "key-value" and "cloud" serving stores."

http://labs.yahoo.com/news/yahoo-cloud-serving-benchmark/

YCSB Redis client: https://github.com/brianfrankcooper/YCSB/blob/master/redis/src/main/java/com/yahoo/ycsb/db/RedisClient.java

The YCSB Challenge

• The original YCSB client implementation for Redis• Uses a Hash for storing multiple fields per key

• Uses Redis' commands correctly (i.e. HMGET)

• … but is still too slow

• "Weapon-grade", modified YCSB Redis client:• https://gist.github.com/dragnot/06ef5d01df8cb181555a

• Uses GET/SET

• Objects are serialized client-side with MessagePack

• Objects are updated server-side with Lua (cmsgpack)

YCSB Results