Redis Use PatternsAn Introduction to the SQL

Practitioner@ItamarHaber #DevConTLV 2014

About

A Redis Geek and Chief Developers Advocate at .com

We provide Redis Cloud – an enterprise-class Redis service for developers (infinitely scalable, highly-available with auto failover, top performing hosted Redis off AWS, Google, Azure & IBM SoftLayer)

Get your free t-shirt and

sticker outside!

What’s Redis? (REmote DIctionary Server)• Open-source (BSD), in-memory, persist-able key-value advanced datastore• Key-value means something like

CREATE TABLE redis (k VARCHAR(512MB) NOT NULL,v VARCHAR(512MB),PRIMARY KEY (k)

);

• 6 data types, 160 commands, blazing fast• Created in 2009 by @antirez

(a.k.a Salvatore Sanfilippo)• Source: https://github.com/antirez/redis• Website: http://redis.io

Why Redis? Because It Is Fun!

• Simplicity rich functionality, great flexibility• Performance easily serves 100K’s of ops/sec• Lightweight ~ 2MB footprint• Production proven (name dropping)

Twitter Pintrest StackOverflow teowaki many more…

RedisMakesYou....

THINK!

• about how data is stored• about how data is accessed• about efficiency• about performance• about the network• …

• Redis is a database construction kit

• Beware of Maslow's "Golden" Gavel/Law of Instrument:

"If all you have is a hammer, everything looks like a nail"

Pattern: Caching Calls to the DB

Motivation: quick responses, reduce load on DBMSHow: keep the statement's results using the Redis STRING data type

def get_results(sql):

hash = md5.new(sql).digest()

result = redis.get(hash)

if result is None:

result = db.execute(sql)

redis.set(hash, result)

# or use redis.setex to set a TTL for the key

return result

STRINGs

• Are the most basic data type• Are binary-safe• Is used for storing:• Strings (duh) – APPEND, GETRANGE,

SETRANGE, STRLEN• Integers – INCR, INCRBY, DECR, DECRBY• Floats – INCRBYFLOAT• Bits – SETBIT, GETBIT, BITPOS, BITCOUNT,

BITOP

http://xkcd.com/171/

Pattern: Avoiding Calls to the DB

Motivation: server-side storage and sharing of data that doesn't need a full-fledged RDBMS, e.g. sessions and shopping cartsHow: depending on the case, use STRING or HASH to store data in Redis

def add_to_cart(session, product, quantity):

if quantity > 0:

redis.hset('cart:' + session, product, quantity)

else:

redis.hrem('cart:' + session, product)

def get_cart_contents(session):

return redis.hgetall('cart:' + session)

The HASH Data Type

• Acts as a Redis-within-Redis contains key-value pairs• Have their own commands: HINCRBY, HINCRBYFLOAT, HLEN, HKEYS, HVALS

…• Usually used for aggregation, i.e. keeping related data together for easy

fetching/updating (remember that Redis is not a relational database). Example:

Using separate keys Using hash aggregationuser:1:id 1 user:1 id 1user:1:fname Foo fname Foouser:1:lname Bar lname Baruser:1:email foo@acme.com email foo@acme.com

Denormalization

• Non relational no foreign keys, no referential integrity constraints• Thus, data normalization isn't practical• Be prepared to have duplicated data, e.g.:> HSET user:1 country Mordor

> HSET user:2 country Mordor

…• Tradeoff:

Processing Complexity ↔ Data Volume

Pattern: Lists of Items

Motivation: keeping track of a sequence, e.g. last viewed profilesHow: use Redis' LIST data type

def view_product(uid, product):

redis.lpush('user:' + uid + ':viewed', product)

redis.ltrim('user:' + uid + ':viewed', 0, 9)

…

def get_last_viewed_products(uid):

return redis.lrange('user:' + uid + ':viewed', 0, -1)

Key Points About Key Names

• Key names are "limited" to 512MB (also the values btw)• To conserve RAM & CPU, try avoid using

unnecessarily_longish_names_for_your_redis_keys because they are more expensive to store and compare (unlike an RDBMS's column names, key names are saved for each key-value pair)• On the other hand, don't be too stringent (e.g 'u:<uid>:r')• Although not mandatory, the convention is to use colons

(':') to separate the parts of the key's name• Your schema is your keys' names so keep them in order

Pattern: Queues (apropos the list data type)Motivation: a producer-consumer use case, asynchronous job management, e.g. processing photo uploads

def enqueue(queue, item):

redis.lpush(queue, item)

def dequeue(queue):

return redis.rpop(queue)

# or use brpop for blocking pop

Is Redis ACID? (mostly) Yes!

• Redis is (mostly) single threaded, hence every operation is• Atomic• Consistent• Isolated

• WATCH/MULTI/EXEC allow something like transactions (no rollbacks)• Server-side Lua scripts ("stored procedures")

also behave like transactions• Durability is configurable and is a tradeoff

between efficiency and safety

Pattern: Searching

Motivation: finding keys in the database, for example all the usersHow #1: use a LIST to store key namesHow #2: the *SCAN commands

def do_something_with_all_users():

first = True

cursor = 0

while cursor != 0 or first:

first = False

cursor, data = redis.scan(cursor, 'user:*')

do_something(data)

Pattern: Indexing

Motivation: Redis doesn't have indices, you need to maintain themHow: the SET data type (a collection of unordered unique members)

def update_country_idx(country, uid):

redis.sadd('country:' + country, uid)

def get_users_in_country(country):

return redis.smembers('country:' + country)

Pattern: Relationships

Motivation: Redis doesn't have foreign keys, you need to maintain them

> SADD user:1:friends 3 4 5 // Foo is social and makes friends

> SCARD user:1:friends // How many friends does Foo have?

> SINTER user:1:friends user:2:friends // Common friends

> SDIFF user:1:friends user:2:friends // Exclusive friends

> SUNION user:1:friends user:2:friends // All the friends

ZSETs (Sorted Sets)I HAVE

CDOIT'S LIKE

OCDBUT ALL THE LETTERS ARE

IN ALPHABETICAL ORDERAS THEY SHOULD BE

• Are just like SETs:• Members are unique• ZADD, ZCARD, ZINCRBY, …

• ZSET members have a score that's used for sorting• ZCOUNT, ZRANGE, ZRANGEBYSCORE

• When the scores are identical, members are sorted alphabetically• Lexicographical ranges are also supported:

• ZLEXCOUNT, ZRANGEBYLEX

Pattern: Sorting

Motivation: anything that needs to be sortedHow: ZSETs

> ZADD friends_count 3 1 1 2 999 3

> ZREVRANGE friends_count 0 -1

3

1

2

Set members (uids)

Scores (friends count)

The SORT Command

• A command that sorts LISTs, SETs and SORTED SETs• SORT's syntax is the most complex (comparatively) but SQLers should

feel right at home with it:SORT key [BY pattern] [LIMIT offset count][GET pattern [GET pattern ...]][ASC|DESC] [ALPHA][STORE destination]

• SORT is also expensive in terms of complexity O(N+M*log(M))• BTW, SORT is perhaps the only ad-hoc-like command in Redis

Pattern: Counting Things

Motivation: statistics, real-time analytics, dashboards, throttlingHow #1: use the *INCR commandsHow #2: use a little bit of BIT*def user_log_login(uid):

joined = redis.hget('user:' + uid, 'joined')

d0 = datetime.strptime(joined, '%Y-%m-$d')

d1 = datetime.date.today()

delta = d1 – d0

redis.setbit('user:' + uid + ':logins', delta, 1)

def user_logins_count(uid):

return redis.bitcount(

'user:' + uid + ':logins', 0, -1)

I love to COUNT(*) data!

One datum,two data,

three data!HA HA HA HA!

Pattern: Counting Unique Items

How #1: SADD items and SCARD for the count Problem: more unique items more RAM

How #2: the HyperLogLog data structure> PFADD counter item1 item2 item3 …• HLL is a probabilistic data structure that counts (PFCOUNT) unique items• Sacrifices accuracy: standard error of 0.81%• Gains: constant complexity and memory – 12KB per counter• Bonus: HLLs are merge-able with PFMERGE

Wait, There's More!

• There are 107 additional commands that we didn't cover • Expiration and eviction policies• Publish/Subscribe• Data persistency and durability• Server-side scripting with Lua• Master-Slave(s) replication• High availability with Sentinel• Redis v3 == Cluster (currently in beta)• …

Where Next?

• Try the interactive demo and get a free 25MB Redis database in the cloud at http://redislabs.com• Need help?• RTFM: http://redis.io/documentation• Ask the redis-db mailing list• Visit #redis on Freenode IRC• Email me: itamar@ .com