Post on 27-Jan-2015
description
Redis Use PatternsAn Introduction to the SQL
Practitioner@ItamarHaber #DevConTLV 2014
About
A Redis Geek and Chief Developers Advocate at .com
We provide Redis Cloud – an enterprise-class Redis service for developers (infinitely scalable, highly-available with auto failover, top performing hosted Redis off AWS, Google, Azure & IBM SoftLayer)
Get your free t-shirt and
sticker outside!
What’s Redis? (REmote DIctionary Server)• Open-source (BSD), in-memory, persist-able key-value advanced datastore• Key-value means something like
CREATE TABLE redis (k VARCHAR(512MB) NOT NULL,v VARCHAR(512MB),PRIMARY KEY (k)
);
• 6 data types, 160 commands, blazing fast• Created in 2009 by @antirez
(a.k.a Salvatore Sanfilippo)• Source: https://github.com/antirez/redis• Website: http://redis.io
Why Redis? Because It Is Fun!
• Simplicity rich functionality, great flexibility• Performance easily serves 100K’s of ops/sec• Lightweight ~ 2MB footprint• Production proven (name dropping)
Twitter Pintrest StackOverflow teowaki many more…
RedisMakesYou....
THINK!
• about how data is stored• about how data is accessed• about efficiency• about performance• about the network• …
• Redis is a database construction kit
• Beware of Maslow's "Golden" Gavel/Law of Instrument:
"If all you have is a hammer, everything looks like a nail"
Pattern: Caching Calls to the DB
Motivation: quick responses, reduce load on DBMSHow: keep the statement's results using the Redis STRING data type
def get_results(sql):
hash = md5.new(sql).digest()
result = redis.get(hash)
if result is None:
result = db.execute(sql)
redis.set(hash, result)
# or use redis.setex to set a TTL for the key
return result
STRINGs
• Are the most basic data type• Are binary-safe• Is used for storing:• Strings (duh) – APPEND, GETRANGE,
SETRANGE, STRLEN• Integers – INCR, INCRBY, DECR, DECRBY• Floats – INCRBYFLOAT• Bits – SETBIT, GETBIT, BITPOS, BITCOUNT,
BITOP
http://xkcd.com/171/
Pattern: Avoiding Calls to the DB
Motivation: server-side storage and sharing of data that doesn't need a full-fledged RDBMS, e.g. sessions and shopping cartsHow: depending on the case, use STRING or HASH to store data in Redis
def add_to_cart(session, product, quantity):
if quantity > 0:
redis.hset('cart:' + session, product, quantity)
else:
redis.hrem('cart:' + session, product)
def get_cart_contents(session):
return redis.hgetall('cart:' + session)
The HASH Data Type
• Acts as a Redis-within-Redis contains key-value pairs• Have their own commands: HINCRBY, HINCRBYFLOAT, HLEN, HKEYS, HVALS
…• Usually used for aggregation, i.e. keeping related data together for easy
fetching/updating (remember that Redis is not a relational database). Example:
Using separate keys Using hash aggregationuser:1:id 1 user:1 id 1user:1:fname Foo fname Foouser:1:lname Bar lname Baruser:1:email foo@acme.com email foo@acme.com
Denormalization
• Non relational no foreign keys, no referential integrity constraints• Thus, data normalization isn't practical• Be prepared to have duplicated data, e.g.:> HSET user:1 country Mordor
> HSET user:2 country Mordor
…• Tradeoff:
Processing Complexity ↔ Data Volume
Pattern: Lists of Items
Motivation: keeping track of a sequence, e.g. last viewed profilesHow: use Redis' LIST data type
def view_product(uid, product):
redis.lpush('user:' + uid + ':viewed', product)
redis.ltrim('user:' + uid + ':viewed', 0, 9)
…
def get_last_viewed_products(uid):
return redis.lrange('user:' + uid + ':viewed', 0, -1)
Key Points About Key Names
• Key names are "limited" to 512MB (also the values btw)• To conserve RAM & CPU, try avoid using
unnecessarily_longish_names_for_your_redis_keys because they are more expensive to store and compare (unlike an RDBMS's column names, key names are saved for each key-value pair)• On the other hand, don't be too stringent (e.g 'u:<uid>:r')• Although not mandatory, the convention is to use colons
(':') to separate the parts of the key's name• Your schema is your keys' names so keep them in order
Pattern: Queues (apropos the list data type)Motivation: a producer-consumer use case, asynchronous job management, e.g. processing photo uploads
def enqueue(queue, item):
redis.lpush(queue, item)
def dequeue(queue):
return redis.rpop(queue)
# or use brpop for blocking pop
Is Redis ACID? (mostly) Yes!
• Redis is (mostly) single threaded, hence every operation is• Atomic• Consistent• Isolated
• WATCH/MULTI/EXEC allow something like transactions (no rollbacks)• Server-side Lua scripts ("stored procedures")
also behave like transactions• Durability is configurable and is a tradeoff
between efficiency and safety
Pattern: Searching
Motivation: finding keys in the database, for example all the usersHow #1: use a LIST to store key namesHow #2: the *SCAN commands
def do_something_with_all_users():
first = True
cursor = 0
while cursor != 0 or first:
first = False
cursor, data = redis.scan(cursor, 'user:*')
do_something(data)
Pattern: Indexing
Motivation: Redis doesn't have indices, you need to maintain themHow: the SET data type (a collection of unordered unique members)
def update_country_idx(country, uid):
redis.sadd('country:' + country, uid)
def get_users_in_country(country):
return redis.smembers('country:' + country)
Pattern: Relationships
Motivation: Redis doesn't have foreign keys, you need to maintain them
> SADD user:1:friends 3 4 5 // Foo is social and makes friends
> SCARD user:1:friends // How many friends does Foo have?
> SINTER user:1:friends user:2:friends // Common friends
> SDIFF user:1:friends user:2:friends // Exclusive friends
> SUNION user:1:friends user:2:friends // All the friends
ZSETs (Sorted Sets)I HAVE
CDOIT'S LIKE
OCDBUT ALL THE LETTERS ARE
IN ALPHABETICAL ORDERAS THEY SHOULD BE
• Are just like SETs:• Members are unique• ZADD, ZCARD, ZINCRBY, …
• ZSET members have a score that's used for sorting• ZCOUNT, ZRANGE, ZRANGEBYSCORE
• When the scores are identical, members are sorted alphabetically• Lexicographical ranges are also supported:
• ZLEXCOUNT, ZRANGEBYLEX
Pattern: Sorting
Motivation: anything that needs to be sortedHow: ZSETs
> ZADD friends_count 3 1 1 2 999 3
> ZREVRANGE friends_count 0 -1
3
1
2
Set members (uids)
Scores (friends count)
The SORT Command
• A command that sorts LISTs, SETs and SORTED SETs• SORT's syntax is the most complex (comparatively) but SQLers should
feel right at home with it:SORT key [BY pattern] [LIMIT offset count][GET pattern [GET pattern ...]][ASC|DESC] [ALPHA][STORE destination]
• SORT is also expensive in terms of complexity O(N+M*log(M))• BTW, SORT is perhaps the only ad-hoc-like command in Redis
Pattern: Counting Things
Motivation: statistics, real-time analytics, dashboards, throttlingHow #1: use the *INCR commandsHow #2: use a little bit of BIT*def user_log_login(uid):
joined = redis.hget('user:' + uid, 'joined')
d0 = datetime.strptime(joined, '%Y-%m-$d')
d1 = datetime.date.today()
delta = d1 – d0
redis.setbit('user:' + uid + ':logins', delta, 1)
def user_logins_count(uid):
return redis.bitcount(
'user:' + uid + ':logins', 0, -1)
I love to COUNT(*) data!
One datum,two data,
three data!HA HA HA HA!
Pattern: Counting Unique Items
How #1: SADD items and SCARD for the count Problem: more unique items more RAM
How #2: the HyperLogLog data structure> PFADD counter item1 item2 item3 …• HLL is a probabilistic data structure that counts (PFCOUNT) unique items• Sacrifices accuracy: standard error of 0.81%• Gains: constant complexity and memory – 12KB per counter• Bonus: HLLs are merge-able with PFMERGE
Wait, There's More!
• There are 107 additional commands that we didn't cover • Expiration and eviction policies• Publish/Subscribe• Data persistency and durability• Server-side scripting with Lua• Master-Slave(s) replication• High availability with Sentinel• Redis v3 == Cluster (currently in beta)• …
Where Next?
• Try the interactive demo and get a free 25MB Redis database in the cloud at http://redislabs.com• Need help?• RTFM: http://redis.io/documentation• Ask the redis-db mailing list• Visit #redis on Freenode IRC• Email me: itamar@ .com