Sharding Redis at Flite
-
Upload
eugene-feingold -
Category
Technology
-
view
1.383 -
download
8
description
Transcript of Sharding Redis at Flite
![Page 1: Sharding Redis at Flite](https://reader031.fdocuments.net/reader031/viewer/2022020115/546f93c4b4af9f350b8b45d3/html5/thumbnails/1.jpg)
SHARDING REDIS at FLITE
Eugene Feingold & Chakri Kodali
![Page 2: Sharding Redis at Flite](https://reader031.fdocuments.net/reader031/viewer/2022020115/546f93c4b4af9f350b8b45d3/html5/thumbnails/2.jpg)
WHAT IS FLITE?
• Display advertising platform • Create and publish rich,
interactive ads • Serve ad impressions • Collect and analyze metrics,
batch AND realtime
![Page 3: Sharding Redis at Flite](https://reader031.fdocuments.net/reader031/viewer/2022020115/546f93c4b4af9f350b8b45d3/html5/thumbnails/3.jpg)
WHAT IS FLITE?
• Display advertising platform • Create and publish rich,
interactive ads • Serve ad impressions • Collect and analyze metrics,
batch AND realtime with
![Page 4: Sharding Redis at Flite](https://reader031.fdocuments.net/reader031/viewer/2022020115/546f93c4b4af9f350b8b45d3/html5/thumbnails/4.jpg)
REALTIME YOU SAY?
• Realtime monitoring of ad performance • Debugging of ads, with instant feedback on triggered events
![Page 5: Sharding Redis at Flite](https://reader031.fdocuments.net/reader031/viewer/2022020115/546f93c4b4af9f350b8b45d3/html5/thumbnails/5.jpg)
METRICS ARCHITECTURE
JVM
JVM
JVM
ad
ad
ad
ad
ad
ad
ad
node.js
node.js
pipelined write of ~30 items
![Page 6: Sharding Redis at Flite](https://reader031.fdocuments.net/reader031/viewer/2022020115/546f93c4b4af9f350b8b45d3/html5/thumbnails/6.jpg)
WRITING TO REDIS
Persistence: Up to 4 hours for most data
Amount: ~30 writes per event, pipelined
Granularity: By second, minute, hour
Write Types: • HSET - JSON blobs (Event body data) • LPUSH - Lists of events (Events by session) • HINCRBY - Simple counters (Events by ad) • ZINCRBY - Sorted set counters (To retrieve “Top 100”) • PUBSUB - Stream event data (Debugger)
![Page 7: Sharding Redis at Flite](https://reader031.fdocuments.net/reader031/viewer/2022020115/546f93c4b4af9f350b8b45d3/html5/thumbnails/7.jpg)
READING FROM REDIS
Redis transactions are used extensively like multi, exec Read types: • HGETALL - get all data for event • HGET - get event counts by ad • LRANGE - list of all events by session • ZREVRANGE - top 100 ads with highest number of events • SUBSCRIBE
![Page 8: Sharding Redis at Flite](https://reader031.fdocuments.net/reader031/viewer/2022020115/546f93c4b4af9f350b8b45d3/html5/thumbnails/8.jpg)
AND ALL OF THIS AT SCALE
• Daily traffic peaks: 100k - 200k events per minute
• Peaks are really plateaus that last for hours
• Read load is negligible by comparison, but reads must be fast
• In fact, everything must be fast: <1 sec latency for debugger to work
![Page 9: Sharding Redis at Flite](https://reader031.fdocuments.net/reader031/viewer/2022020115/546f93c4b4af9f350b8b45d3/html5/thumbnails/9.jpg)
WHAT’S WRONG WITH THIS PICTURE?
JVM
JVM
JVM
ad
ad
ad
ad
ad
ad
ad
node.js
node.js
![Page 10: Sharding Redis at Flite](https://reader031.fdocuments.net/reader031/viewer/2022020115/546f93c4b4af9f350b8b45d3/html5/thumbnails/10.jpg)
WHAT’S WRONG WITH THIS PICTURE?
Bottleneck!
JVM
JVM
JVM
ad
ad
ad
ad
ad
ad
ad
node.js
node.js
![Page 11: Sharding Redis at Flite](https://reader031.fdocuments.net/reader031/viewer/2022020115/546f93c4b4af9f350b8b45d3/html5/thumbnails/11.jpg)
SCALABILITY FAIL!
What did it look like?
2-3x of our usual load
![Page 12: Sharding Redis at Flite](https://reader031.fdocuments.net/reader031/viewer/2022020115/546f93c4b4af9f350b8b45d3/html5/thumbnails/12.jpg)
SCALABILITY FAIL!
Note how only 1 core is being used 14,000 open connections!
What did it look like?
![Page 13: Sharding Redis at Flite](https://reader031.fdocuments.net/reader031/viewer/2022020115/546f93c4b4af9f350b8b45d3/html5/thumbnails/13.jpg)
A QUICK SOLUTION?
• MOAR Megahurtz!!: m2.2xl is already about as fast as Amazon gets.
• Redis-As-A-Service: Expensive, not fast enough for even our usual load.
• twemproxy: Twitter’s sharding solution.
Doesn’t support all commands: PING, MULTI, INFO, MGET, etc…
![Page 14: Sharding Redis at Flite](https://reader031.fdocuments.net/reader031/viewer/2022020115/546f93c4b4af9f350b8b45d3/html5/thumbnails/14.jpg)
WHAT ABOUT JEDIS’S NATIVE SHARDING?
• No pipelining
• No pubsub
• Complicated consistent hashing mechanism makes reading in other
environments more difficult
![Page 15: Sharding Redis at Flite](https://reader031.fdocuments.net/reader031/viewer/2022020115/546f93c4b4af9f350b8b45d3/html5/thumbnails/15.jpg)
LET’S ROLL OUR OWN!
Goals: Speed, speed, speed
Not Goals: Fault tolerance, redundancy, resiliency
![Page 16: Sharding Redis at Flite](https://reader031.fdocuments.net/reader031/viewer/2022020115/546f93c4b4af9f350b8b45d3/html5/thumbnails/16.jpg)
HOW HARD CAN IT BE?
Sharding method: Java hashCode of key for every item written
JVM ad event items
node.js
Write to many Read from one
![Page 17: Sharding Redis at Flite](https://reader031.fdocuments.net/reader031/viewer/2022020115/546f93c4b4af9f350b8b45d3/html5/thumbnails/17.jpg)
WHAT HAPPENED?
Before Sharding After Sharding
Items per Event 30 30
Items written per Event per Redis 30 10
Redis Connections per Event 1 3
Connections per second per Redis box n n
Reality: When n gets to around 500, Redis maxes out CPU and starts rejecting connections.
Theory: Since Redis claims to be able to handle 70k connections per second, the amount of data being sent per connection is the problem.
![Page 18: Sharding Redis at Flite](https://reader031.fdocuments.net/reader031/viewer/2022020115/546f93c4b4af9f350b8b45d3/html5/thumbnails/18.jpg)
SO HOW HARD CAN IT BE?
HARD.
![Page 19: Sharding Redis at Flite](https://reader031.fdocuments.net/reader031/viewer/2022020115/546f93c4b4af9f350b8b45d3/html5/thumbnails/19.jpg)
TAKE TWO
Sharding method: Java hashCode of EVENT key for every item written. A single key now lives on multiple Redis boxes
JVM ad event items
node.js
Write to one Read from many
![Page 20: Sharding Redis at Flite](https://reader031.fdocuments.net/reader031/viewer/2022020115/546f93c4b4af9f350b8b45d3/html5/thumbnails/20.jpg)
BETTER!
Before Sharding After Take 1 After Take 2
Items per Event 30 30 30
Items written per Event per Redis 30 10 30
Redis Connections per Event 1 3 1
Connections per second per Redis box n n n/3
More load can be easily accommodated by adding boxes
![Page 21: Sharding Redis at Flite](https://reader031.fdocuments.net/reader031/viewer/2022020115/546f93c4b4af9f350b8b45d3/html5/thumbnails/21.jpg)
CODING CHALLENGES Java • Managing multiple connection pools • Managing multiple pipelines • Automatic health checks
node.js • Finding hashing function that works in different environments • Managing multiple pipelines • Fanout requests and merging response once pipeline is
executed
![Page 22: Sharding Redis at Flite](https://reader031.fdocuments.net/reader031/viewer/2022020115/546f93c4b4af9f350b8b45d3/html5/thumbnails/22.jpg)
SINGLE-REDIS JEDIS WORKFLOW
On application startup: 1. Initialize jedisPool with connection info
Every time: 1. Jedis jedisClient = jedisPool.getClient(); 2. Pipeline pipeline = jedisClient.pipelined(); 3. State your business 4. pipeline.sync(); 5. jedisPool.returnResource(jedisClient);
![Page 23: Sharding Redis at Flite](https://reader031.fdocuments.net/reader031/viewer/2022020115/546f93c4b4af9f350b8b45d3/html5/thumbnails/23.jpg)
SHARDED JEDIS WORKFLOW
On application startup: 1. Initialize n jedisPools with connection info
Every time: 1. Jedis jedisClient = jedisPool.getClient(); 2. Pipeline pipeline = jedisClient.pipelined(); 3. State your business 4. pipeline.sync(); 5. jedisPool.returnResource(jedisClient);
![Page 24: Sharding Redis at Flite](https://reader031.fdocuments.net/reader031/viewer/2022020115/546f93c4b4af9f350b8b45d3/html5/thumbnails/24.jpg)
SHARDED JEDIS WORKFLOW
On application startup: 1. Initialize n jedisPools with connection info
Every time: 1. Jedis jedisClient = jedisPool.getClient(); Which pool? 2. Pipeline pipeline = jedisClient.pipelined(); Which client? 3. State your business To whom? 4. pipeline.sync(); Which pipeline? 5. jedisPool.returnResource(jedisClient); Return what where?
![Page 25: Sharding Redis at Flite](https://reader031.fdocuments.net/reader031/viewer/2022020115/546f93c4b4af9f350b8b45d3/html5/thumbnails/25.jpg)
SHARDED JEDIS WORKFLOW
RedisNodeManager Spring-created singleton
JedisPoolManager JedisPoolManager JedisPoolManager RedisPoolManager
![Page 26: Sharding Redis at Flite](https://reader031.fdocuments.net/reader031/viewer/2022020115/546f93c4b4af9f350b8b45d3/html5/thumbnails/26.jpg)
SHARDED JEDIS WORKFLOW
RedisNodeManager Spring-created singleton
JedisPoolManager JedisPoolManager JedisPoolManager RedisPoolManager
RedisPipelineManager JedisPoolManager JedisPoolManager JedisPoolManager RedisPoolManager
getPipelineManager()
![Page 27: Sharding Redis at Flite](https://reader031.fdocuments.net/reader031/viewer/2022020115/546f93c4b4af9f350b8b45d3/html5/thumbnails/27.jpg)
SHARDED JEDIS WORKFLOW
RedisNodeManager Spring-created singleton
JedisPoolManager JedisPoolManager JedisPoolManager RedisPoolManager
RedisPipelineManager JedisPoolManager JedisPoolManager JedisPoolManager RedisPoolManager
Pipeline
getPipelineManager()
getPipeline(shardKey)
![Page 28: Sharding Redis at Flite](https://reader031.fdocuments.net/reader031/viewer/2022020115/546f93c4b4af9f350b8b45d3/html5/thumbnails/28.jpg)
LET’S LOOK AT SOME CODE!
![Page 29: Sharding Redis at Flite](https://reader031.fdocuments.net/reader031/viewer/2022020115/546f93c4b4af9f350b8b45d3/html5/thumbnails/29.jpg)
OPERATIONAL DETAILS
• Redis is single threaded o You can run multiple Redises on one server o Bind each Redis instance to a specific core
![Page 30: Sharding Redis at Flite](https://reader031.fdocuments.net/reader031/viewer/2022020115/546f93c4b4af9f350b8b45d3/html5/thumbnails/30.jpg)
QUESTIONS?