Application Caching: The Hidden Microservice
-
Upload
scott-mansfield -
Category
Technology
-
view
631 -
download
0
Transcript of Application Caching: The Hidden Microservice
![Page 1: Application Caching: The Hidden Microservice](https://reader035.fdocuments.net/reader035/viewer/2022062523/58a477861a28aba34c8b63d9/html5/thumbnails/1.jpg)
Application Caching:The Hidden Microservice
Scott MansfieldSenior Software EngineerEVCache
![Page 2: Application Caching: The Hidden Microservice](https://reader035.fdocuments.net/reader035/viewer/2022062523/58a477861a28aba34c8b63d9/html5/thumbnails/2.jpg)
![Page 3: Application Caching: The Hidden Microservice](https://reader035.fdocuments.net/reader035/viewer/2022062523/58a477861a28aba34c8b63d9/html5/thumbnails/3.jpg)
![Page 4: Application Caching: The Hidden Microservice](https://reader035.fdocuments.net/reader035/viewer/2022062523/58a477861a28aba34c8b63d9/html5/thumbnails/4.jpg)
![Page 5: Application Caching: The Hidden Microservice](https://reader035.fdocuments.net/reader035/viewer/2022062523/58a477861a28aba34c8b63d9/html5/thumbnails/5.jpg)
![Page 6: Application Caching: The Hidden Microservice](https://reader035.fdocuments.net/reader035/viewer/2022062523/58a477861a28aba34c8b63d9/html5/thumbnails/6.jpg)
![Page 7: Application Caching: The Hidden Microservice](https://reader035.fdocuments.net/reader035/viewer/2022062523/58a477861a28aba34c8b63d9/html5/thumbnails/7.jpg)
![Page 8: Application Caching: The Hidden Microservice](https://reader035.fdocuments.net/reader035/viewer/2022062523/58a477861a28aba34c8b63d9/html5/thumbnails/8.jpg)
![Page 9: Application Caching: The Hidden Microservice](https://reader035.fdocuments.net/reader035/viewer/2022062523/58a477861a28aba34c8b63d9/html5/thumbnails/9.jpg)
![Page 10: Application Caching: The Hidden Microservice](https://reader035.fdocuments.net/reader035/viewer/2022062523/58a477861a28aba34c8b63d9/html5/thumbnails/10.jpg)
![Page 11: Application Caching: The Hidden Microservice](https://reader035.fdocuments.net/reader035/viewer/2022062523/58a477861a28aba34c8b63d9/html5/thumbnails/11.jpg)
![Page 12: Application Caching: The Hidden Microservice](https://reader035.fdocuments.net/reader035/viewer/2022062523/58a477861a28aba34c8b63d9/html5/thumbnails/12.jpg)
![Page 13: Application Caching: The Hidden Microservice](https://reader035.fdocuments.net/reader035/viewer/2022062523/58a477861a28aba34c8b63d9/html5/thumbnails/13.jpg)
![Page 14: Application Caching: The Hidden Microservice](https://reader035.fdocuments.net/reader035/viewer/2022062523/58a477861a28aba34c8b63d9/html5/thumbnails/14.jpg)
![Page 15: Application Caching: The Hidden Microservice](https://reader035.fdocuments.net/reader035/viewer/2022062523/58a477861a28aba34c8b63d9/html5/thumbnails/15.jpg)
![Page 16: Application Caching: The Hidden Microservice](https://reader035.fdocuments.net/reader035/viewer/2022062523/58a477861a28aba34c8b63d9/html5/thumbnails/16.jpg)
![Page 17: Application Caching: The Hidden Microservice](https://reader035.fdocuments.net/reader035/viewer/2022062523/58a477861a28aba34c8b63d9/html5/thumbnails/17.jpg)
![Page 18: Application Caching: The Hidden Microservice](https://reader035.fdocuments.net/reader035/viewer/2022062523/58a477861a28aba34c8b63d9/html5/thumbnails/18.jpg)
90 seconds
![Page 19: Application Caching: The Hidden Microservice](https://reader035.fdocuments.net/reader035/viewer/2022062523/58a477861a28aba34c8b63d9/html5/thumbnails/19.jpg)
What do caches touch?
Signing up*Logging inChoosing a profilePicking liked videosPersonalization*Loading home page*Scrolling home page*A/B testsVideo image selection
Searching*Viewing title detailsPlaying a title*Subtitle / language prefsRating a titleMy ListVideo history*UI stringsVideo production*
* multiple caches involved
![Page 20: Application Caching: The Hidden Microservice](https://reader035.fdocuments.net/reader035/viewer/2022062523/58a477861a28aba34c8b63d9/html5/thumbnails/20.jpg)
Home PageRequest
![Page 21: Application Caching: The Hidden Microservice](https://reader035.fdocuments.net/reader035/viewer/2022062523/58a477861a28aba34c8b63d9/html5/thumbnails/21.jpg)
![Page 22: Application Caching: The Hidden Microservice](https://reader035.fdocuments.net/reader035/viewer/2022062523/58a477861a28aba34c8b63d9/html5/thumbnails/22.jpg)
KV store optimized for AWS and tuned for Netflix use cases.
Ephemeral Volatile Cache
![Page 23: Application Caching: The Hidden Microservice](https://reader035.fdocuments.net/reader035/viewer/2022062523/58a477861a28aba34c8b63d9/html5/thumbnails/23.jpg)
What is EVCache?
Distributed, sharded, replicated key-value storeBased on MemcachedTunable in-region and global replicationResilient to failureTopology awareLinearly scalableSeamless deployments
![Page 24: Application Caching: The Hidden Microservice](https://reader035.fdocuments.net/reader035/viewer/2022062523/58a477861a28aba34c8b63d9/html5/thumbnails/24.jpg)
Why Optimize for AWS
Instances disappearZones failRegions become unstableNetwork is lossyCustomer requests bounce between regions
Failures happen, and we test all the time
![Page 25: Application Caching: The Hidden Microservice](https://reader035.fdocuments.net/reader035/viewer/2022062523/58a477861a28aba34c8b63d9/html5/thumbnails/25.jpg)
EVCache Use @ Netflix Hundreds of terabytes of dataTrillions of ops / dayTens of billions of items storedTens of millions of ops / secMillions of replications / secThousands of serversHundreds of instances per clusterHundreds of microservice clientsTens of distinct clusters3 regions4 engineers
![Page 26: Application Caching: The Hidden Microservice](https://reader035.fdocuments.net/reader035/viewer/2022062523/58a477861a28aba34c8b63d9/html5/thumbnails/26.jpg)
Architecture
Server
Memcached
Prana (Sidecar)
Application
Client Library
Client
Eureka(Service Discovery)
![Page 27: Application Caching: The Hidden Microservice](https://reader035.fdocuments.net/reader035/viewer/2022062523/58a477861a28aba34c8b63d9/html5/thumbnails/27.jpg)
Architecture
![Page 28: Application Caching: The Hidden Microservice](https://reader035.fdocuments.net/reader035/viewer/2022062523/58a477861a28aba34c8b63d9/html5/thumbnails/28.jpg)
Reading
Primary Secondary
![Page 29: Application Caching: The Hidden Microservice](https://reader035.fdocuments.net/reader035/viewer/2022062523/58a477861a28aba34c8b63d9/html5/thumbnails/29.jpg)
Writing
![Page 30: Application Caching: The Hidden Microservice](https://reader035.fdocuments.net/reader035/viewer/2022062523/58a477861a28aba34c8b63d9/html5/thumbnails/30.jpg)
Use Case: Lookaside Cache
Application
Client Library
Client Ribbon Client
S S S S
C C C CData Flow
![Page 31: Application Caching: The Hidden Microservice](https://reader035.fdocuments.net/reader035/viewer/2022062523/58a477861a28aba34c8b63d9/html5/thumbnails/31.jpg)
Use Case: Transient Data Store
Application
Client Library
Client
Application
Client Library
Client
Application
Client Library
Client
Time
![Page 32: Application Caching: The Hidden Microservice](https://reader035.fdocuments.net/reader035/viewer/2022062523/58a477861a28aba34c8b63d9/html5/thumbnails/32.jpg)
Use Case: Primary Store
Offline / Nearline Precomputes for
Recommendations
Online Services
Offline Services
Online Application
Client Library
Client
Data Flow
![Page 33: Application Caching: The Hidden Microservice](https://reader035.fdocuments.net/reader035/viewer/2022062523/58a477861a28aba34c8b63d9/html5/thumbnails/33.jpg)
Use Case: High Volume && High Availability
Compute & Publish on schedule
Data Flow
Application
Client Library
Client Ribbon ClientIn-memory Cache
Optional
S S S S
C C C C
![Page 34: Application Caching: The Hidden Microservice](https://reader035.fdocuments.net/reader035/viewer/2022062523/58a477861a28aba34c8b63d9/html5/thumbnails/34.jpg)
Pipeline of Personalization
Compute A
Compute B Compute C
Compute D
Online Services
Offline Services
Compute E
Data Flow
Online 1 Online 2
![Page 35: Application Caching: The Hidden Microservice](https://reader035.fdocuments.net/reader035/viewer/2022062523/58a477861a28aba34c8b63d9/html5/thumbnails/35.jpg)
Polyglot Clients
APP
Java Client
APP
Prana
Local Proxy
Memcached
APP
Memcached Proxy
HTTP
APP
HTTP Proxy
![Page 36: Application Caching: The Hidden Microservice](https://reader035.fdocuments.net/reader035/viewer/2022062523/58a477861a28aba34c8b63d9/html5/thumbnails/36.jpg)
Additional Features
Global data replicationCache warmingSecondary indexingConsistency checking
All powered by metadata flowing through Kafka
![Page 37: Application Caching: The Hidden Microservice](https://reader035.fdocuments.net/reader035/viewer/2022062523/58a477861a28aba34c8b63d9/html5/thumbnails/37.jpg)
Region BRegion A
APP APP
Repl Proxy
Repl Relay
1 mutate
2 send metadata
3 poll msg
5 https s
end msg
6 mutate4 get data
for set
Kafka Repl Relay Kafka
Repl Proxy
Cross-Region Replication
7 read
![Page 38: Application Caching: The Hidden Microservice](https://reader035.fdocuments.net/reader035/viewer/2022062523/58a477861a28aba34c8b63d9/html5/thumbnails/38.jpg)
Cache Warming
Cache Warmer
Kafka
Application
Client Library
Client
![Page 39: Application Caching: The Hidden Microservice](https://reader035.fdocuments.net/reader035/viewer/2022062523/58a477861a28aba34c8b63d9/html5/thumbnails/39.jpg)
Code
![Page 40: Application Caching: The Hidden Microservice](https://reader035.fdocuments.net/reader035/viewer/2022062523/58a477861a28aba34c8b63d9/html5/thumbnails/40.jpg)
Moneta
Next-generationEVCache server
![Page 41: Application Caching: The Hidden Microservice](https://reader035.fdocuments.net/reader035/viewer/2022062523/58a477861a28aba34c8b63d9/html5/thumbnails/41.jpg)
Moneta
Moneta: The Goddess of MemoryJuno Moneta: The Protectress of Funds for Juno
● Evolution of the EVCache server● EVCache on SSD● Cost optimization● Ongoing lower EVCache cost per stream● Takes advantage of global request patterns
![Page 42: Application Caching: The Hidden Microservice](https://reader035.fdocuments.net/reader035/viewer/2022062523/58a477861a28aba34c8b63d9/html5/thumbnails/42.jpg)
Old Server
● Stock Memcached and Prana (Netflix sidecar)● All data stored in RAM in Memcached● Expensive with global expansion / N+1 architecture
external
![Page 43: Application Caching: The Hidden Microservice](https://reader035.fdocuments.net/reader035/viewer/2022062523/58a477861a28aba34c8b63d9/html5/thumbnails/43.jpg)
Optimization
● Global data means many copies● Access patterns are heavily region-oriented● In one region:
○ Hot data is used often○ Cold data is almost never touched
● Keep hot data in RAM, cold data on SSD● Size RAM for working set, SSD for overall dataset
![Page 44: Application Caching: The Hidden Microservice](https://reader035.fdocuments.net/reader035/viewer/2022062523/58a477861a28aba34c8b63d9/html5/thumbnails/44.jpg)
New Server
● Adds Rend and Mnemonic● Still looks like Memcached● Unlocks cost-efficient storage & server-side intelligence
external internal
![Page 45: Application Caching: The Hidden Microservice](https://reader035.fdocuments.net/reader035/viewer/2022062523/58a477861a28aba34c8b63d9/html5/thumbnails/45.jpg)
Rend
![Page 46: Application Caching: The Hidden Microservice](https://reader035.fdocuments.net/reader035/viewer/2022062523/58a477861a28aba34c8b63d9/html5/thumbnails/46.jpg)
Rend
● High-performance Memcached proxy & server● Written in Go
○ Powerful concurrency primitives○ Productive and fast
● Manages the L1/L2 relationship● Tens of thousands of connections
![Page 47: Application Caching: The Hidden Microservice](https://reader035.fdocuments.net/reader035/viewer/2022062523/58a477861a28aba34c8b63d9/html5/thumbnails/47.jpg)
Rend
● Modular to allow future changes / expansion of scope○ Set of libraries and a default
● Manages connections, request orchestration, and communication
● Low-overhead metrics library● Multiple orchestrators● Parallel locking for data integrity
![Page 48: Application Caching: The Hidden Microservice](https://reader035.fdocuments.net/reader035/viewer/2022062523/58a477861a28aba34c8b63d9/html5/thumbnails/48.jpg)
Mnemonic
Open source soon™
![Page 49: Application Caching: The Hidden Microservice](https://reader035.fdocuments.net/reader035/viewer/2022062523/58a477861a28aba34c8b63d9/html5/thumbnails/49.jpg)
Mnemonic
● Manages data storage on SSD● Reuses Rend server libraries● Maps Memcached ops to RocksDB ops
![Page 50: Application Caching: The Hidden Microservice](https://reader035.fdocuments.net/reader035/viewer/2022062523/58a477861a28aba34c8b63d9/html5/thumbnails/50.jpg)
Why RocksDB?
● Fast at medium to high write load○ Goal was 99% read latency below 20ms
● Log-Structured Merge minimizes random writes to SSD○ Writes are buffered in memory
● Immutable Static Sorted Tables
. . .
![Page 51: Application Caching: The Hidden Microservice](https://reader035.fdocuments.net/reader035/viewer/2022062523/58a477861a28aba34c8b63d9/html5/thumbnails/51.jpg)
How we use RocksDB
● FIFO "Compaction"○ More suitable for our precompute use cases
● Bloom filters and indices pinned in memory○ Trade L1 space for faster L2
● Records sharded across many RocksDBs per instance○ Reduces number of files checked, decreasing latency
R...
Key: ABCKey: XYZ
R R R
![Page 52: Application Caching: The Hidden Microservice](https://reader035.fdocuments.net/reader035/viewer/2022062523/58a477861a28aba34c8b63d9/html5/thumbnails/52.jpg)
FIFO Limitation
● FIFO compaction not suitable for all use cases○ Very frequently updated records may push out valid records
● Future: custom compaction or level compaction
SST
time
SSTSST
![Page 53: Application Caching: The Hidden Microservice](https://reader035.fdocuments.net/reader035/viewer/2022062523/58a477861a28aba34c8b63d9/html5/thumbnails/53.jpg)
Moneta in Production
● Serving all of our personalization data● Rend runs with two ports:
○ One for standard users (read heavy or active data management)○ Another for async and batch users: Replication and Precompute
● Maintains working set in RAM● Optimized for precomputes
○ Smartly replaces data in L1
external internal
![Page 54: Application Caching: The Hidden Microservice](https://reader035.fdocuments.net/reader035/viewer/2022062523/58a477861a28aba34c8b63d9/html5/thumbnails/54.jpg)
Get: 229 μs (139 μs) avgPercentiles:● 50th: 140 μs (113 μs)● 75th: 199 μs (139 μs)● 90th: 318 μs (195 μs)● 95th: 486 μs (231 μs)● 99th: 1.67 ms (508 μs)● 99.9th: 9.68 ms (1.97 ms)
Moneta Performance in Production
Set: 367 μs (200 μs) avgPercentiles:● 50th: 227 μs (172 μs)● 75th: 301 μs (202 μs)● 90th: 411 μs (243 μs)● 95th: 579 μs (295 μs)● 99th: 3.25 ms (502 μs)● 99.9th: 18.5 ms (4.73 ms)
Latencies: peak (trough)
![Page 55: Application Caching: The Hidden Microservice](https://reader035.fdocuments.net/reader035/viewer/2022062523/58a477861a28aba34c8b63d9/html5/thumbnails/55.jpg)
70%Reduction in cost*
70%
![Page 56: Application Caching: The Hidden Microservice](https://reader035.fdocuments.net/reader035/viewer/2022062523/58a477861a28aba34c8b63d9/html5/thumbnails/56.jpg)
Open Source
![Page 58: Application Caching: The Hidden Microservice](https://reader035.fdocuments.net/reader035/viewer/2022062523/58a477861a28aba34c8b63d9/html5/thumbnails/58.jpg)
![Page 59: Application Caching: The Hidden Microservice](https://reader035.fdocuments.net/reader035/viewer/2022062523/58a477861a28aba34c8b63d9/html5/thumbnails/59.jpg)
Failure Resilience in Client
● Operation Fast Failure● Tunable Retries● Operation Queues● Tunable Latch for Mutations● Async Replication through Kafka