Cassandra @ eBay Jay Patel Architect, Platform Systems @pateljay3001
eBay Marketplaces
Thousands of servers Petabytes of data
Billions of SQLs/day 24x7x365 99.98+% Availability
turning over a TB every second Multiple Datacenters
Near-Real-time Always online
400+ million items for sale
$75 billion+ per year in goods are sold on eBay
Big Data
112 million active users
Billions of page views/day
3
eBay Site Data Infrastructure
Don’t force! One size does not fit all.
It’s a mixture of multiple SQL & NoSQL databases. We use the right database for the right problem.
eBay Site Data Infrastructure A heterogeneous mixture
Thousands of nodes > 2K sharded logical host > 16K tables > 27K indexes > 140 billion SQLs/day > 5 PB provisioned
Hundreds of nodes Persistent & in-memory > 40 billion SQLs/day
10+ clusters, 100+ nodes > 250 TB provisioned (local HDD + shared SSD) > 9 billion writes/day > 5 billion reads/day
Hundreds of nodes > 50 TB > 2 billion ops/day
Thousands of nodes The world largest cluster with 2K+ nodes
Dozens of nodes
How do we scale RDBMS?
Shard
– Patterns: Modulus, lookup-based, range, etc.
– Application sees only logical shard/database
Replicate
– Disaster recovery, read availability & read scalability
Big NOs
– No transactions
– No joins
– No referential integrity constraints
5
Why Cassandra?
Multi-datacenter (active-active)
Always Available - No SPOF
Easy to scale up & down
6
Write performance
Distributed counters
Hadoop support
Not replacing RDBMS, but complementing!
Some use cases don’t fit well in RDBMS - sparse data, big data, flexible schema, real-time analytics, …
Many use cases don’t need top-tier set-ups.
Cassandra Growth
Au
g, 2
01
1
Au
g, 2
01
2
May
, 20
13
1
2
3
4
5
6
7
Billions (per day)
writes async. reads
sync. site reads
Terabytes
50
100
200
250
300
350 storage capacity
Doesn’t predict business
7
eBay Use Cases on Cassandra Time-series data, real-time insights & immediate actions
• Fraud detection & prevention
• Quality Click Pricing for affiliates
• Order & shipment tracking and insights
• Mobile notification logging & tracking
• Cloud CMS change history storage
• RedLaser server logs and analytics
Server metrics collection for monitoring & alerting
Taste graph based next-gen recommendation system
Personalization Data Service
Social Signals on eBay Product & Item pages
Milo’s store-item availability inventory (evaluation phase)
8
Real-time insights & actions for
9
Fraud Prevention Reporting
Quality Click Pricing More…
10
System Overview
Business Event Stream
Checkout Shipping Refund & Recoup …
Order placed (bin/bid)
Paid Shipped Refunded
Raw
dat
a
Simple in-memory aggregations +/ Complex Event Processing +/
Cassandra’s distributed counters
Label printed per day per user User segmentation for affiliate pricing Orders per hour, …
Multiple Cassandra clusters
Payment
Act
in r
eal-
tim
e
Fraud Prevention
Affiliate Pricing Engine (eBay Partner Network)
Order tracking
Real-time reporting
… (Kept from several months to years)
A glimpse on Data Model
11
Historic & real-time insights per user per carrier. Sudden & drastic change might be suspicious.
User bucketing based on historic & real-time buying activity.
A glimpse on Data Model
12
Fraud Detection & Prevention
13
Shop with Confidence
System Overview
14
Cassandra
Fraud Detection & Prevention System
Sign
-in
in
fo
Business events (checkout, sell,…)
StaaS Oracle
Checkout Shipping … Payment Selling
Real-time Beacons data
Real-time Insights
Other data Machine
Learned Models
15
A glimpse on Data Model
Collected at sign-in & stored as key-value.
Pulled periodically to StaaS for training machine learned models.
Metrics collection for monitoring & alerting
16
System Overview
17
Transport (HTTP, …)
Scalable NIO servers based on Netty
Thousands of production machines
Cassandra
Stats for CPU, Memory, Disk, ..
…
agent agent agent agent …
Server
Server
Server
Server
Server In-memory grid (hazelcast) for rollups
A glimpse on Data Model
18
Granular data points
Rolled up metrics for various time intervals
Taste graph based recommendation system
19
Data Model
20
Tast
e G
rap
h
Tast
e V
ect
or
50 billion+ edges, 600 million+ writes, 3 billion+ reads, 30TB+ of data on SSD
System Overview
21
Business Event Stream
Recommendation system
Taste Graph Taste Vector
1. Item purchased.
2a. Write purchase edge. 2b. Read other edges for this user & item.
4. Req. recommendations.
5. Finds other items close to user’s coordinates.
6. Reco. shown to user
More, http://www.slideshare.net/planetcassandra/e-bay-nyc
Real-time Personalization Data Service
22
User performs search using keyword User gets personalized pages based on implicit/explicit profile
System Overview
23
Personalization Data Service
CacheMesh (write-back cache)
Heavy writes
eBay site pages (personalized)
Every few mins
in-memory MySQL & XMP DB
Cassandra Oracle (scaled out)
Hea
vy r
ead
s
Cache miss
user profiles
Application SOA services (multiple)
Data Warehouse
Data Model
24
• Keep column names short. • Don’t overload one CF with all the data:
- Split hot & cold data in separate CF. - Splitting & sharding can help compaction.
Static column families
25
Served by Cassandra
Social Signals
Manage signals via “Your Favorites”
26
Whole page is served by Cassandra
More, http://www.slideshare.net/jaykumarpatel/cassandra-at-ebay-13920376
Multi-Datacenter Deployment
27
Topology - NTS RF - 1:1 or 2:2 or 3:3 Read CL - ONE/QUORUM Write CL - ONE
Data is backed up periodically to protect against human or software error
User request has no datacenter affinity
Non-sticky load balancing
Multi-Datacenter Deployment
Topology - NTS RF – 1:1:1 or 2:2:2
Lessons & Best Practices
• One size does not fit all
– Use Cassandra for the right use cases.
• Choose proper Replication Factor and Consistency Level
– They alter latency, availability, durability, consistency and cost.
– Cassandra supports tunable consistency, but remember strong consistency is not free.
• Many ways to model data in Cassandra
– The best way depends on your use case and query patterns.
• De-normalize and duplicate for read performance
– But don’t de-normalize if you don’t need to.
http://www.slideshare.net/jaykumarpatel/cassandra-data-modeling-best-practices
29
Are you excited? Come Join Us!
30
Thank You @pateljay3001
#cassandra13
Top Related