Download - Cassandra at eBay

Transcript
Page 1: Cassandra at eBay

Cassandra at eBay

August  8,  2012  

Jay  Patel  Architect,  Pla6orm  Systems  @pateljay3001    

Time left: 29m 59s

Page 2: Cassandra at eBay

eBay  Marketplaces  §  97  million  acAve  buyers  and  sellers  

§  200+  million  items  

§  2  billion  page  views  each  day  §  80  billion  database  calls  each  day  §  5+  petabytes  of  site  storage  capacity  §  80+  petabytes  of  analyAcs  storage  capacity  

2

Page 3: Cassandra at eBay

How  do  we  scale  databases?  §  Shard  

–  PaIerns:  Modulus,  lookup-­‐based,  range,  etc.  

–  ApplicaAon  sees  only  logical  shard/database  

§  Replicate  –  Disaster  recovery,  read  availability/scalability  

§  Big  NOs  –  No  transacAons  –  No  joins  –  No  referenAal  integrity  constraints  

3

Page 4: Cassandra at eBay

We  like  Cassandra  §  MulA-­‐datacenter  (acAve-­‐acAve)  §  Availability  -­‐  No  SPOF  §  Scalability      

4

We  also  u)lize  MongoDB  &  HBase    

§  Write  performance  §  Distributed  counters  §  Hadoop  support  

Page 5: Cassandra at eBay

Are  we  replacing  RDBMS  with  NoSQL?  

§  Some  use  cases  don’t  fit  well  -­‐    sparse  data,  big  data,  schema  opAonal,  real-­‐Ame  analyAcs,  …  

§  Many  use  cases  don’t  need  top-­‐Aer  set-­‐ups  -­‐  logging,  tracking,  …  

5

Not  at  all!  But,  complemenAng.  

Page 6: Cassandra at eBay

A  glimpse  on  our  Cassandra  deployment    

§  Dozens  of  nodes  across  mulAple  clusters  

§  200  TB+  storage  provisioned  §  400M+  writes  &  100M+  reads  per  day,  and  growing  

§  QA,  LnP,  and  mulAple  ProducAon  clusters  

6

Page 7: Cassandra at eBay

Use  Cases  on  Cassandra  

Social  Signals  on  eBay  product  &  item  pages  Hunch  taste  graph  for  eBay  users  &  items    Time  series  use  cases  (many):  

§  Mobile  noAficaAon  logging  and  tracking  §  Tracking  for  fraud  detecAon  §  SOA  request/response  payload  logging  §  RedLaser  server  logs  and  analyAcs  

7

Page 8: Cassandra at eBay

8

Served  by  Cassandra  

Page 9: Cassandra at eBay

Manage  signals  via  “Your  Favorites”  

9

Whole  page  is  served  by  Cassandra  

Page 10: Cassandra at eBay

Why  Cassandra  for  Social  Signals?  

§  Need  scalable  counters  §  Need  real  (or  near)  Ame  analyAcs  on  collected  social  data  

§  Need  good  write  performance  

§  Reads  are  not  latency  sensiAve  

10

Page 11: Cassandra at eBay

Deployment  

11

Topology  -­‐  NTS  RF  -­‐  2:2  Read  CL  -­‐  ONE  Write  CL  –  ONE  

Data  is  backed  up  periodically  to  protect  against  human  or  sohware  error  

User  request  has  no  datacenter  affinity  

Non-­‐sAcky  load  balancing  

Page 12: Cassandra at eBay

Data  Model  

depends  on  query  paIerns  

12

Page 13: Cassandra at eBay

Data  Model  (simplified)  

13

Page 14: Cassandra at eBay

Wait…  

14

Duplicates!  

Oh,  toggle  buIon!      Signal  -­‐-­‐>  De-­‐signal  -­‐-­‐>  Signal…    

Page 15: Cassandra at eBay

Yes,  eventual  consistency!  

One  scenario  that  produces  duplicate  signals  in  UserLike  CF:  

1.  Signal        

2.  De-­‐signal  (1st  operaAon  is  not  propagated  to  all  replica)  

3.  Signal,  again  (1st  operaAon  is  not  propagated  yet!)  

15

So,  what’s  the  soluQon?   Later…  

Page 16: Cassandra at eBay

Social  Signals,  next  phase:    Real-­‐Qme  AnalyQcs  

16

§  Most  signaled  or  popular  items  per  affinity  groups  (category,  etc.)  

§  Aggregated  item  count  per  affinity  group  

Example  affinity  group  

Page 17: Cassandra at eBay

IniQal  Data  Model  for  real-­‐Qme  analyQcs    

Update  counters  for  both  individual  item    and  all  the  affinity  groups  that  item  belongs  to  

Items  in  an  affinitygroup    is  physically  stored    sorted  by  their  signal  count    

Page 18: Cassandra at eBay

Deployment,  next  phase  

Topology  -­‐  NTS  RF  -­‐  2:2:2  

Page 19: Cassandra at eBay

19

item1  

user2  

user1  

item2  

buy  

bid  

sell  watch  

Page 20: Cassandra at eBay

Graph  in  Cassandra  

Event  consumers  listen  for  site  events  (sell/bid/buy/watch)  &  populate  graph  in  Cassandra  

20

§  30  million+    writes  daily  §  14  billion+  edges  already  

§  Batch-­‐oriented  reads              (for  taste  vector  updates)  

Page 21: Cassandra at eBay

§  Mobile  noAficaAon  logging  and  tracking  

§  Tracking  for  fraud  detecAon  §  SOA  request/response  payload  logging  §  RedLaser  server  logs  and  analyAcs  

21

Page 22: Cassandra at eBay

A  glimpse  on  Data  Model  

Page 23: Cassandra at eBay

RedLaser  tracking  &  monitoring  console  

23

Page 24: Cassandra at eBay

Remember  the  duplicate  problem  in  Use  Case  #1?  

24

Let’s  see  some  opAons  we  considered  to  solve  this…  

That’s  all  about  the  use  cases..  

Page 25: Cassandra at eBay

OpQon  1  –  Make  ‘Like’  idempotent  for  UserLike  

§  Remove  Ame  (Ameuuid)  from  the  composite  column  name:  §  MulAple  signal  operaAons  are  now  Idempotent  

§  No  need  to  read  before  de-­‐signaling  (deleAng)  

25

   X                            Need  Ameuuid  for  ordering!                                          Already  have  a  user  with  more  than  1300  signals  

Page 26: Cassandra at eBay

OpQon  2  –  Use  strong  consistency  

§  Local  Quorum  – Won’t  help  us.  User  requests  are  not  geo-­‐load  balanced  (no  DC  affinity).  

§  Quorum    

– Won’t  survive  during  parAAon  between  DCs  (or,  one  of  the  DC  is  down).  Also,  adds  addiAonal  latency.  

 

26

   X              Need  to  survive!    

Page 27: Cassandra at eBay

OpQon  3  –  Adapt  to  eventual  consistency  

27

If  desire  survival!  

hIp://www.strangecosmos.com/content/item/101254.html  

Page 28: Cassandra at eBay

Adjustments  to  eventual  consistency  De-­‐signal  steps:  

–  Don’t  check  whether  item  is  already  signaled  by  a  user,  or  not  –  Read  all  (duplicate)  signals  from  UserLike_unordered  (new  CF  to  avoid  reading  

whole  row  from  UserLike)  –  Delete  those  signals  from  UserLike_unordered  and  UserLike    

   

28

SAll,  can  get  duplicate  signals  or  false  posiAves  as  there  is  a  ‘read  before  delete’.    

Not  a  full  story!  To  shield  further,  do  ‘repair  on  read’.  

Page 29: Cassandra at eBay

Lessons  &  Best  PracQces  •  Choose  proper  ReplicaAon  Factor  and  Consistency  Level.  

–  They  alter  latency,  availability,  durability,  consistency  and  cost.  –  Cassandra  supports  tunable  consistency,  but  remember  strong  consistency  is  not  free.  

•  Consider  all  overheads  in  capacity  planning.  

–  Replicas,  compacAon,  secondary  indexes,  etc.  

•  De-­‐normalize  and  duplicate  for  read  performance.    

–  But  don’t  de-­‐normalize  if  you  don’t  need  to.    

•  Many  ways  to  model  data  in  Cassandra.  

–  The  best  way  depends  on  your  use  case  and  query  paIerns.  More  on    hIp://ebaytechblog.com?p=1308  

Page 30: Cassandra at eBay

30

@pateljay3001    

#cassandra12    

Thank  You