Eventual  Consistency  !=  Hopeful  Consistency  

Embracing  Op:mis:c  Design  in  the  Persistence  Layer  

Who  am  I?  Christos  Kalantzis  Ne0lix  Inc.  Engineering  Manager  –  Cloud  Database  Engineering    @chriskalan  ckalantzis@neHlix.com  www.linkedin.com/in/christoskalantzis    

C*  Replica:on  &  Consistency  Recap  

•  C*  is  eventually  Consistent  –  That  means  it  WILL  get  there…eventually  

•  Eventually  is  not:  – A  day  from  now  – A  minute  from  now  – A  second  from  now  

•  In  most  cases  it  is  milliseconds  

•  C*  has  tunable  consistency  – All  

– Quorum  

– One  

C*  Replica:on  &  Consistency  Recap  

Remember  When...?  •  In  early  2000s,  Read  was  scaled  out  by  adding  replica:on  Slaves  to  a  Master  DB.  

•  Writes  went  to  a  single  Master,  Reads  went  to  Slaves  

       

Remember  When…?  •  Slaves  could  lose  transac:ons  

Remember  When…?  •  No  “Repair”  func:on  

Remember  When…?  •  We  trusted  it  •  We  s:ll  trust  it  

C*  Consistency  Concerns  •  “I  want  high  consistency  in  my  Reads/Writes  just  like  I  had  in  my  RDBMS  setup”  –  You  never  really  had  it.  

•  “I  want  my  DB  to  catch  integrity  issues”  – We’ve  been  turning  FK  off  for  years!  

•  See  Rails,  Grails  &  other  MVC  frameworks  •  Can  I  trust  that  C*  will  replicate  my  data  when  wri:ng  at  CL  1”  

NeHlix  Experiment  •  Created  a  mul:-­‐datacenter  C*  1.1.7  cluster  of  48  nodes  in  

each  DC  •  Put  load  on  C*  Cluster  

–  100K  total  opera:ons  per  second  (50  K  in  each  DC)  •  Wrote  1,000,000  records  at  CL1  in  one  DC  •  Read  same  1,000,000  records  at  CL1  in  other  DC  •  ALL  records  were  read  successfully  •  You  can  trust  it!  

NeHlix  Experiment  

Op:mis:c  vs.  Pessimis:c  Design  •  Pessimis:c  Design  – Design  with  high  consistency,  you  punish  your  users  99.9%  of  the  :me  •  Higher  consistency  =  higher  latency  •  Diminished  user  experience  

Op:mis:c  vs.  Pessimis:c  Design  •  Op:mis:c  Design  – Trust  your  data  store  

•  Know  your  business  and  your  applica:on  – Always  ask  yourself,  is  it  really  that  important?  

•  Handle  edge  cases  through  con:ngency  plans  

Low  Consistency  Example  1  

Low  Consistency  Example  2  

Hurdles  Faced  •  Engineers  are  stubborn!  –  1+1=2  ..  not  eventually  2  

•  Middle  management  is  scared  –  It’s  a  feat  you  convinced  them  to  use  C*  –  Now  you  got  to  convince  them  to  accept  low  consistency?  

•  Engaging  Product  team  to  implement  some  type  of  con:ngency  plans  

How  to  Overcome  those  Hurdles  •  Prove  it  through  a  POC  •  Show  them  the  benefits  of  an  improved  user  experience  –  Its  all  about  the  user  

•  You  may  be  working  for  the  wrong  company  – We’re  hiring!  –  jobs.neHlix.com  

Q&A  

Thank  you!