Cassandra at Disqus — SF Cassandra Users Group July 31st

download Cassandra at Disqus — SF Cassandra Users Group July 31st

of 63

  • date post

    08-May-2015
  • Category

    Technology

  • view

    892
  • download

    0

Embed Size (px)

description

San Francisco Cassandra Users Meetup Group: http://www.meetup.com/CassandraSF/

Transcript of Cassandra at Disqus — SF Cassandra Users Group July 31st

  • 1.C* @Disqus July 31, 2013 Cassandra SF Meetup 1Thursday, August 1, 13

2. INTRO Software Engineer at Disqus Built the current Data Pipeline Enjoy working on large ecosystems Who am I? 2Thursday, August 1, 13 3. SO YOU MADE SOME ANALYTICS 200,000 unique users creating 1,000,000 unique comments on 1,000,000 unique articles on 20,000 unique websites Needed to build a system to track events from across the Disqus network. On a given day we have 4*10^21 4,000,000,000,000,000,000,000 4 sextillion (zetta) potential combinations PER DAY 3Thursday, August 1, 13 4. INTROTHE BIG ONE 4Thursday, August 1, 13 5. DESIGNING THE SYSTEM 5Thursday, August 1, 13 6. 3. ABILITY TO ACCESS A SUBSET IN REAL TIME 2. ABILITY TO QUERY AND JOIN LARGE DATA SETS 1. SCALABLE AND AVAILABLE DATA PIPELINE GOALS 6Thursday, August 1, 13 7. 3. ABILITY TO ACCESS A SUBSET IN REAL TIME 2. ABILITY TO QUERY AND JOIN LARGE DATA SETS 1. SCALABLE AND AVAILABLE DATA PIPELINE GOALS This is where Cassandra comes in 7Thursday, August 1, 13 8. DATA FORMAT You need a format for your data 8Thursday, August 1, 13 9. You need a format for your data Avro Thrift Protobuf JSON DATA FORMAT 9Thursday, August 1, 13 10. We chose JSON Avro Thrift Protobuf JSON DATA FORMAT 10Thursday, August 1, 13 11. At Disqus we do comments { ! "category": "comment", ! "data": { ! ! "text": "What's going on", ! ! "author": "gjcourt" ! }, ! "meta": { ! ! "endpoint": "/event.js", ! ! "useragent": { ! ! ! "flavor": { "version": "X" }, ! ! ! "browser": { "version": "6.0", "name": "Safari" } ! ! } ! }, ! "timestamp": 1375228800 } DATA FORMAT 11Thursday, August 1, 13 12. At Disqus we do comments { ! "category": "comment", ! "data": { ! ! "text": "What's going on", ! ! "author": "gjcourt" ! }, ! "meta": { ! ! "endpoint": "/event.js", ! ! "useragent": { ! ! ! "flavor": { "version": "X" }, ! ! ! "browser": { "version": "6.0", "name": "Safari" } ! ! } ! }, ! "timestamp": 1375228800 } DATA FORMAT 12Thursday, August 1, 13 13. At Disqus we do comments { ! "category": "comment", ! "data": { ! ! "text": "What's going on", ! ! "author": "gjcourt" ! }, ! "meta": { ! ! "endpoint": "/event.js", ! ! "useragent": { ! ! ! "flavor": { "version": "X" }, ! ! ! "browser": { "version": "6.0", "name": "Safari" } ! ! } ! }, ! "timestamp": 1375228800 } DATA FORMAT 13Thursday, August 1, 13 14. At Disqus we do comments { ! "category": "comment", ! "data": { ! ! "text": "What's going on", ! ! "author": "gjcourt" ! }, ! "meta": { ! ! "endpoint": "/event.js", ! ! "useragent": { ! ! ! "flavor": { "version": "X" }, ! ! ! "browser": { "version": "6.0", "name": "Safari" } ! ! } ! }, ! "timestamp": 1375228800 } DATA FORMAT 14Thursday, August 1, 13 15. At Disqus we do comments { ! "category": "comment", ! "data": { ! ! "text": "What's going on", ! ! "author": "gjcourt" ! }, ! "meta": { ! ! "endpoint": "/event.js", ! ! "useragent": { ! ! ! "flavor": { "version": "X" }, ! ! ! "browser": { "version": "6.0", "name": "Safari" } ! ! } ! }, ! "timestamp": 1375228800 } DATA FORMAT 15Thursday, August 1, 13 16. Random Aside Handling time in python is a pain in the ass RANDOM ASIDE time.time() Return the time in seconds since the epoch as a oating point number. Note that even though the time is always returned as a oating point number, not all systems provide time with a better precision than 1 second. While this function normally returns non-decreasing values, it can return a lower value than a previous call if the system clock has been set back between the two calls. 16Thursday, August 1, 13 17. Random Aside Handling time in python is a pain in the ass RANDOM ASIDE time.time() Return the time in seconds since the epoch as a oating point number. Note that even though the time is always returned as a oating point number, not all systems provide time with a better precision than 1 second. While this function normally returns non-decreasing values, it can return a lower value than a previous call if the system clock has been set back between the two calls. >>> print time.time(); print time.mktime(time.gmtime()) 1375244678.64 1375273478.0 17Thursday, August 1, 13 18. PICKING A DATABASE IS HARD 18Thursday, August 1, 13 19. Mainly because there are so many choices PICKING A DATABASE 19Thursday, August 1, 13 20. PICKING A DATABASE In an early startup, opportunity cost is king While the choice of a system is important there are a range of possible choices. A system that provides value is more important than choosing a local maximum. 20Thursday, August 1, 13 21. PICKING A DATABASE We need a large sparse matrix Requires horizontal scalability Fast reads and inserts High cardinality 21Thursday, August 1, 13 22. PICKING A DATABASE We need a large sparse matrix Requires horizontal scalability Fast reads and inserts High cardinality Almost rules out most RDBMS 22Thursday, August 1, 13 23. PICKING A DATABASE We chose Cassandra 23Thursday, August 1, 13 24. PICKING A DATABASE We chose Cassandra 24Thursday, August 1, 13 25. PICKING A DATABASE What made the difference We wanted counters and 0.8.0 has this capability Fast inserts and reads Tunable consistency guarantees Simple data model 25Thursday, August 1, 13 26. DESIGNING A DATA MODEL 26Thursday, August 1, 13 27. 3. SCALABLE AND AVAILABLE 2. FAST AND ACCURATE COUNTERS 1. HIGH VOLUME SPARSE MATRIX (billions of dimensions) DATA THAT SCALES 27Thursday, August 1, 13 28. DATA MODEL How do you store arbitrary dimensionality over time? Cassandra is a 2D sorted array 28Thursday, August 1, 13 29. DATA MODEL A simple way to build a counter CREATE TABLE counts ( key text, time_dimension text, value counter, PRIMARY KEY (key, time_dimension) ); 29Thursday, August 1, 13 30. DATA MODEL A simple way to build a counter +--------------+-----------------+-----------------+-----------------+-----------------+ |! ! ! | 2013 | 2013.7 | 2013.7.30 | 2013.7.30.0 | | comment |-----------------+------------------------------------------------------ |! ! ! | 1000 | 100 | 10 | 1 | +--------------+-----------------+-----------------+-----------------+-----------------+ 30Thursday, August 1, 13 31. DATA MODEL A simple way to build a counter +--------------+-----------------+-----------------+-----------------+-----------------+ |! ! ! | 2013 | 2013.7 | 2013.7.30 | 2013.7.30.0 | | comment |-----------------+------------------------------------------------------ |! ! ! | 1000 | 100 | 10 | 1 | +--------------+-----------------+-----------------+-----------------+-----------------+ ----------------------------+-----------------+-----------------+-----------------+-----------------+ |! ! ! | 2013 | 2013.7 | 2013.7.30 | 2013.7.30.0 | | comment.author.gjcourt |-----------------+------------------------------------------------------ |! ! ! | 23 | 17 | 7 | 1 | ----------------------------+-----------------+-----------------+-----------------+-----------------+ Dimensions are easy 31Thursday, August 1, 13 32. DATA MODEL And if you increment the time bucket 2013-07-31 +--------------+-----------------+-----------------+-----------------+-----------------+ |! ! ! | 2013 | 2013.7 | 2013.7.30 | 2013.7.30.0 | | comment |-----------------+------------------------------------------------------ |! ! ! | 1001 | 101 | 10 | 1 | +--------------+-----------------+-----------------+-----------------+-----------------+ ----------------------------+-----------------+-----------------+-----------------+-----------------+ |! ! ! | 2013 | 2013.7 | 2013.7.30 | 2013.7.30.0 | | comment.author.gjcourt |-----------------+------------------------------------------------------ |! ! ! | 24 | 18 | 7 | 1 | ----------------------------+-----------------+-----------------+-----------------+-----------------+ Dimensions are easy 32Thursday, August 1, 13 33. DATA MODEL Some major disadvantages All time intervals are in the same row Queries are non linear Time buckets in lexical order Dimensions can not be indexed Rows can grow unbounded 33Thursday, August 1, 13 34. DATA MODEL A better version of counters --------------------+-----------------+ |! ! ! | 2013 | | comment.year |-----------------+ |! ! ! | 1000 | --------------------+-----------------+ ---------------------+-----------------+-----------------+-----------------+ |! ! ! | 2013.5 | 2013.6 | 2013.7 | | comment.month |-----------------+-----------------+-----------------+ |! ! ! | 96 | 78 | 100 | ---------------------+-----------------+-----------------+-----------------+ ---------------------+-----------------+-----------------+-----------------+ |! ! ! | 2013.7.28 | 2013.7.29 | 2013.7.30 | | comment.day |-----------------+-----------------+-----------------+ |! ! ! | 8 | 6 | 13 | ---------------------+-----------------+-----------------+-----------------+ 34Thursday, August 1, 13 35. DATA MODEL This is a large improvement Efficient range queries Rollups are possible 35Thursday, August 1, 13 36. DATA MODEL However still has some problems Dimensions are not indexed Rows can grow unbounded 36Thursday, August 1, 13 37. DATA MODEL Remember the schema CREATE TABLE counts ( key text, time_dimension text, value counter, PRIMARY KEY (key, time_dimension) ); 37Thursday, August 1, 13 38. DATA MODEL Remember the schema CREATE TABLE counts ( key text, time_dimension text, value counter, PRIMARY KEY (key, time_dimension) ); 38Thursday, August 1, 13 39. DATA MODEL Remember the schema CREATE TABLE counts ( key text, time_dimension text, value counter, PRIMARY KEY (key, time_dimension) ); Should this be a ? 39Thursday, August 1, 13 40. DATA MODEL A better version of counters CREATE TABLE better_counts ( key text, time_dimension 'org.apache.cassandra.db.marshal.ReversedType' , value counter, PRIMARY KEY (key, time_dimension) ); 40Thursday, August 1, 13 41. DATA MODEL The problem with counters Operations are NOT Idempotent Limited protection for overcounting https://issues.apache.org/jira/browse/CASSANDRA-4775 41Thursday, August 1, 13 42. DATA MODEL And you end