Download - Cassandra at Disqus — SF Cassandra Users Group July 31st

C* @Disqus · July 31, 2013

Cassandra SF Meetup

1Thursday, August 1, 13

INTRO

Software Engineer at Disqus

Built the current Data Pipeline

Enjoy working on large ecosystems

Who am I?


SO YOU MADE SOME ANALYTICS

200,000 unique users creating

1,000,000 unique comments on

1,000,000 unique articles on

20,000 unique websites

Needed to build a system to track events from across the Disqus network. On a given day we have

4*10^214,000,000,000,000,000,000,000

4 sextillion (zetta)potential combinations PER DAY


INTROTHE BIG ONE


DESIGNING THE SYSTEM


3. ABILITY TO ACCESS A SUBSET IN REAL TIME

2. ABILITY TO QUERY AND JOIN LARGE DATA SETS

1. SCALABLE AND AVAILABLE DATA PIPELINE

GOALS


3. ABILITY TO ACCESS A SUBSET IN REAL TIME

2. ABILITY TO QUERY AND JOIN LARGE DATA SETS

1. SCALABLE AND AVAILABLE DATA PIPELINE

GOALS

This is where Cassandra comes in


DATA FORMAT

You need a format for your data


You need a format for your data

Avro

Thrift

Protobuf

JSON

DATA FORMAT


We chose JSON

Avro

Thrift

Protobuf

JSON

DATA FORMAT


At Disqus we do comments

{! "category": "comment",! "data": {! ! "text": "What's going on",! ! "author": "gjcourt"! },! "meta": {! ! "endpoint": "/event.js", ! ! "useragent": {! ! ! "flavor": { "version": "X" }, ! ! ! "browser": { "version": "6.0", "name": "Safari" }! ! }! },! "timestamp": 1375228800}

DATA FORMAT




DATA FORMAT


Random Aside

Handling time in python is a pain in the ass

RANDOM ASIDE

time.time()Return the time in seconds since the epoch as a floating point number. Note that even though the time is always returned as a floating point number, not all systems provide time with a better precision than 1 second. While this function normally returns non-decreasing values, it can return a lower value than a previous call if the system clock has been set back between the two calls.


Random Aside

Handling time in python is a pain in the ass

RANDOM ASIDE

time.time()Return the time in seconds since the epoch as a floating point number. Note that even though the time is always returned as a floating point number, not all systems provide time with a better precision than 1 second. While this function normally returns non-decreasing values, it can return a lower value than a previous call if the system clock has been set back between the two calls.

>>> print time.time(); print time.mktime(time.gmtime())1375244678.641375273478.0


PICKING A DATABASE IS HARD


Mainly because there are so many choices

PICKING A DATABASE


PICKING A DATABASE

In an early startup, opportunity cost is king

While the choice of a system is important there are a range of possible choices.

A system that provides value is more important than choosing a local maximum.


PICKING A DATABASE

We need a large sparse matrix

Requires horizontal scalability

Fast reads and inserts

High cardinality


PICKING A DATABASE

We need a large sparse matrix

Requires horizontal scalability

Fast reads and inserts

High cardinality

Almost rules out most RDBMS


PICKING A DATABASE

We chose Cassandra


PICKING A DATABASE

What made the difference

We wanted counters and 0.8.0 has this capability

Fast inserts and reads

Tunable consistency guarantees

Simple data model


DESIGNING A DATA MODEL


3. SCALABLE AND AVAILABLE

2. FAST AND ACCURATE COUNTERS

1. HIGH VOLUME SPARSE MATRIX (billions of dimensions)

DATA THAT SCALES


DATA MODEL

How do you store arbitrary dimensionality over time?

Cassandra is a 2D sorted array


DATA MODEL

A simple way to build a counter

CREATE TABLE counts ( key text, time_dimension text, value counter, PRIMARY KEY (key, time_dimension));


DATA MODEL


+--------------+-----------------+-----------------+-----------------+-----------------+|! ! ! | 2013 | 2013.7 | 2013.7.30 | 2013.7.30.0 || comment |-----------------+------------------------------------------------------|! ! ! | 1000 | 100 | 10 | 1 |+--------------+-----------------+-----------------+-----------------+-----------------+


DATA MODEL


+--------------+-----------------+-----------------+-----------------+-----------------+|! ! ! | 2013 | 2013.7 | 2013.7.30 | 2013.7.30.0 || comment |-----------------+------------------------------------------------------|! ! ! | 1000 | 100 | 10 | 1 |+--------------+-----------------+-----------------+-----------------+-----------------+

----------------------------+-----------------+-----------------+-----------------+-----------------+|! ! ! | 2013 | 2013.7 | 2013.7.30 | 2013.7.30.0 || comment.author.gjcourt |-----------------+------------------------------------------------------|! ! ! | 23 | 17 | 7 | 1 |----------------------------+-----------------+-----------------+-----------------+-----------------+

Dimensions are easy


DATA MODEL

And if you increment the time bucket 2013-07-31

+--------------+-----------------+-----------------+-----------------+-----------------+|! ! ! | 2013 | 2013.7 | 2013.7.30 | 2013.7.30.0 || comment |-----------------+------------------------------------------------------|! ! ! | 1001 | 101 | 10 | 1 |+--------------+-----------------+-----------------+-----------------+-----------------+

----------------------------+-----------------+-----------------+-----------------+-----------------+|! ! ! | 2013 | 2013.7 | 2013.7.30 | 2013.7.30.0 || comment.author.gjcourt |-----------------+------------------------------------------------------|! ! ! | 24 | 18 | 7 | 1 |----------------------------+-----------------+-----------------+-----------------+-----------------+

Dimensions are easy


DATA MODEL

Some major disadvantages

All time intervals are in the same row

Queries are non linear

Time buckets in lexical order

Dimensions can not be indexed

Rows can grow unbounded


DATA MODEL

A better version of counters

--------------------+-----------------+|! ! ! | 2013 || comment.year |-----------------+|! ! ! | 1000 |--------------------+-----------------+

---------------------+-----------------+-----------------+-----------------+|! ! ! | 2013.5 | 2013.6 | 2013.7 || comment.month |-----------------+-----------------+-----------------+|! ! ! | 96 | 78 | 100 |---------------------+-----------------+-----------------+-----------------+

---------------------+-----------------+-----------------+-----------------+|! ! ! | 2013.7.28 | 2013.7.29 | 2013.7.30 || comment.day |-----------------+-----------------+-----------------+|! ! ! | 8 | 6 | 13 |---------------------+-----------------+-----------------+-----------------+


DATA MODEL

This is a large improvement

Efficient range queries

Rollups are possible


DATA MODEL

However still has some problems

Dimensions are not indexed

Rows can grow unbounded


DATA MODEL

Remember the schema



DATA MODEL

Remember the schema


Should this be a <timestamp>?


DATA MODEL

A better version of counters

CREATE TABLE better_counts ( key text, time_dimension 'org.apache.cassandra.db.marshal.ReversedType' <timestamp>, value counter, PRIMARY KEY (key, time_dimension));


DATA MODEL

The problem with counters

Operations are NOT Idempotent

Limited protection for overcounting

https://issues.apache.org/jira/browse/CASSANDRA-4775




DATA MODEL

And you end up having to write code like this

def swallow_cassandra_timeouts(func): @wraps(func) def inner(*args, **kwargs): try: return func(*args, **kwargs) except TimedOutException, e: logger.warning("processor.pycassa.exception.timeout") except UnavailableException, e: # raise so that we retry this batch logger.error("processor.pycassa.exception.unavailable") raise CassandraError(e) except MaximumRetryException, e: logger.warning("processor.pycassa.exception.max_retry") except Exception, e: logger.error("processor.pycassa.exception.unknown") raise return inner


DATA MODEL

And this

if LOCAL: CASSANDRA_TIMEOUT = 60 CASSANDRA_RETRIES = 0elif "prod" in hostname: CASSANDRA_TIMEOUT = 2 # Seconds CASSANDRA_RETRIES = 0 # Noneelif "storm" in hostname: CASSANDRA_TIMEOUT = 0.2 CASSANDRA_RETRIES = 0else: # proxy (read only) CASSANDRA_TIMEOUT = 60 CASSANDRA_RETRIES = 3


DATA MODEL

And this too

CASSANDRA_CONFIG = { 'stats': { 'pool': PoolConfig(CASSANDRA_TIMEOUT, CASSANDRA_RETRIES, CASSANDRA_POOL_SIZE), 'cf': { 'counts': ColumnFamilyConfig(ConsistencyLevel.LOCAL_QUORUM, ConsistencyLevel.ONE), 'durable_counts': ColumnFamilyConfig(ConsistencyLevel.LOCAL_QUORUM, ConsistencyLevel.LOCAL_QUORUM), 'sets': ColumnFamilyConfig(ConsistencyLevel.LOCAL_QUORUM, ConsistencyLevel.LOCAL_QUORUM), } } }


DATA MODEL

And operations to Cassandra look like this

@swallow_cassandra_timeoutsdef side_effecting_function(): # insert/update into cassandra pass


DATA MODEL

Durable counts

CREATE TABLE durable_counts ( key text, time_dimension 'org.apache.cassandra.db.marshal.ReversedType'<timestamp>, random uuid, value int, PRIMARY KEY (key, time_dimension, random));


DATA MODEL

Durable counts

---------------------+----------------------------------------+----------------------------------------+|! ! ! | 2013-07-30 05:21:38+0000 | 2013-07-30 05:23:44+0000 ||! ! ! | eb401386-f420-11e2-a26b-002590024b08 | b320a95c-f240-11e2-a26b-002590024b08 || comment.year |----------------------------------------+----------------------------------------+|! ! ! | 20 | 50 |---------------------+----------------------------------------+----------------------------------------+

---------------------+----------------------------------------+----------------------------------------+|! ! ! | 2013-07-30 05:21:38+0000 | 2013-07-30 05:23:44+0000 ||! ! ! | eb401386-f420-11e2-a26b-002590024b08 | b320a95c-f240-11e2-a26b-002590024b08 || comment.month |----------------------------------------+----------------------------------------+|! ! ! | 20 | 50 |---------------------+----------------------------------------+----------------------------------------+

---------------------+----------------------------------------+----------------------------------------+|! ! ! | 2013-07-30 05:21:38+0000 | 2013-07-30 05:23:44+0000 ||! ! ! | eb401386-f420-11e2-a26b-002590024b08 | b320a95c-f240-11e2-a26b-002590024b08 || comment.day |----------------------------------------+----------------------------------------+|! ! ! | 20 | 50 |---------------------+----------------------------------------+----------------------------------------+


DATA MODEL

And even doing all that hackeryHive count C* counter % Similar C* durable counts % Similar

8101 8179 99.046338 8179 99.046338

7328 7390 99.161028 7390 99.161028

6255 6304 99.222715 6304 99.222715

6604 6665 99.150141 6665 99.150141

7700 7766 99.150141 7766 99.150141

5 week days of countable data


DATA MODEL

Over 99% accuracy100% (allegedly) counter parity


DATA MODEL

Since our data is time series what if you could view it that way


DATA MODEL

With arbitrary dimensionality


DATA MODEL

With arbitrary multi dimensionality


DATA MODEL

Sets (our first iteration)CREATE TABLE sets ( key text, time_dimension timestamp, element blob, value double, PRIMARY KEY (key, time_dimension));

Insert only workload. Items are deleted by TTL


DATA MODEL

Better SetsCREATE TABLE sets ( key text, time_dimension timestamp, element blob, deleted boolean, value double, PRIMARY KEY (key, time_dimension));

Insert only workload. When you want to delete, you insert with deleted set to true.

Read require you to iterate over all columns in chronological order. You sum values to calculate a score.


DATA MODEL

Counters with indexable dimensions CREATE TABLE catalog ( key text, time_dimension 'org.apache.cassandra.db.marshal.ReversedType' <timestamp>, dimension_1 text, dimension_1_val text, dimension_2 text, dimension_2_val text, ... value counter, PRIMARY KEY (key, time_dimension) );


DATA MODEL

Dimension Catalog CREATE TABLE catalog ( key text, dimension text, value text, PRIMARY KEY (key, dimension) );


DATA MODEL

Dimension Catalog CREATE TABLE catalog ( key text, dimension text, value text, PRIMARY KEY (key, dimension) );

cqlsh:> insert into catalog (key, dimension, value) values ('comment', 'author', 'gjcourt'); cqlsh:> insert into catalog (key, dimension, value) values ('comment', 'forum', 'disqus');cqlsh:> select dimension from catalog where key='comment';

dimension----------- author forum


WHERE ARE WE GOING


3. EXPLORE NEW AND INTERESTING DATA PRODUCTS

2. PRODUCTIZE OUR DATA PIPELINE

1. EVOLVE CONTENT RECOMMENDATION AND ADVERTISING

OUR 2013 MISSIONS


THE FUTURE

casscached

Comparable performance

2GB max “key” (instead of 1mb)

Tunable consistency levels

Useful for SSI, mat-views


THE FUTURE

Postgres Foreign Data Wrapper

Could use a cass_fdw


THE FUTURE

Graph of users and views

g.V('username','gjcourt').out('thread_views').in('thread_views').except('username', 'gjcourt')

The Netflix algorithm:

All articles that people that have viewed the thread I’m currently viewing have also viewed.


C* @Disqus · July 31, 2013

Cassandra SF Meetup

Thanks for listening

We’re hiring http://disqus.com/jobs/


https://disqus.com/jobs/

https://disqus.com/jobs/