Distributed Counters in Cassandra (Cassandra Summit 2010)

Distributed Countersin Cassandra

Friday, August 13, 2010

Distributed Counters in Cassandra

I: Goal II: DesignIII: Implementation


I: Goal



Goal


Low Latency,Highly AvailableCounters


II: Design




I: Traditional Counter Design II: Abstract StrategyIII: Distributed Counter Design


I: Traditional Counter Design


Design



Traditional Counter DesignAtomic Counters

1. single machine2. one order of execution3. strongly consistent



Traditional Counter DesignProblems

1. SPOF / single master2. high latency3. manually sharded



Traditional Counter DesignQuestion

What constraints can we relax?


II: Abstract Strategy


Design



Abstract StrategyConstraints to Relax

1. one order of execution2. strong consistency



Abstract StrategyRelax: One Order of Execution

commutative operation:

- operations must be re-orderable



Abstract StrategyRelax: Strong Consistency

partitioned work:

- each op must occur once

- unique partition identifieridempotent repair:

- recognize ops from other partitions


III: Distributed Counter Design


Design



Distributed Counter DesignRequirements

1. commutative operation2. partitioned work3. idempotent repair



Distributed Counter DesignCommutative Operation

addition:

- commutative operation

- sum ops performed by all replicas

- a + b = b + a



Distributed Counter DesignPartitioned Work

each op assigned to a replica:

- every replica sums all of its ops



Distributed Counter DesignIdempotent Repair

save counts from remote replicas:

- keep highest count seenprevent multiple execution:

- do not transfer the target replica’s count


III: Implementation




I: Data Structure II: Single NodeIII: Eventual Consistency


I: Data Structure




Data StructureRequirements

local counts:

- incrementally updateremote counts:

- independently track partitions



Data StructureContext Format

list of (replica id, count) tuples:[(replica A, count), (replica B, count), ...]



Data StructureContext Mutations

local write:sum local count and write deltanote: memtable



Data StructureContext Mutations

remote repair:for each replica,keep highest count seen(local or from repair)


II: Single Node




Single NodeWrite Path

client1. construct column

- value: delta (big-endian long)

- clock: empty2. thrift: insert / batch_mutate




coordinator1. choose partition

- choose target replica

- requirement: ConsistencyLevel.ONE

2. construct clock- context format: [(target replica id, count delta)]




target replicainsert:

1. memtable does not contain column2. insert column into memtable




target replicaupdate:

1. memtable contains column2. retrieve existing column3. create new column

- context: sum local count w/ delta from write4. replace column in ConcurrentSkipListMap5. if failed to replace column, go to step 2.




Interesting Note:MTs are serialized to SSTs, as-is

- each SST encapsulates the updateswhen it was an MT

- local count total must be aggregated across the MT and all SSTs



Single NodeRead Pathtarget replicaread:

1. construct collating iterator over:- frozen snapshot of MT- all relevant SSTs

2. resolve column- local counts: sum- remote counts: keep max

3. construct value- sum local and remote counts (big-endian long)



Single NodeCompaction

replicacompaction:

1. construct collating iterator over all SSTs2. resolve every column in the CF

- local counts: sum

- remote counts: keep max3. write out resolved CF


III: Eventual Consistency




Eventual ConsistencyRead Repair

coordinator / replicaread repair:

1. calculate resolved (superset) CF

- resolve every column (local: sum, remote: max)2. return resolved CF to client



Eventual ConsistencyRead Repair

coordinator / replicaread repair:

1. calculate repair CF for each replica

- calculate diff CF between resolved and received

- modify columns to remove target replica’s counts2. send repair CF to each replica



Eventual ConsistencyAnti-Entropy Service

sending replicaAES:

1. follow normal AES code path

- calculate repair SST based on shared ranges

- send repair SST



Eventual ConsistencyAnti-Entropy Service

receiving replicaAES:

1. post-process streamed SST

- re-build streamed SST

- note: strip out local replica’s counts2. remove temporary descriptor3. add to SSTableTracker



Questions?


Issues:#580: Vector Clocks#1072: Distributed Counters

Related Work:Helland and Campbell, Building on Quicksand, CIDR (2009),Sections 5 & 6.

My email address:[email protected]


More Information


mailto:[email protected]

mailto:[email protected]

Distributed Counters in Cassandra (Cassandra Summit 2010)

Technology

Transcript of Distributed Counters in Cassandra (Cassandra Summit 2010)