Cassandra introduction at FinishJUG

download Cassandra introduction at FinishJUG

of 94

  • date post

    18-Jul-2015
  • Category

    Technology

  • view

    123
  • download

    1

Embed Size (px)

Transcript of Cassandra introduction at FinishJUG

  • @doanduyhai

    Introduction to CassandraDuyHai DOAN, Technical Advocate

  • @doanduyhai

    Who Am I ?!Duy Hai DOAN Cassandra technical advocate talks, meetups, confs open-source devs (Achilles, ) OSS Cassandra point of contact

    duy_hai.doan@datastax.com @doanduyhai

    2

  • @doanduyhai

    Datastax! Founded in April 2010

    We contribute a lot to Apache Cassandra

    400+ customers (25 of the Fortune 100), 200+ employees

    Headquarter in San Francisco Bay area

    EU headquarter in London, offices in France and Germany

    Datastax Enterprise = OSS Cassandra + extra features

    3

  • @doanduyhai

    Agenda!Architecture Cluster, Replication, Consistency

    Data model Last Write Win (LWW), CQL basics, From SQL to CQL,

    Lightweight Transaction

    DSE

    Use Cases

    4

  • @doanduyhai

    Cassandra history!NoSQL database created at Facebook open-sourced since 2008 current version = 2.1 column-oriented distributed table

    5

  • @doanduyhai

    Cassandra 5 key facts!Key fact 1: linear scalability

    C*

    C* C*

    NetcoSports 3 nodes, 3GB

    1k+ nodes, PB+

    YOU

    6

  • @doanduyhai

    Cassandra 5 key facts!Key fact 2: continuous availability (100% up-time) resilient architecture (Dynamo)

    7

  • @doanduyhai

    Cassandra 5 key facts!Key fact 3: multi-data centers out-of-the-box (config only) AWS conf for multi-region DCs GCE/CloudStack support Microsoft Azure

    8

  • @doanduyhai

    Multi-DC usages!

    New York (DC1) London (DC2)

    Data-locality, disaster recovery

    n2

    n3

    n4

    n5

    n6

    n7

    n8

    n1

    n2

    n3

    n4 n5

    n1

    Async replication

    9

  • @doanduyhai

    Multi-DC usages!Workload segregation/virtual DC

    n2

    n3

    n4

    n5

    n6

    n7

    n8

    n1

    n2

    n3

    n4 n5

    n1

    Production (Live)

    Analytics (Spark/Hadoop)

    Same room

    Async replication

    10

  • @doanduyhai

    Multi-DC usages!Prod data copy for testing/benchmarking

    n2

    n3

    n4

    n5

    n6

    n7

    n8

    n1

    n2

    n3 n1

    Use LOCAL

    consistency

    My tiny test cluster

    Data copy

    NEVER WRITE HERE !!!

    11

  • @doanduyhai

    Cassandra 5 key facts!Key fact 4: operational simplicity 1 node = 1 process + 2 config file (main + IP) deployment automation OpsCenter for monitoring

    12

  • @doanduyhai

    Cassandra 5 key facts!

    13

  • @doanduyhai

    Cassandra 5 key facts!Key fact 5: analytics combo Cassandra + Spark = awesome ! realtime streaming/analytics/aggregation

    14

  • Cassandra architecture!Cluster

    Replication Consistency

  • @doanduyhai

    Cassandra architecture!Cluster layer Amazon DynamoDB paper masterless architecture

    Data-store layer Google Big Table paper Columns/columns family

    16

  • @doanduyhai

    Data distribution!Random: hash of #partition token = hash(#p)

    Hash: ]-X, X]

    X = huge number (264/2)

    n1

    n2

    n3

    n4

    n5

    n6

    n7

    n8

    17

  • @doanduyhai

    Token Ranges!A: ]0, X/8]B: ] X/8, 2X/8]C: ] 2X/8, 3X/8]D: ] 3X/8, 4X/8]E: ] 4X/8, 5X/8]F: ] 5X/8, 6X/8]G: ] 6X/8, 7X/8]H: ] 7X/8, X]

    n1

    n2

    n3

    n4

    n5

    n6

    n7

    n8

    A

    B

    C

    D

    E

    F

    G

    H

    18

  • @doanduyhai

    Distributed Table!

    n1

    n2

    n3

    n4

    n5

    n6

    n7

    n8

    A

    B

    C

    D

    E

    F

    G

    H

    user_id1 user_id2 user_id3 user_id4 user_id5

    19

  • @doanduyhai

    Distributed Table!

    n1

    n2

    n3

    n4

    n5

    n6

    n7

    n8

    A

    B

    C

    D

    E

    F

    G

    H

    user_id1

    user_id2

    user_id3

    user_id4

    user_id5

    20

  • @doanduyhai

    Linear scalability!

    n1

    n2

    n3

    n4

    n5

    n6

    n7

    n8

    n1

    n2

    n3 n4

    n5

    n6

    n7

    n8 n9

    n10

    8 nodes 10 nodes

    21

  • @doanduyhai

    Failure tolerance!Replication Factor (RF) = 3

    n1

    n2

    n3

    n4

    n5

    n6

    n7

    n8

    1

    2

    3{B, A, H}

    {C, B, A}

    {D, C, B}

    A

    B

    C

    D

    E

    F

    G

    H

    22

  • @doanduyhai

    Coordinator node!Incoming requests (read/write)

    Coordinator node handles the request

    Every node can be coordinator masterless

    n1

    n2

    n3

    n4

    n5

    n6

    n7

    n8

    1

    2

    3

    coordinatorrequest

    23

  • @doanduyhai

    Consistency!Tunable at runtime ONE QUORUM (strict majority w.r.t. RF) ALL

    Apply both to read & write

    24

  • @doanduyhai

    Consistency in action!RF = 3, Write ONE, Read ONE

    B A A

    B A A

    Read ONE: A

    data replication in progress

    Write ONE: B

    25

  • @doanduyhai

    Consistency in action!RF = 3, Write ONE, Read QUORUM

    B A A

    Write ONE: B

    Read QUORUM: A

    B A A

    data replication in progress

    26

  • @doanduyhai

    Consistency in action!RF = 3, Write ONE, Read ALL

    B A A

    Read ALL: B

    B A A

    data replication in progress

    Write ONE: B

    27

  • @doanduyhai

    Consistency in action!RF = 3, Write QUORUM, Read ONE

    B B A

    Write QUORUM: B

    Read ONE: A

    B B A

    data replication in progress

    28

  • @doanduyhai

    Consistency in action!RF = 3, Write QUORUM, Read QUORUM

    B B A

    Read QUORUM: B

    B B A

    data replication in progress

    Write QUORUM: B

    29

  • @doanduyhai

    Consistency trade-off!

    30

  • @doanduyhai

    Consistency level!

    ONE Fast, may not read latest written value

    31

  • @doanduyhai

    Consistency level!

    QUORUM Strict majority w.r.t. Replication Factor

    Good balance

    32

  • @doanduyhai

    Consistency level!

    ALL Paranoid

    Slow, no high availability

    33

  • @doanduyhai

    Consistency summary!

    ONERead + ONEWrite available for read/write even (N-1) replicas down

    QUORUMRead + QUORUMWrite available for read/write even 1+ replica down

    34

  • Q & R

    ! " !

  • Data model!Last Write Win!

    CQL basics!From SQL to CQL!

    Lightweight Transaction!

  • @doanduyhai

    Cassandra Write Path!

    Commit log1

    . . .

    1

    Commit log2

    Commit logn

    Memory

    37

  • @doanduyhai

    Cassandra Write Path!Memory

    Commit log1

    . . .

    1

    Commit log2

    Commit logn

    MemTable Table1

    MemTable Table2

    MemTable TableN 2

    . . .

    38

  • @doanduyhai

    Cassandra Write Path!

    Commit log1

    Commit log2

    Commit logn

    Table1

    SSTable1

    Table2 Table3 SSTable2 SSTable3

    3

    Memory

    . . .

    39

  • @doanduyhai

    Cassandra Write Path!

    Commit log1

    Commit log2

    Commit logn

    Table1

    SSTable1

    Table2 Table3 SSTable2 SSTable3

    Memory . . . MemTable Table1 MemTable

    Table2 MemTable

    TableN

    . . .

    40

  • @doanduyhai

    Cassandra Write Path!

    Commit log1

    Commit log2

    Commit logn

    Table1

    SSTable1

    Table2 Table3 SSTable2 SSTable3

    Memory

    SSTable1 SSTable2

    SSTable3 . . .

    41

  • @doanduyhai

    Last Write Win (LWW)!

    jdoe age name

    33 John DOE

    INSERT INTO users(login, name, age) VALUES(jdoe, John DOE, 33);

    #partition

    42

  • @doanduyhai

    Last Write Win (LWW)!

    jdoe age (t1) name (t1)

    33 John DOE

    INSERT INTO users(login, name, age) VALUES(jdoe, John DOE, 33);

    auto-generated timestamp

    .

    43

  • @doanduyhai

    Last Write Win (LWW)!

    UPDATE users SET age = 34 WHERE login = jdoe;

    jdoe age (t1) name (t1)

    33 John DOE jdoe

    age (t2)

    34

    SSTable1 SSTable2

    44

  • @doanduyhai

    Last Write Win (LWW)!

    DELETE age FROM users WHERE login = jdoe;

    jdoe age (t3)

    tombstone

    jdoe age (t1) name (t1)

    33 John DOE jdoe

    age (t2)

    34

    SSTable1 SSTable2 SSTable3

    45

  • @doanduyhai

    Last Write Win (LWW)!

    SELECT age FROM users WHERE login = jdoe;

    ? ? ?

    SSTable1 SSTable2 SSTable3

    jdoe age (t3)

    jdoe

    age (t1) name (t1)

    33 John DOE jdoe

    age (t2)

    34

    46

  • @doanduyhai

    Last Write Win (LWW)!

    SELECT age FROM users WHERE login = jdoe;

    SSTable1 SSTable2 SSTable3

    jdoe age (t3)

    jdoe

    age (t1) name (t1)

    33 John DOE jdoe

    age (t2)

    34

    47

  • @doanduyhai

    Compaction!SSTable1 SSTable2 SSTable3

    jdoe age (t3)

    jdoe

    age (t1) name (t1)

    33 John DOE jdoe

    age (t2)

    34

    New SSTable