Cassandra introduction 2016

download Cassandra introduction 2016

of 80

  • date post

    11-Jan-2017
  • Category

    Technology

  • view

    513
  • download

    0

Embed Size (px)

Transcript of Cassandra introduction 2016

  • Introduction to CassandraDuyHai DOANApache Cassandra Evangelist

  • @doanduyhai

    Datastax Founded in April 2010

    We contribute a lot to Apache Cassandra

    400+ customers (25 of the Fortune 100), 450+ employees

    Headquarter in San Francisco Bay area

    EU headquarter in London, offices in France and Germany

    Datastax Enterprise = OSS Cassandra + extra features

    2

  • @doanduyhai

    Cassandra history created at Facebook

    open-sourced since 2008

    current version: 3.4

    column-oriented distributed table

    3

  • 5 Cassandra key points Linear scalability Continuous availability Multi Data-center native Operational simplicity Spark integration

  • @doanduyhai

    1) Linear scalability

    5

    C*

    C*C*

    NetcoSports 3 nodes, 3GB

    1k+ nodes, PB+

    YOU

  • @doanduyhai

    2) Continuous availability

    6

    thanks to the Dynamo architecture

  • @doanduyhai

    3) Multi Data-centers

    7

    out-of-the-box (config only)

    AWS config for multi-regions DCs

    GCE support

    Microsoft Azure support

    CloudStack support

  • @doanduyhai

    Multi DC usagesData locality, disaster recovery

    8

    C*

    C*

    C*

    C*

    C* C*

    C* C* C*

    C*

    C*

    C*

    C*

    New York (DC1) London (DC2)

    Async replication

  • @doanduyhai

    Multi DC usagesVirtual DC for workload segregation

    9

    C*

    C*

    C*

    C*

    C* C*

    C* C* C*

    C*

    C*

    C*

    C*

    Production (LIVE)

    Analytics(Spark)

    Async replication

    Same room

  • @doanduyhai

    Multi DC usagesProd data copy for back-up/benchmark

    10

    C*

    C*

    C*

    C*

    C* C*

    C* C* C*

    C*

    C*

    C*

    C*

    Use LOCAL_XXX Consistency

    Levels

    My tiny test DCREAD-ONLY!!!

    Async replication

  • @doanduyhai

    4) Operational simplicity

    11

    1 node = 1 process + 2 config files (cassandra.yaml + cassandra-rackdc.properties)

    No role between nodes, perfect symmetry

    deployment automation

    OpsCenter* for monitoring provisioning services (repair, performance, )

    * only with Datastax Enterprise from Cassandra 3.x

  • @doanduyhai

    4) Operational simplicity

    12

  • @doanduyhai

    5) Eco System

    13

    Apache Spark Apache Cassandra integration analytics joins, aggregation SparkSQL/Dataframe integration with CQL (predicates push down)

    Apache Zeppelin Apache Cassandra integration web-based notebook tabular/graph display

  • 14

    Q & A

    ! "

  • Main Cassandra use-cases

  • @doanduyhai

    Cassandra use-cases

    16 Messaging

    Collections/ Playlists

    Fraud detection

    Recommendation/ Personalization

    Internet of things/ Sensor data

  • @doanduyhai

    Cassandra use-cases

    17 Messaging

    Collections/ Playlists

    Fraud detection

    Recommendation/ Personalization

    Internet of things/ Sensor data

  • 18

    Q & A

    ! "

  • Data Distribution

  • @doanduyhai

    The tokens

    20

    Random hash of #partition token = hash(#p)

    Hash: ] x, x ] hash range: 264 values

    x = 264/2

    C*

    C*

    C*

    C*

    C* C*

    C* C*

  • @doanduyhai

    Token ranges

    21

    A : x,3x4

    B : 3x4

    , 2x4

    C : 2x4

    , x4

    D : x4

    ,0

    E : 0, x4

    F : x4

    ,2x4

    G : 2x4

    ,3x4

    H : 3x4

    ,x

    C*

    C*

    C*

    C*

    C* C*

    C* C*

  • @doanduyhai

    Distributed tables

    22

    H

    A

    E

    D

    B C

    G F

    user_id1 user_id2 user_id3 user_id4 user_id5

    CREATE TABLE users( user_id int, , PRIMARY KEY(user_id)),

  • @doanduyhai

    Distributed tables

    23

    H

    A

    E

    D

    B C

    G F

    user_id1

    user_id2

    user_id3

    user_id4

    user_id5

  • @doanduyhai

    Linear scalability

    24

    H

    A

    E

    D

    B C

    G F

    Today = high load disk occupation 80%

    CPU 70%

    saturated memory

  • @doanduyhai

    Scaling out

    25

    H

    A

    E

    D

    B

    C

    G

    F

    I

    J

    +2 nodes disk occupation 50%

    CPU 50%

    memory

    Automatic data rebalancing each node gives up some tokens flag to throttle network bandwidth streamingthroughput

  • @doanduyhai

    Automatic data re-balancing with virtual nodes

    26

    A: B: C: D: E: F: G:H:

    A: B: C: D: E: F: G:H: I:J:

    +2 nodes

  • 27

    Q & A

    ! "

  • Replication Model & Consistency

  • @doanduyhai

    Failure tolerance

    29

    Replication factor (RF) = 3

    H

    A

    E

    D

    B C

    G F

    1

    2 3

    {A, H, G}

    {B, A, H} {C, B, A}

  • @doanduyhai

    Coordinator node

    30

    Responsible for handling requests (read/write)

    Every node can be coordinator masterless no SPOF proxy role

    H

    A

    E

    D

    B C

    G F

    coordinator

    request

    1

    2 3

  • @doanduyhai

    Consistency level

    31

    Tunable at runtime ONE QUORUM (strict majority w.r.t RF) ALL

    Applicable to any request (read/write)

  • @doanduyhai

    Consistency in action

    32

    B A A

    B A A

    Read ONE: A

    data replication in progress

    Write ONE: B

    ack

    RF = 3, Write ONE, Read ONE

  • @doanduyhai

    Consistency in action

    33

    B A A

    B A A

    Read QUORUM: A

    data replication in progress

    Write ONE: B

    ack

    RF = 3, Write ONE, Read QUORUM

  • @doanduyhai

    Consistency in action

    34

    B A A

    B A A

    Read ALL: B

    data replication in progress

    Write ONE: B

    ack

    RF = 3, Write ONE, Read ALL

  • @doanduyhai

    Last Write Win

    35

    H

    A

    E

    D

    B C

    G F

    coordinator

    Read the value back

    1

    2 3

    B (t2) A (t1)

    A (t1)

  • @doanduyhai

    Consistency in action

    36

    B B A

    B B A

    Read ONE: A

    data replication in progress

    Write QUORUM: B

    ack

    RF = 3, Write QUORUM, Read ONE

  • @doanduyhai

    Consistency in action

    37

    B B A

    B B A

    Read QUORUM: A

    data replication in progress

    Write QUORUM: B

    ack

    RF = 3, Write QUORUM, Read QUORUM

  • @doanduyhai

    Consistency level = trade-off

    38

  • @doanduyhai

    Consistency level

    39

    ONE Fast, may not read latest written value

  • @doanduyhai

    Consistency level

    40

    QUORUM Strict majority w.r.t. Replication Factor

    Good balance

  • @doanduyhai

    Consistency level

    41

    ALL Paranoid

    Slow, lost of high availability

  • @doanduyhai

    Consistency level common patterns

    42

    ONERead + ONEWrite available for read/write even (N-1) replicas down

    QUORUMRead + QUORUMWrite available for read/write even if (RF - 1) replica (s) down

  • 43

    Q & A

    ! "

  • Last Write Win & Compaction

  • @doanduyhai

    Last Write Win (LWW)

    45

    jdoe age name

    33 John DOE

    INSERT INTO users(login, name, age) VALUES('jdoe', 'John DOE', 33);

    #partition

  • @doanduyhai

    Last Write Win (LWW)

    46

    INSERT INTO users(login, name, age) VALUES('jdoe', 'John DOE', 33);

    jdoe age (t1) name (t1)

    33 John DOE

    auto-generated timestamp (s)

    .

  • @doanduyhai

    Last Write Win (LWW)

    47

    UPDATE users SET age = 34 WHERE login = 'jdoe';

    jdoe age (t1) name (t1)

    33 John DOE jdoe

    age (t2)

    34

    SSTable1 SSTable2

  • @doanduyhai

    Last Write Win (LWW)

    48

    DELETE age FROM users WHERE login = 'jdoe';

    jdoe age (t1) name (t1)

    33 John DOE jdoe

    age (t2)

    34

    SSTable1 SSTable2

    tombstone

    SSTable3

    jdoe age (t3)

  • @doanduyhai

    Last Write Win (LWW)

    49

    SELECT age FROM users WHERE login = 'jdoe';

    jdoe age (t1) name (t1)

    33 John DOE jdoe

    age (t2)

    34

    SSTable1 SSTable2 SSTable3

    jdoe age (t3)

    ? ? ?

  • @doanduyhai

    Last Write Win (LWW)

    50

    SELECT age FROM users WHERE login = 'jdoe';

    jdoe age (t1) name (t1)

    33 John DOE jdoe

    age (t2)

    34

    SSTable1 SSTable2 SSTable3

    jdoe age (t3)

  • @doanduyhai

    Compaction

    51

    SSTable1 SSTable2 SSTable3

    jdoe age (t3)

    jdoe

    age (t1) name (t1)

    33 John DOE jdoe

    age (t2)

    34

    New SSTable

    jdoe age (t3) name (t1)

    John DOE

  • Basic Data Modeling

  • @doanduyhai

    Table creation

    53

    CREATE TABLE users (login text,name text,age int,PRIMARY KEY(login));

    partition key (#partition)

  • @doanduyhai

    DML statements

    54

    INSERT INTO users(login, name, age) VALUES('