LA Cassandra Day 2015 - Testing Cassandra

download LA Cassandra Day 2015  - Testing Cassandra

of 43

  • date post

    15-Jul-2015
  • Category

    Software

  • view

    371
  • download

    4

Embed Size (px)

Transcript of LA Cassandra Day 2015 - Testing Cassandra

  • 2013 DataStax Confidential. Do not distribute without consent.

    @chbatey

    Christopher BateyTechnical Evangelist for Apache Cassandra

    LA 2015: Testing Cassandra applications

  • @chbatey

    Who am I?Based in LondonTechnical Evangelist for Apache CassandraWork on Stubbed CassandraHelp out Apache Cassandra users

    Previous: Cassandra backed apps at BSkyB

  • @chbatey

    Agenda Cassandra failure scenarios Developer tools Stubbed Cassandra

  • @chbatey

    Production?

    Application

  • @chbatey

    Production?

    ApplicationApplicationApplicationApplicationApplication

    replication factor: 3

    consistency: ONE

  • @chbatey

    Production?ApplicationApplicationApplicationApplicationApplication

    ApplicationApplicationApplicationApplicationApplication

    DC1

    DC2

  • @chbatey

    Read timeout

    Application C

    R1

    R2

    R3C=QUROUM

    Replication factor: 3

    timeout

    timeout

    Read timeout

  • @chbatey

    Read timeout Received acknowledgements Required acknowledgements Consistency level wasDataReceived

  • @chbatey

    Write timeout

    Application C

    R1

    R2

    R3C=QUROUM

    Replication factor: 3

    timeout

    timeout

    Write timeout

  • @chbatey

    Retrying writes Cassandra does not roll back!

  • @chbatey

    Write timeout Received acknowledgements Required acknowledgements Consistency level CAS and Batches are more complicated:- WriteType: SIMPLE, BATCH, UNLOGGED_BATCH,

    BATCH_LOG, CAS

  • @chbatey

    Batches BATCH_LOG- Timed out waiting for batch log replicas- Retry if you like BATCH- Written to batch log but timed out waiting for actual replica- Will eventually be committed (serial read) UNLOGGED_BATCH- Unknown - retry if you like

  • @chbatey

    Idempotent writes All writes are idempotent with the following exceptions:- Counters- lists

  • @chbatey

    Unavailable

    Application C

    R1

    R2

    R3C=QUROUM

    Replication factor: 3

    Unavailable

  • @chbatey

    Unavailable Alive Replicas Required Replicas Consistency

  • @chbatey

    Coordinator issue

    Application C

    R1

    R2

    R3C=QUROUM

    Replication factor: 3

    ???

    Set a socket read timeout!

  • @chbatey

    Unit & Integration testing Requirements: Quick to run, deterministic, not brittle!! Integration tests against an embedded Cassandra? cassandra-unit Integration tests against an external Cassandra running on developer / CI

    machines CCM Regular install Docker Vagrant Mocking the driver

  • @chbatey

    Acceptance testing

    ApplicationAcceptance

    test

    prepare data

    verification

  • @chbatey

    How do this if it was a HTTP service? Release it - great book Wiremock - mocking HTTP

    services Saboteur - adding network

    latency

    If you want to build a fault tolerant applicationyou better test faults!

  • @chbatey

    Stubbed Cassandra

    ApplicationAcceptance

    test

    prime on admin port (REST)

    verification on admin port

    Admin endpoints

    Native protocol

  • @chbatey

    Two sides of Scassandra Java-Acts as a unit/integration testing library-90% of effort on this sideStandalone-Runs as a separate process

  • @chbatey

    Java ClientEmbeds Stubbed Cassandra in a Java library- Start / stop the server- Prime the server to return rows and/or errors- Verify exactly the queries your application has executed

  • @chbatey

    Starting / Stopping

    @ClassRulepublic static final ScassandraServerRule SCASSANDRA = new ScassandraServerRule();

    Scassandra scassandra = ScassandraFactory.createServer();

    scassandra.start();

    PrimingClient primingClient = scassandra.primingClient();

    ActivityClient activityClient = scassandra.activityClient();

  • @chbatey

    Activity Client

    Query Query text Consistency

    PrepreparedStatementExecution Prepared statement text Bound variables

    public List retrieveQueries();

    public List retrievePreparedStatementExecutions()

    public List retrieveConnections();

    public void clearAllRecordedActivity() public void clearConnections();

    public void clearQueries();

    public void clearPreparedStatementExecutions();

  • @chbatey

    Priming Client

    PrimingRequest Either a Query or PreparedStatement Query text or QueryPattern (regex) Consistency (default all) Result (success, read timeout, unavailable etc) Rows for successful response Column types for rows Variable types for prepared statements

    public void prime(PrimingRequest prime) public List retrievePreparedPrimes()public List retrieveQueryPrimes()

    public void clearAllPrimes()public void clearQueryPrimes()public void clearPreparedPrimes()

  • @chbatey

    Example timepublic interface PersonDao { void connect(); void disconnect(); List retrievePeople(); List retrievePeopleByName(String firstName); void storePerson(Person person);}

  • @chbatey

    Testing the connect method@Testpublic void shouldConnectToCassandraWhenConnectCalled() { //given activityClient.clearConnections(); //when underTest.connect(); //then assertTrue("Expected at least one connection", activityClient.retrieveConnections().size() > 0); }

  • @chbatey

    Testing retrievepublic List retrieveNames() { ResultSet result; try { Statement statement = new SimpleStatement("select * from person"); statement.setConsistencyLevel(ConsistencyLevel.QUORUM); result = session.execute(statement); } catch (ReadTimeoutException e) { throw new UnableToRetrievePeopleException(); } List people = new ArrayList(); for (Row row : result) { people.add(new Person(row.getString("first_name"), row.getInt("age"))); } return people;}

  • @chbatey

    Test the query + consistency@Testpublic void testQueryIssuedWithCorrectConsistencyUsingMatcher() { //given Query expectedQuery = Query.builder() .withQuery("select * from person") .withConsistency("QUORUM").build(); //when underTest.retrievePeople(); //then assertThat(activityClient.retrieveQueries(), containsQuery(expectedQuery));}

    Hamcrest matcher

  • @chbatey

    Testing behaviour@Testpublic void testRetrievingOfNames() throws Exception { // given Map row = ImmutableMap.of( "first_name", "Chris", "last_name", "Batey", "age", 29); primingClient.prime(PrimingRequest.queryBuilder() .withQuery("select * from person") .withColumnTypes(column("age", PrimitiveType.INT)) .withRows(row) .build()); //when List names = underTest.retrievePeople(); //then assertEquals(1, names.size()); assertEquals(Chris Batey", names.get(0).getName());}

    Each Cassandra Row == Java map

    Tell Scassandra about your schema

  • @chbatey

    Testing errors@Test(expected = UnableToRetrievePeopleException.class) public void testHandlingOfReadRequestTimeout() throws Exception { // given PrimingRequest primeReadRequestTimeout = PrimingRequest.queryBuilder() .withQuery("select * from person") .withResult(Result.read_request_timeout) .build(); primingClient.prime(primeReadRequestTimeout); //when underTest.retrievePeople(); //then} Expecting custom exception

  • @chbatey

    Testing retries @Override public RetryDecision onReadTimeout(Statement statement, ConsistencyLevel cl, int requiredResponses, int receivedResponses, boolean dataRetrieved, int nbRetry) { if (nbRetry < configuredRetries) { return RetryDecision.retry(cl); } else { return RetryDecision.rethrow(); } } @Override public RetryDecision onWriteTimeout(Statement s, ConsistencyLevel cl, WriteType wt, int requiredAcks, int receivedAcks, int nbRetry) { return DefaultRetryPolicy.INSTANCE.onWriteTimeout(s, cl, wt, receivedAcks, receivedAcks, nbRetry); } @Override public RetryDecision onUnavailable(Statement statement, ConsistencyLevel cl, int requiredReplica, int aliveReplica, int nbRetry) { return DefaultRetryPolicy.INSTANCE.onUnavailable(statement, cl, requiredReplica, aliveReplica, nbRetry); }

  • @chbatey

    Testing retries@Testpublic void testRetriesConfiguredNumberOfTimes() throws Exception { PrimingRequest readTimeoutPrime = PrimingRequest.queryBuilder() .withQuery("select * from person") .withResult(Result.read_request_timeout) .build(); primingClient.prime(readTimeoutPrime); try { underTest.retrievePeople(); } catch (UnableToRetrievePeopleException e) { } assertEquals(CONFIGURED_RETRIES + 1, activityClient.retrieveQueries().size());}

  • @chbatey

    Coordinator issue

    Application C

    R1

    R2

    R3C=QUROUM

    Replication factor: 3

    ???

  • @chbatey

    Testing slow connection@Test(expected = UnableToSavePersonException.class) public void testThatSlowQueriesTimeout() throws Exception { // given PrimingRequest preparedStatementPrime = PrimingRequest.preparedStatementBuilder() .withQueryPattern("insert into person.*") .withVariableTypes(VARCHAR, INT, list(TIMESTAMP)) .withFixedDelay(1000) .build(); primingClient.prime(prep