Cassandra Day Atlanta 2015: Troubleshooting with Apache Cassandra

Click here to load reader

  • date post

    15-Jul-2015
  • Category

    Technology

  • view

    340
  • download

    1

Embed Size (px)

Transcript of Cassandra Day Atlanta 2015: Troubleshooting with Apache Cassandra

  • Troubleshooting Cassandranodetool & system.log deep dive

    J.B. Langston, Senior Support Engineer

  • Company Confidential 2014 DataStax, All Rights Reserved.

    Agenda

    2

    1 Things to keep in mind

    2 Useful Linux & Java tools

    3 Useful nodetool commands

    4 Things to look for in system.log

  • Company Confidential 2014 DataStax, All Rights Reserved.

    Troubleshooting Process

    3

    1 Ask what changed

    2 Determine which nodes have problems

    3 Find and understand errors

    4 Examine bottlenecks

    5 Determine root cause

    6 Take corrective action

  • Company Confidential 2014 DataStax, All Rights Reserved.

    Troubleshooting Process

    4

    1 Ask what changed

    2 Determine which nodes have problems

    3 Find and understand errors

    4 Examine bottlenecks

    5 Share relevant information

    6 Ask for help

  • Company Confidential 2014 DataStax, All Rights Reserved. 5

    What changed?

  • Company Confidential 2014 DataStax, All Rights Reserved. 5

    What changed?

    Did it work before?

  • Company Confidential 2014 DataStax, All Rights Reserved. 5

    What changed?

    Did it work before?

    What did you change?

  • Company Confidential 2014 DataStax, All Rights Reserved. 5

    What changed?

    Did it work before?

    What did you change? Settings

  • Company Confidential 2014 DataStax, All Rights Reserved. 5

    What changed?

    Did it work before?

    What did you change? Settings Application Code

  • Company Confidential 2014 DataStax, All Rights Reserved. 5

    What changed?

    Did it work before?

    What did you change? Settings Application Code Read/Write Load

  • Company Confidential 2014 DataStax, All Rights Reserved. 5

    What changed?

    Did it work before?

    What did you change? Settings Application Code Read/Write Load Data Volume

  • Company Confidential 2014 DataStax, All Rights Reserved. 5

    What changed?

    Did it work before?

    What did you change? Settings Application Code Read/Write Load Data Volume Hardware

  • Company Confidential 2014 DataStax, All Rights Reserved. 5

    What changed?

    Did it work before?

    What did you change? Settings Application Code Read/Write Load Data Volume Hardware Network

  • Company Confidential 2014 DataStax, All Rights Reserved. 5

    Does it work in another environment?

    What changed?

    Did it work before?

    What did you change? Settings Application Code Read/Write Load Data Volume Hardware Network

  • Company Confidential 2014 DataStax, All Rights Reserved. 5

    Does it work in another environment?

    Did you upgrade?

    What changed?

    Did it work before?

    What did you change? Settings Application Code Read/Write Load Data Volume Hardware Network

  • Company Confidential 2014 DataStax, All Rights Reserved. 5

    Does it work in another environment?

    Did you upgrade? Cassandra

    What changed?

    Did it work before?

    What did you change? Settings Application Code Read/Write Load Data Volume Hardware Network

  • Company Confidential 2014 DataStax, All Rights Reserved. 5

    Does it work in another environment?

    Did you upgrade? Cassandra Kernel

    What changed?

    Did it work before?

    What did you change? Settings Application Code Read/Write Load Data Volume Hardware Network

  • Company Confidential 2014 DataStax, All Rights Reserved. 5

    Does it work in another environment?

    Did you upgrade? Cassandra Kernel JVM

    What changed?

    Did it work before?

    What did you change? Settings Application Code Read/Write Load Data Volume Hardware Network

  • Company Confidential 2014 DataStax, All Rights Reserved. 5

    Does it work in another environment?

    Did you upgrade? Cassandra Kernel JVM Driver

    What changed?

    Did it work before?

    What did you change? Settings Application Code Read/Write Load Data Volume Hardware Network

  • Company Confidential 2014 DataStax, All Rights Reserved. 5

    Does it work in another environment?

    Did you upgrade? Cassandra Kernel JVM Driver

    Change one thing at a time!

    What changed?

    Did it work before?

    What did you change? Settings Application Code Read/Write Load Data Volume Hardware Network

  • Company Confidential 2014 DataStax, All Rights Reserved. 6

    Cassandra Processes

  • Company Confidential 2014 DataStax, All Rights Reserved. 6

    Cassandra Processes

    Startup

  • Company Confidential 2014 DataStax, All Rights Reserved. 6

    Cassandra Processes

    Startup

    Foreground

  • Company Confidential 2014 DataStax, All Rights Reserved. 6

    Cassandra Processes

    Startup

    Foreground Coordinating requests

  • Company Confidential 2014 DataStax, All Rights Reserved. 6

    Cassandra Processes

    Startup

    Foreground Coordinating requests Local reads

  • Company Confidential 2014 DataStax, All Rights Reserved. 6

    Cassandra Processes

    Startup

    Foreground Coordinating requests Local reads Local writes

  • Company Confidential 2014 DataStax, All Rights Reserved. 6

    Background

    Cassandra Processes

    Startup

    Foreground Coordinating requests Local reads Local writes

  • Company Confidential 2014 DataStax, All Rights Reserved. 6

    Background Flushes

    Cassandra Processes

    Startup

    Foreground Coordinating requests Local reads Local writes

  • Company Confidential 2014 DataStax, All Rights Reserved. 6

    Background Flushes Compactions

    Cassandra Processes

    Startup

    Foreground Coordinating requests Local reads Local writes

  • Company Confidential 2014 DataStax, All Rights Reserved. 6

    Background Flushes Compactions Garbage collections

    Cassandra Processes

    Startup

    Foreground Coordinating requests Local reads Local writes

  • Company Confidential 2014 DataStax, All Rights Reserved. 6

    Background Flushes Compactions Garbage collections Gossip

    Cassandra Processes

    Startup

    Foreground Coordinating requests Local reads Local writes

  • Company Confidential 2014 DataStax, All Rights Reserved. 6

    Background Flushes Compactions Garbage collections Gossip Hinted Handoff

    Cassandra Processes

    Startup

    Foreground Coordinating requests Local reads Local writes

  • Company Confidential 2014 DataStax, All Rights Reserved. 6

    Background Flushes Compactions Garbage collections Gossip Hinted Handoff Read Repair

    Cassandra Processes

    Startup

    Foreground Coordinating requests Local reads Local writes

  • Company Confidential 2014 DataStax, All Rights Reserved. 7

    System Resources

  • Company Confidential 2014 DataStax, All Rights Reserved. 7

    System Resources

    CPU

  • Company Confidential 2014 DataStax, All Rights Reserved. 7

    System Resources

    CPU Single Core utilization

  • Company Confidential 2014 DataStax, All Rights Reserved. 7

    System Resources

    CPU Single Core utilization Multi Core utilization

  • Company Confidential 2014 DataStax, All Rights Reserved. 7

    System Resources

    CPU Single Core utilization Multi Core utilization

    Memory

  • Company Confidential 2014 DataStax, All Rights Reserved. 7

    System Resources

    CPU Single Core utilization Multi Core utilization

    Memory Heap space

  • Company Confidential 2014 DataStax, All Rights Reserved. 7

    System Resources

    CPU Single Core utilization Multi Core utilization

    Memory Heap space Off-heap space

  • Company Confidential 2014 DataStax, All Rights Reserved. 7

    System Resources

    CPU Single Core utilization Multi Core utilization

    Memory Heap space Off-heap space OS page cache

  • Company Confidential 2014 DataStax, All Rights Reserved. 7

    System Resources

    CPU Single Core utilization Multi Core utilization

    Memory Heap space Off-heap space OS page cache

    Garbage Collection

  • Company Confidential 2014 DataStax, All Rights Reserved. 7

    Disk

    System Resources

    CPU Single Core utilization Multi Core utilization

    Memory Heap space Off-heap space OS page cache

    Garbage Collection

  • Company Confidential 2014 DataStax, All Rights Reserved. 7

    Disk Disk space

    System Resources

    CPU Single Core utilization Multi Core utilization

    Memory Heap space Off-heap space OS page cache

    Garbage Collection

  • Company Confidential 2014 DataStax, All Rights Reserved. 7

    Disk Disk space I/O bandwidth

    System Resources

    CPU Single Core utilization Multi Core utilization

    Memory Heap space Off-heap space OS page cache

    Garbage Collection

  • Company Confidential 2014 DataStax, All Rights Reserved. 7

    Disk Disk space I/O bandwidth

    Network

    System Resources

    CPU Single Core utilization Multi Core utilization

    Memory Heap space Off-heap space OS page cache

    Garbage Collection

  • Company Confidential 2014 DataStax, All Rights Reserved. 7

    Disk Disk space I/O bandwidth

    Network

    OS Resources/Limits

    System Resources

    CPU Single Core utilization Multi Core utilization

    Memory Heap space Off-heap space OS page cache

    Garbage Collection

  • Company Confident