Cassandra Day Atlanta 2015: Troubleshooting with Apache Cassandra

Click here to load reader

  • date post

    15-Jul-2015

  • Category

    Technology

  • view

    340

  • download

    1

TAGS:

Embed Size (px)

Transcript of Cassandra Day Atlanta 2015: Troubleshooting with Apache Cassandra

  • Troubleshooting Cassandranodetool & system.log deep dive

    J.B. Langston, Senior Support Engineer

  • Company Confidential 2014 DataStax, All Rights Reserved.

    Agenda

    2

    1 Things to keep in mind

    2 Useful Linux & Java tools

    3 Useful nodetool commands

    4 Things to look for in system.log

  • Company Confidential 2014 DataStax, All Rights Reserved.

    Troubleshooting Process

    3

    1 Ask what changed

    2 Determine which nodes have problems

    3 Find and understand errors

    4 Examine bottlenecks

    5 Determine root cause

    6 Take corrective action

  • Company Confidential 2014 DataStax, All Rights Reserved.

    Troubleshooting Process

    4

    1 Ask what changed

    2 Determine which nodes have problems

    3 Find and understand errors

    4 Examine bottlenecks

    5 Share relevant information

    6 Ask for help

  • Company Confidential 2014 DataStax, All Rights Reserved. 5

    What changed?

  • Company Confidential 2014 DataStax, All Rights Reserved. 5

    What changed?

    Did it work before?

  • Company Confidential 2014 DataStax, All Rights Reserved. 5

    What changed?

    Did it work before?

    What did you change?

  • Company Confidential 2014 DataStax, All Rights Reserved. 5

    What changed?

    Did it work before?

    What did you change? Settings

  • Company Confidential 2014 DataStax, All Rights Reserved. 5

    What changed?

    Did it work before?

    What did you change? Settings Application Code

  • Company Confidential 2014 DataStax, All Rights Reserved. 5

    What changed?

    Did it work before?

    What did you change? Settings Application Code Read/Write Load

  • Company Confidential 2014 DataStax, All Rights Reserved. 5

    What changed?

    Did it work before?

    What did you change? Settings Application Code Read/Write Load Data Volume

  • Company Confidential 2014 DataStax, All Rights Reserved. 5

    What changed?

    Did it work before?

    What did you change? Settings Application Code Read/Write Load Data Volume Hardware

  • Company Confidential 2014 DataStax, All Rights Reserved. 5

    What changed?

    Did it work before?

    What did you change? Settings Application Code Read/Write Load Data Volume Hardware Network

  • Company Confidential 2014 DataStax, All Rights Reserved. 5

    Does it work in another environment?

    What changed?

    Did it work before?

    What did you change? Settings Application Code Read/Write Load Data Volume Hardware Network

  • Company Confidential 2014 DataStax, All Rights Reserved. 5

    Does it work in another environment?

    Did you upgrade?

    What changed?

    Did it work before?

    What did you change? Settings Application Code Read/Write Load Data Volume Hardware Network

  • Company Confidential 2014 DataStax, All Rights Reserved. 5

    Does it work in another environment?

    Did you upgrade? Cassandra

    What changed?

    Did it work before?

    What did you change? Settings Application Code Read/Write Load Data Volume Hardware Network

  • Company Confidential 2014 DataStax, All Rights Reserved. 5

    Does it work in another environment?

    Did you upgrade? Cassandra Kernel

    What changed?

    Did it work before?

    What did you change? Settings Application Code Read/Write Load Data Volume Hardware Network

  • Company Confidential 2014 DataStax, All Rights Reserved. 5

    Does it work in another environment?

    Did you upgrade? Cassandra Kernel JVM

    What changed?

    Did it work before?

    What did you change? Settings Application Code Read/Write Load Data Volume Hardware Network

  • Company Confidential 2014 DataStax, All Rights Reserved. 5

    Does it work in another environment?

    Did you upgrade? Cassandra Kernel JVM Driver

    What changed?

    Did it work before?

    What did you change? Settings Application Code Read/Write Load Data Volume Hardware Network

  • Company Confidential 2014 DataStax, All Rights Reserved. 5

    Does it work in another environment?

    Did you upgrade? Cassandra Kernel JVM Driver

    Change one thing at a time!

    What changed?

    Did it work before?

    What did you change? Settings Application Code Read/Write Load Data Volume Hardware Network

  • Company Confidential 2014 DataStax, All Rights Reserved. 6

    Cassandra Processes

  • Company Confidential 2014 DataStax, All Rights Reserved. 6

    Cassandra Processes

    Startup

  • Company Confidential 2014 DataStax, All Rights Reserved. 6

    Cassandra Processes

    Startup

    Foreground

  • Company Confidential 2014 DataStax, All Rights Reserved. 6

    Cassandra Processes

    Startup

    Foreground Coordinating requests

  • Company Confidential 2014 DataStax, All Rights Reserved. 6

    Cassandra Processes

    Startup

    Foreground Coordinating requests Local reads

  • Company Confidential 2014 DataStax, All Rights Reserved. 6

    Cassandra Processes

    Startup

    Foreground Coordinating requests Local reads Local writes

  • Company Confidential 2014 DataStax, All Rights Reserved. 6

    Background

    Cassandra Processes

    Startup

    Foreground Coordinating requests Local reads Local writes

  • Company Confidential 2014 DataStax, All Rights Reserved. 6

    Background Flushes

    Cassandra Processes

    Startup

    Foreground Coordinating requests Local reads Local writes

  • Company Confidential 2014 DataStax, All Rights Reserved. 6

    Background Flushes Compactions

    Cassandra Processes

    Startup

    Foreground Coordinating requests Local reads Local writes

  • Company Confidential 2014 DataStax, All Rights Reserved. 6

    Background Flushes Compactions Garbage collections

    Cassandra Processes

    Startup

    Foreground Coordinating requests Local reads Local writes

  • Company Confidential 2014 DataStax, All Rights Reserved. 6

    Background Flushes Compactions Garbage collections Gossip

    Cassandra Processes

    Startup

    Foreground Coordinating requests Local reads Local writes

  • Company Confidential 2014 DataStax, All Rights Reserved. 6

    Background Flushes Compactions Garbage collections Gossip Hinted Handoff

    Cassandra Processes

    Startup

    Foreground Coordinating requests Local reads Local writes

  • Company Confidential 2014 DataStax, All Rights Reserved. 6

    Background Flushes Compactions Garbage collections Gossip Hinted Handoff Read Repair

    Cassandra Processes

    Startup

    Foreground Coordinating requests Local reads Local writes

  • Company Confidential 2014 DataStax, All Rights Reserved. 7

    System Resources

  • Company Confidential 2014 DataStax, All Rights Reserved. 7

    System Resources

    CPU

  • Company Confidential 2014 DataStax, All Rights Reserved. 7

    System Resources

    CPU Single Core utilization

  • Company Confidential 2014 DataStax, All Rights Reserved. 7

    System Resources

    CPU Single Core utilization Multi Core utilization

  • Company Confidential 2014 DataStax, All Rights Reserved. 7

    System Resources

    CPU Single Core utilization Multi Core utilization

    Memory

  • Company Confidential 2014 DataStax, All Rights Reserved. 7

    System Resources

    CPU Single Core utilization Multi Core utilization

    Memory Heap space

  • Company Confidential 2014 DataStax, All Rights Reserved. 7

    System Resources

    CPU Single Core utilization Multi Core utilization

    Memory Heap space Off-heap space

  • Company Confidential 2014 DataStax, All Rights Reserved. 7

    System Resources

    CPU Single Core utilization Multi Core utilization

    Memory Heap space Off-heap space OS page cache

  • Company Confidential 2014 DataStax, All Rights Reserved. 7

    System Resources

    CPU Single Core utilization Multi Core utilization

    Memory Heap space Off-heap space OS page cache

    Garbage Collection

  • Company Confidential 2014 DataStax, All Rights Reserved. 7

    Disk

    System Resources

    CPU Single Core utilization Multi Core utilization

    Memory Heap space Off-heap space OS page cache

    Garbage Collection

  • Company Confidential 2014 DataStax, All Rights Reserved. 7

    Disk Disk space

    System Resources

    CPU Single Core utilization Multi Core utilization

    Memory Heap space Off-heap space OS page cache

    Garbage Collection

  • Company Confidential 2014 DataStax, All Rights Reserved. 7

    Disk Disk space I/O bandwidth

    System Resources

    CPU Single Core utilization Multi Core utilization

    Memory Heap space Off-heap space OS page cache

    Garbage Collection

  • Company Confidential 2014 DataStax, All Rights Reserved. 7

    Disk Disk space I/O bandwidth

    Network

    System Resources

    CPU Single Core utilization Multi Core utilization

    Memory Heap space Off-heap space OS page cache

    Garbage Collection

  • Company Confidential 2014 DataStax, All Rights Reserved. 7

    Disk Disk space I/O bandwidth

    Network

    OS Resources/Limits

    System Resources

    CPU Single Core utilization Multi Core utilization

    Memory Heap space Off-heap space OS page cache

    Garbage Collection

  • Company Confident

Apache Cassandra in Action - O'Reilly Cassandra in Action...Apache Cassandra in Action. Why Cassandra? ... Cassandra in production. FUD? ... Inside CFs, columns are dynamic
Apache Cassandra in Action - O'Reilly Cassandra in Action...Apache Cassandra in Action. Why Cassandra? ... Cassandra in production. FUD? ... Inside CFs, columns are dynamic
Cassandra Community Webinar - Introduction To Apache Cassandra 1.2
Cassandra Community Webinar - Introduction To Apache Cassandra 1.2
Cassandra + Hadoop: Analisi Batch con Apache Cassandra
Cassandra + Hadoop: Analisi Batch con Apache Cassandra
Cassandra Day Atlanta 2015: Python & Cassandra
Cassandra Day Atlanta 2015: Python & Cassandra
Apache Cassandra 2.0
Apache Cassandra 2.0
Amazon Managed Apache Cassandra Service - Developer Guide Cassandra Query Language (CQL) is the primary language for communicating with Apache Cassandra. Amazon Managed Apache Cassandra
Amazon Managed Apache Cassandra Service - Developer Guide Cassandra Query Language (CQL) is the primary language for communicating with Apache Cassandra. Amazon Managed Apache Cassandra
Apache Cassandra overview
Apache Cassandra overview
CASSANDRA + SIGNALFX INTEGRATION CASSANDRA + SIGNALFX INTEGRATION What is Cassandra? Apache Cassandra
CASSANDRA + SIGNALFX INTEGRATION CASSANDRA + SIGNALFX INTEGRATION What is Cassandra? Apache Cassandra
Presentation of Apache Cassandra
Presentation of Apache Cassandra
Do more Apache Cassandra distributed database work .Do more Apache Cassandra distributed database
Do more Apache Cassandra distributed database work .Do more Apache Cassandra distributed database
Taller Apache Cassandra - .Introducci³n Que es Apache Cassandra 3 Apache Cassandra es un motor de
Taller Apache Cassandra - .Introducci³n Que es Apache Cassandra 3 Apache Cassandra es un motor de
Pycon 2012 Apache Cassandra
Pycon 2012 Apache Cassandra
Apache Cassandra in Bangalore - Cassandra Internals and Performance
Apache Cassandra in Bangalore - Cassandra Internals and Performance
Cassandra Day London 2015: Apache Cassandra Anti-Patterns
Cassandra Day London 2015: Apache Cassandra Anti-Patterns
Apache Cassandra - part 2
Apache Cassandra - part 2
Cassandra Summit 2014: Apache Cassandra on Pivotal CloudFoundry
Cassandra Summit 2014: Apache Cassandra on Pivotal CloudFoundry
Introduction to Apache Cassandra
Introduction to Apache Cassandra
Apache Cassandra at Target - Cassandra Summit 2014
Apache Cassandra at Target - Cassandra Summit 2014
Apache Cassandra™ Apache Cassandra 1.0 Documentation Introduction to Apache Cassandra Apache Cassandra is a free, open-source, distributed database system for managing large amounts
Apache Cassandra™ Apache Cassandra 1.0 Documentation Introduction to Apache Cassandra Apache Cassandra is a free, open-source, distributed database system for managing large amounts
Realtime Analytics with Apache Cassandra .Realtime Analytics with Apache Cassandra Tom Wilkie Founder
Realtime Analytics with Apache Cassandra .Realtime Analytics with Apache Cassandra Tom Wilkie Founder
View more