Cassandra Day Atlanta 2015: Troubleshooting with Apache Cassandra

Troubleshooting Cassandranodetool & system.log deep dive

J.B. Langston, Senior Support Engineer

Company Confidential© 2014 DataStax, All Rights Reserved.

Agenda

2

1 Things to keep in mind

2 Useful Linux & Java tools

3 Useful nodetool commands

4 Things to look for in system.log


Troubleshooting Process

3

1 Ask what changed

2 Determine which nodes have problems

3 Find and understand errors

4 Examine bottlenecks

5 Determine root cause

6 Take corrective action


Troubleshooting Process

4

1 Ask what changed




5 Share relevant information

6 Ask for help

Company Confidential© 2014 DataStax, All Rights Reserved. 5

What changed?


What changed?

• Did it work before?


What changed?


• What did you change?


What changed?


• What did you change?• Settings


What changed?


• What did you change?• Settings• Application Code


What changed?


• What did you change?• Settings• Application Code• Read/Write Load


What changed?


• What did you change?• Settings• Application Code• Read/Write Load• Data Volume


What changed?


• What did you change?• Settings• Application Code• Read/Write Load• Data Volume• Hardware


What changed?


• What did you change?• Settings• Application Code• Read/Write Load• Data Volume• Hardware• Network


• Does it work in another environment?

What changed?





• Did you upgrade?

What changed?





• Did you upgrade?• Cassandra

What changed?





• Did you upgrade?• Cassandra• Kernel

What changed?





• Did you upgrade?• Cassandra• Kernel• JVM

What changed?





• Did you upgrade?• Cassandra• Kernel• JVM• Driver

What changed?





• Did you upgrade?• Cassandra• Kernel• JVM• Driver

• Change one thing at a time!

What changed?




Cassandra Processes


Cassandra Processes

• Startup


Cassandra Processes

• Startup

• Foreground


Cassandra Processes

• Startup

• Foreground• Coordinating requests


Cassandra Processes

• Startup

• Foreground• Coordinating requests• Local reads


Cassandra Processes

• Startup

• Foreground• Coordinating requests• Local reads• Local writes


• Background

Cassandra Processes

• Startup



• Background• Flushes

Cassandra Processes

• Startup



• Background• Flushes• Compactions

Cassandra Processes

• Startup



• Background• Flushes• Compactions• Garbage collections

Cassandra Processes

• Startup



• Background• Flushes• Compactions• Garbage collections• Gossip

Cassandra Processes

• Startup



• Background• Flushes• Compactions• Garbage collections• Gossip• Hinted Handoff

Cassandra Processes

• Startup



• Background• Flushes• Compactions• Garbage collections• Gossip• Hinted Handoff• Read Repair

Cassandra Processes

• Startup



System Resources


System Resources

• CPU


System Resources

• CPU• Single Core utilization


System Resources

• CPU• Single Core utilization• Multi Core utilization


System Resources


• Memory


System Resources


• Memory• Heap space


System Resources


• Memory• Heap space• Off-heap space


System Resources


• Memory• Heap space• Off-heap space• OS page cache


System Resources



• Garbage Collection


• Disk

System Resources





• Disk• Disk space

System Resources





• Disk• Disk space• I/O bandwidth

System Resources






• Network

System Resources






• Network

• OS Resources/Limits

System Resources






• Network

• OS Resources/Limits• File Handles

System Resources






• Network

• OS Resources/Limits• File Handles• Processes

System Resources






• Network

• OS Resources/Limits• File Handles• Processes• Mapped Memory

System Resources





Linux monitoring commands



Command What it tells you…



Command What it tells you…top CPU utilization and memory use per process



Command What it tells you…top CPU utilization and memory use per processtop -H CPU utilization per thread, memory use is still per process



Command What it tells you…top CPU utilization and memory use per processtop -H CPU utilization per thread, memory use is still per processdf Free disk space



Command What it tells you…top CPU utilization and memory use per processtop -H CPU utilization per thread, memory use is still per processdf Free disk spaceiostat -x I/O bandwidth utilization



Command What it tells you…top CPU utilization and memory use per processtop -H CPU utilization per thread, memory use is still per processdf Free disk spaceiostat -x I/O bandwidth utilizationfree -m Memory and cache usage



Command What it tells you…top CPU utilization and memory use per processtop -H CPU utilization per thread, memory use is still per processdf Free disk spaceiostat -x I/O bandwidth utilizationfree -m Memory and cache usagenetstat -an Network connections established



Command What it tells you…top CPU utilization and memory use per processtop -H CPU utilization per thread, memory use is still per processdf Free disk spaceiostat -x I/O bandwidth utilizationfree -m Memory and cache usagenetstat -an Network connections establishediftop Network bandwidth utilization



Command What it tells you…top CPU utilization and memory use per processtop -H CPU utilization per thread, memory use is still per processdf Free disk spaceiostat -x I/O bandwidth utilizationfree -m Memory and cache usagenetstat -an Network connections establishediftop Network bandwidth utilizationsar All (most) of the above, with history!


Java monitoring commands



Command What it tells you…jstack -l Status and stack trace of each thread



Command What it tells you…jstack -l Status and stack trace of each threadjmap -histo Types of objects on the heap (optionally only live objects)



Command What it tells you…jstack -l Status and stack trace of each threadjmap -histo Types of objects on the heap (optionally only live objects)jmap -heap Size and usage of each java heap generation



Command What it tells you…jstack -l Status and stack trace of each threadjmap -histo Types of objects on the heap (optionally only live objects)jmap -heap Size and usage of each java heap generationjstat -gccause Causes of gc activity



Command What it tells you…jstack -l Status and stack trace of each threadjmap -histo Types of objects on the heap (optionally only live objects)jmap -heap Size and usage of each java heap generationjstat -gccause Causes of gc activityjmap -dump Take a heap dump for further analysis



Command What it tells you…jstack -l Status and stack trace of each threadjmap -histo Types of objects on the heap (optionally only live objects)jmap -heap Size and usage of each java heap generationjstat -gccause Causes of gc activityjmap -dump Take a heap dump for further analysisMemoryAnalyzer Post-mortem heap-dump analysis


nodetool commands


nodetool commands

Command What it tells you…status / ring Overall cluster status


nodetool commands

Command What it tells you…status / ring Overall cluster statusinfo Status, memory usage, and caches for a single node


nodetool commands

Command What it tells you…status / ring Overall cluster statusinfo Status, memory usage, and caches for a single nodetpstats Statistics about each thread pool on a single node


nodetool commands

Command What it tells you…status / ring Overall cluster statusinfo Status, memory usage, and caches for a single nodetpstats Statistics about each thread pool on a single nodecfstats Summary statistics for all tables and keyspaces on a single node


nodetool commands

Command What it tells you…status / ring Overall cluster statusinfo Status, memory usage, and caches for a single nodetpstats Statistics about each thread pool on a single nodecfstats Summary statistics for all tables and keyspaces on a single nodecfhistograms Detailed statistics for a specific table on the local node


nodetool commands

Command What it tells you…status / ring Overall cluster statusinfo Status, memory usage, and caches for a single nodetpstats Statistics about each thread pool on a single nodecfstats Summary statistics for all tables and keyspaces on a single nodecfhistograms Detailed statistics for a specific table on the local nodeproxyhistograms Latency statistics for requests coordinated the local node


nodetool commands

Command What it tells you…status / ring Overall cluster statusinfo Status, memory usage, and caches for a single nodetpstats Statistics about each thread pool on a single nodecfstats Summary statistics for all tables and keyspaces on a single nodecfhistograms Detailed statistics for a specific table on the local nodeproxyhistograms Latency statistics for requests coordinated the local nodecompactionstats Compactions pending and in progress


nodetool commands

Command What it tells you…status / ring Overall cluster statusinfo Status, memory usage, and caches for a single nodetpstats Statistics about each thread pool on a single nodecfstats Summary statistics for all tables and keyspaces on a single nodecfhistograms Detailed statistics for a specific table on the local nodeproxyhistograms Latency statistics for requests coordinated the local nodecompactionstats Compactions pending and in progressnetstats Network activity: streams, read repair, and in-flight commands


nodetool statusNote: Ownership information does not include topology; for complete information, specify a keyspace

Datacenter: us-east =================== Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns Host ID Rack UN 54.173.171.164 22.78 MB 1 16.7% 67b3823f-6663-47d0-a04f-5914081e275c 1b DN 54.174.19.98 22.82 MB 1 16.7% 48d6f717-017b-4868-a525-b396d3f899aa 1b UN 54.174.245.247 22.68 MB 1 16.7% 6817e9ca-e79d-4fed-946e-7318bcfd5343 1b

Datacenter: us-west =================== Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns Host ID Rack UN 54.153.107.100 22.72 MB 1 16.7% 4abf0a7a-00ef-441a-9f70-046cd9fe1c0c 1a UN 54.153.108.157 22.79 MB 1 16.7% 303f08dd-2a19-4175-98e7-97920232855b 1a UN 54.153.39.203 22.67 MB 1 16.7% d1a57a91-7aef-4878-a056-88949920724c 1a


nodetool ringNote: Ownership information does not include topology; for complete information, specify a keyspace

Datacenter: us-east ========== Address Rack Status State Load Owns Token -3074457345618258603 54.174.19.98 1b Down Normal 22.82 MB 16.67% -9223372036854775808 54.174.245.247 1b Up Normal 22.68 MB 16.67% -6148914691236517206 54.173.171.164 1b Up Normal 22.78 MB 16.67% -3074457345618258603

Datacenter: us-west ========== Address Rack Status State Load Owns Token 6148914691236517205 54.153.39.203 1a Up Normal 22.67 MB 16.67% 0 54.153.107.100 1a Up Normal 22.72 MB 16.67% 3074457345618258602 54.153.108.157 1a Up Normal 22.79 MB 16.67% 6148914691236517205


nodetool infoToken : -6148914691236517206 ID : 6817e9ca-e79d-4fed-946e-7318bcfd5343 Gossip active : true Thrift active : true Native Transport active: true Load : 22.68 MB Generation No : 1426523950 Uptime (seconds) : 1557 Heap Memory (MB) : 270.85 / 1842.00 Off Heap Memory (MB) : 0.11 Data Center : us-east Rack : 1b Exceptions : 0 Key Cache : size 2368 (bytes), capacity 96468992 (bytes), 104 hits, 128 requests, 0.813 recent hit rate, 14400 save period in seconds Row Cache : size 0 (bytes), capacity 0 (bytes), 0 hits, 0 requests, NaN recent hit rate, 0 save period in seconds


nodetool tpstatsPool Name Active Pending Completed Blocked All time blocked ReadStage 0 2 20487 0 0 RequestResponseStage 0 0 2787111 0 0 MutationStage 32 8713 1735129 0 0 ReadRepairStage 0 0 2641 0 0 ReplicateOnWriteStage 0 0 0 0 0 GossipStage 0 0 34421 0 0 CacheCleanupExecutor 0 0 1 0 0 MigrationStage 0 0 1 0 0 MemoryMeter 1 1 71 0 0 FlushWriter 1 4 73 0 5 ValidationExecutor 0 0 42 0 0 InternalResponseStage 0 0 188 0 0 AntiEntropyStage 0 0 610 0 0 MemtablePostFlusher 1 4 358 0 0 MiscStage 0 0 43 0 0 PendingRangeCalculator 0 0 6 0 0 commitlog_archiver 0 0 0 0 0 CompactionExecutor 2 5 125 0 0 AntiEntropySessions 1 1 13 0 0 HintedHandoff 2 4 34 0 0

Message type Dropped RANGE_SLICE 0 READ_REPAIR 7 PAGED_RANGE 0 BINARY 0 READ 0 MUTATION 42993 _TRACE 0 REQUEST_RESPONSE 0 COUNTER_MUTATION 0


nodetool cfstatsKeyspace: Keyspace1 Read Count: 282781 Read Latency: 1.0155043372786716 ms. Write Count: 2700184 Write Latency: 0.07415620861393149 ms. Pending Tasks: 0

Table: Standard1 SSTable count: 5 Space used (live), bytes: 656644536 Space used (total), bytes: 656644536 Off heap memory used (total), bytes: 4221251 SSTable Compression Ratio: 0.0 Number of keys (estimate): 2182272 Memtable cell count: 215155 Memtable data size, bytes: 67222188 Memtable switch count: 72 Local read count: 282781 Local read latency: 1.016 ms Local write count: 2700943 Local write latency: 0.074 ms Pending tasks: 0 Bloom filter false positives: 379 Bloom filter false ratio: 0.00097 Bloom filter space used, bytes: 3897360 Bloom filter off heap memory used, bytes: 3897320 Index summary off heap memory used, bytes: 323931 Compression metadata off heap memory used, bytes: 0 Compacted partition minimum bytes: 259 Compacted partition maximum bytes: 310 Compacted partition mean bytes: 310 Average live cells per slice (last five minutes): 5.0 Average tombstones per slice (last five minutes): 0.0


nodetool cfhistogramsKeyspace1/Standard1 histograms

SSTables per Read 1 sstables: 14250 2 sstables: 42908 3 sstables: 97878 4 sstables: 126087 5 sstables: 745 6 sstables: 295 7 sstables: 122 8 sstables: 448 10 sstables: 37 12 sstables: 10 14 sstables: 1

Partition Size (bytes) 1131752 bytes: 1093 1358102 bytes: 479 1629722 bytes: 364 1955666 bytes: 343 2346799 bytes: 73 2816159 bytes: 134 3379391 bytes: 7 4055269 bytes: 0 4866323 bytes: 2

Cell Count per Partition 103 cells: 112 124 cells: 5130 149 cells: 2458 179 cells: 1704 215 cells: 1460 258 cells: 823 310 cells: 1327 372 cells: 1331 446 cells: 49295 535 cells: 9091 642 cells: 11892 770 cells: 3908 924 cells: 14532 1109 cells: 20898 1331 cells: 8491 1597 cells: 8140 1916 cells: 4010 2299 cells: 19476 2759 cells: 15112 3311 cells: 10812 3973 cells: 12763 4768 cells: 18597 5722 cells: 5765 6866 cells: 11340


nodetool cfhistogramsWrite Latency (microseconds) 4 us: 83 5 us: 9004 6 us: 73288 7 us: 211239 8 us: 321305 10 us: 607511 12 us: 438101 14 us: 440503 17 us: 459036 20 us: 211763 24 us: 97240 29 us: 43564 35 us: 29579 42 us: 22575 50 us: 10442 60 us: 4842 72 us: 3566 86 us: 4323 103 us: 2038 124 us: 6192 149 us: 6850 179 us: 2657 215 us: 4536 258 us: 1839

Read Latency (microseconds) 42 us: 29 50 us: 132 60 us: 1225 72 us: 4791 86 us: 13031 103 us: 18526 124 us: 21151 149 us: 21085 179 us: 16528 215 us: 19259 258 us: 39965 310 us: 55015 372 us: 29344 446 us: 11347 535 us: 5692 642 us: 5395 770 us: 3550 924 us: 2649 1109 us: 1379 1331 us: 1084 1597 us: 1266 1916 us: 1492 2299 us: 2032 2759 us: 1494


nodetool proxyhistogramsRead Latency (microseconds) 124 us: 18 149 us: 445 179 us: 5866 215 us: 14541 258 us: 19123 310 us: 20941 372 us: 32847 446 us: 37464 535 us: 26451 642 us: 25291 770 us: 37870 924 us: 35865 1109 us: 30432 1331 us: 22929 1597 us: 17592 1916 us: 12895 2299 us: 9760 2759 us: 7887 3311 us: 6024 3973 us: 4751 4768 us: 3691 5722 us: 3038 6866 us: 2453 8239 us: 1991

Write Latency (microseconds) 29 us: 4 35 us: 60 42 us: 331 50 us: 471 60 us: 1334 72 us: 2856 86 us: 4661 103 us: 65741 124 us: 37999 149 us: 42697 179 us: 45598 215 us: 30515 258 us: 23481 310 us: 21831 372 us: 22741 446 us: 26045 535 us: 33045 642 us: 43533 770 us: 51021 924 us: 57496 1109 us: 61548 1331 us: 60741 1597 us: 57538 1916 us: 52255


nodetool compactionstatspending tasks: 7 compaction type keyspace table completed total unit progress Compaction Keyspace1 Standard1 65967769 154639724 bytes 42.66% Compaction Keyspace1 Standard1 53880334 227493794 bytes 23.68% Active compaction remaining time : 0h00m15s

pending tasks: 1 compaction type keyspace table completed total unit progress Validation Keyspace1 Standard1 74684443 95178582 bytes 78.47% Active compaction remaining time : n/a


nodetool netstatsMode: NORMAL Repair 028763b0-cc1e-11e4-a20c-a1d01a3fbf30 /54.174.19.98 Receiving 6 files, 117949006 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-162-Data.db

851792/17950738 bytes(4%) received from /54.174.19.98 Sending 2 files, 47709526 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-157-Data.db

3786324/46561942 bytes(8%) sent to /54.174.19.98 Repair 020ed850-cc1e-11e4-a20c-a1d01a3fbf30 /54.174.245.247 Receiving 4 files, 93304584 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-161-Data.db


34195028/46561942 bytes(73%) sent to /54.174.245.247 Repair 018c88f0-cc1e-11e4-a20c-a1d01a3fbf30 /54.153.39.203 (using /172.31.10.65) Receiving 3 files, 49959102 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-160-Data.db

9371380/46561942 bytes(20%) received from /54.153.39.203 /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-159-Data.db


1147584/1147584 bytes(100%) sent to /54.153.39.203 /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-157-Data.db

46561942/46561942 bytes(100%) sent to /54.153.39.203

Read Repair Statistics: Attempted: 39576 Mismatch (Blocking): 0 Mismatch (Background): 746

Pool Name Active Pending Completed Commands n/a 58 2545817 Responses n/a 0 2833081










46561942/46561942 bytes(100%) sent to /54.153.39.203




system.log


system.log

Default Location /var/log/cassandra/system.log


system.log


Configuration File /etc/dse/cassandra/log4j-server.properties


system.log

Log SettingsLocationLog Settings

log4j.appender.R.File=/opt/dse-4.6.0/logs/system.log




system.log


log4j.appender.R.File=/opt/dse-4.6.0/logs/system.logLog Settings

Logging Level

Log Settings

log4j.rootLogger=INFO,stdout,R # INFO, WARN, ERROR, or DEBUG




system.log



Logging Level

Log Settings


Log Settings

Class Override

Log Settings

log4j.logger.org.apache.cassandra.package.Class=DEBUG




system.log



Logging Level

Log Settings


Log Settings

Class Override

Log Settings


Basic FormatLevelBasic Format

INFO

Basic Format

WARN

Basic Format

ERROR

Basic Format

DEBUG




system.log



Logging Level

Log Settings


Log Settings

Class Override

Log Settings



INFO

Basic Format

WARN

Basic Format

ERROR

Basic Format

DEBUG

Basic FormatThread Type & ID

Basic Format

[CompactionExecutor:155]

Basic Format

[GossipTasks:1]

Basic Format

[AntiEntropySessions:1]

Basic Format

[OptionalTasks:1]




system.log



Logging Level

Log Settings


Log Settings

Class Override

Log Settings



INFO

Basic Format

WARN

Basic Format

ERROR

Basic Format

DEBUG


Basic Format


Basic Format

[GossipTasks:1]

Basic Format


Basic Format

[OptionalTasks:1]

Basic FormatDate & Time

Basic Format

2015-02-13 02:18:40,986

Basic Format

2015-02-17 19:47:37,331

Basic Format

2015-02-17 20:32:11,959

Basic Format

2015-02-20 11:29:14,056




system.log



Logging Level

Log Settings


Log Settings

Class Override

Log Settings



INFO

Basic Format

WARN

Basic Format

ERROR

Basic Format

DEBUG


Basic Format


Basic Format

[GossipTasks:1]

Basic Format


Basic Format

[OptionalTasks:1]


Basic Format

2015-02-13 02:18:40,986

Basic Format

2015-02-17 19:47:37,331

Basic Format

2015-02-17 20:32:11,959

Basic Format

2015-02-20 11:29:14,056

Basic FormatSource File

Basic Format

CompactionTask.java

Basic Format

Gossiper.java

Basic Format

CassandraDaemon.java

Basic Format

ColumnFamilyStore.java




system.log



Logging Level

Log Settings


Log Settings

Class Override

Log Settings



INFO

Basic Format

WARN

Basic Format

ERROR

Basic Format

DEBUG


Basic Format


Basic Format

[GossipTasks:1]

Basic Format


Basic Format

[OptionalTasks:1]


Basic Format

2015-02-13 02:18:40,986

Basic Format

2015-02-17 19:47:37,331

Basic Format

2015-02-17 20:32:11,959

Basic Format

2015-02-20 11:29:14,056

Basic FormatSource File

Basic Format

CompactionTask.java

Basic Format

Gossiper.java

Basic Format

CassandraDaemon.java

Basic Format

ColumnFamilyStore.java

Basic FormatLine No.

Basic Format

(line 287)

Basic Format

(line 648)

Basic Format

(line 199)

Basic Format

(line 298)




Exceptionsjava.io.EOFException at java.io.DataInputStream.readFully(DataInputStream.java:197) at java.io.DataInputStream.readFully(DataInputStream.java:169) at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:395) at org.apache.cassandra.service.CacheService$KeyCacheSerializer.deserialize (CacheService.java:356) at org.apache.cassandra.cache.AutoSavingCache.loadSaved(AutoSavingCache.java:119) at org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:261) at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore (ColumnFamilyStore.java:415) at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore (ColumnFamilyStore.java:386) at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:309) at org.apache.cassandra.db.Keyspace.<init>(Keyspace.java:266) at org.apache.cassandra.db.Keyspace.open(Keyspace.java:110) at org.apache.cassandra.db.Keyspace.open(Keyspace.java:88) at org.apache.cassandra.db.SystemKeyspace.checkHealth(SystemKeyspace.java:536) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:246) at com.datastax.bdp.server.DseDaemon.setup(DseDaemon.java:376) at org.apache.cassandra.service.CassandraDaemon.activate (CassandraDaemon.java:480)


Exceptions – Stack Tracejava.io.EOFException at java.io.DataInputStream.readFully(DataInputStream.java:197) at java.io.DataInputStream.readFully(DataInputStream.java:169) at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:395) at org.apache.cassandra.service.CacheService$KeyCacheSerializer.deserialize (CacheService.java:356) at org.apache.cassandra.cache.AutoSavingCache.loadSaved(AutoSavingCache.java:119) at org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:261) at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore (ColumnFamilyStore.java:415) at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore (ColumnFamilyStore.java:386) at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:309) at org.apache.cassandra.db.Keyspace.<init>(Keyspace.java:266) at org.apache.cassandra.db.Keyspace.open(Keyspace.java:110) at org.apache.cassandra.db.Keyspace.open(Keyspace.java:88) at org.apache.cassandra.db.SystemKeyspace.checkHealth(SystemKeyspace.java:536) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:246) at com.datastax.bdp.server.DseDaemon.setup(DseDaemon.java:376) at org.apache.cassandra.service.CassandraDaemon.activate (CassandraDaemon.java:480)


Exceptions – Organizationjava.io.EOFException at java.io.DataInputStream.readFully(DataInputStream.java:197) at java.io.DataInputStream.readFully(DataInputStream.java:169) at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:395) at org.apache.cassandra.service.CacheService$KeyCacheSerializer.deserialize (CacheService.java:356) at org.apache.cassandra.cache.AutoSavingCache.loadSaved(AutoSavingCache.java:119) at org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:261) at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore (ColumnFamilyStore.java:415) at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore (ColumnFamilyStore.java:386) at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:309) at org.apache.cassandra.db.Keyspace.<init>(Keyspace.java:266) at org.apache.cassandra.db.Keyspace.open(Keyspace.java:110) at org.apache.cassandra.db.Keyspace.open(Keyspace.java:88) at org.apache.cassandra.db.SystemKeyspace.checkHealth(SystemKeyspace.java:536) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:246) at com.datastax.bdp.server.DseDaemon.setup(DseDaemon.java:376) at org.apache.cassandra.service.CassandraDaemon.activate (CassandraDaemon.java:480)


Exceptions – Subsystemjava.io.EOFException at java.io.DataInputStream.readFully(DataInputStream.java:197) at java.io.DataInputStream.readFully(DataInputStream.java:169) at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:395) at org.apache.cassandra.service.CacheService$KeyCacheSerializer.deserialize (CacheService.java:356) at org.apache.cassandra.cache.AutoSavingCache.loadSaved(AutoSavingCache.java:119) at org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:261) at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore (ColumnFamilyStore.java:415) at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore (ColumnFamilyStore.java:386) at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:309) at org.apache.cassandra.db.Keyspace.<init>(Keyspace.java:266) at org.apache.cassandra.db.Keyspace.open(Keyspace.java:110) at org.apache.cassandra.db.Keyspace.open(Keyspace.java:88) at org.apache.cassandra.db.SystemKeyspace.checkHealth(SystemKeyspace.java:536) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:246) at com.datastax.bdp.server.DseDaemon.setup(DseDaemon.java:376) at org.apache.cassandra.service.CassandraDaemon.activate (CassandraDaemon.java:480)


Exceptions – Classjava.io.EOFException at java.io.DataInputStream.readFully(DataInputStream.java:197) at java.io.DataInputStream.readFully(DataInputStream.java:169) at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:395) at org.apache.cassandra.service.CacheService$KeyCacheSerializer.deserialize (CacheService.java:356) at org.apache.cassandra.cache.AutoSavingCache.loadSaved(AutoSavingCache.java:119) at org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:261) at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore (ColumnFamilyStore.java:415) at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore (ColumnFamilyStore.java:386) at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:309) at org.apache.cassandra.db.Keyspace.<init>(Keyspace.java:266) at org.apache.cassandra.db.Keyspace.open(Keyspace.java:110) at org.apache.cassandra.db.Keyspace.open(Keyspace.java:88) at org.apache.cassandra.db.SystemKeyspace.checkHealth(SystemKeyspace.java:536) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:246) at com.datastax.bdp.server.DseDaemon.setup(DseDaemon.java:376) at org.apache.cassandra.service.CassandraDaemon.activate (CassandraDaemon.java:480)


Exceptions – Nested Classesjava.io.EOFException at java.io.DataInputStream.readFully(DataInputStream.java:197) at java.io.DataInputStream.readFully(DataInputStream.java:169) at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:395) at org.apache.cassandra.service.CacheService$KeyCacheSerializer.deserialize (CacheService.java:356) at org.apache.cassandra.cache.AutoSavingCache.loadSaved(AutoSavingCache.java:119) at org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:261) at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore (ColumnFamilyStore.java:415) at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore (ColumnFamilyStore.java:386) at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:309) at org.apache.cassandra.db.Keyspace.<init>(Keyspace.java:266) at org.apache.cassandra.db.Keyspace.open(Keyspace.java:110) at org.apache.cassandra.db.Keyspace.open(Keyspace.java:88) at org.apache.cassandra.db.SystemKeyspace.checkHealth(SystemKeyspace.java:536) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:246) at com.datastax.bdp.server.DseDaemon.setup(DseDaemon.java:376) at org.apache.cassandra.service.CassandraDaemon.activate (CassandraDaemon.java:480)


Exceptions – Methodsjava.io.EOFException at java.io.DataInputStream.readFully(DataInputStream.java:197) at java.io.DataInputStream.readFully(DataInputStream.java:169) at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:395) at org.apache.cassandra.service.CacheService$KeyCacheSerializer.deserialize (CacheService.java:356) at org.apache.cassandra.cache.AutoSavingCache.loadSaved(AutoSavingCache.java:119) at org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:261) at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore (ColumnFamilyStore.java:415) at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore (ColumnFamilyStore.java:386) at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:309) at org.apache.cassandra.db.Keyspace.<init>(Keyspace.java:266) at org.apache.cassandra.db.Keyspace.open(Keyspace.java:110) at org.apache.cassandra.db.Keyspace.open(Keyspace.java:88) at org.apache.cassandra.db.SystemKeyspace.checkHealth(SystemKeyspace.java:536) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:246) at com.datastax.bdp.server.DseDaemon.setup(DseDaemon.java:376) at org.apache.cassandra.service.CassandraDaemon.activate (CassandraDaemon.java:480)


Exceptions – Source Filejava.io.EOFException at java.io.DataInputStream.readFully(DataInputStream.java:197) at java.io.DataInputStream.readFully(DataInputStream.java:169) at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:395) at org.apache.cassandra.service.CacheService$KeyCacheSerializer.deserialize (CacheService.java:356) at org.apache.cassandra.cache.AutoSavingCache.loadSaved(AutoSavingCache.java:119) at org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:261) at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore (ColumnFamilyStore.java:415) at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore (ColumnFamilyStore.java:386) at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:309) at org.apache.cassandra.db.Keyspace.<init>(Keyspace.java:266) at org.apache.cassandra.db.Keyspace.open(Keyspace.java:110) at org.apache.cassandra.db.Keyspace.open(Keyspace.java:88) at org.apache.cassandra.db.SystemKeyspace.checkHealth(SystemKeyspace.java:536) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:246) at com.datastax.bdp.server.DseDaemon.setup(DseDaemon.java:376) at org.apache.cassandra.service.CassandraDaemon.activate (CassandraDaemon.java:480)


Exceptions – Line Numberjava.io.EOFException at java.io.DataInputStream.readFully(DataInputStream.java:197) at java.io.DataInputStream.readFully(DataInputStream.java:169) at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:395) at org.apache.cassandra.service.CacheService$KeyCacheSerializer.deserialize (CacheService.java:356) at org.apache.cassandra.cache.AutoSavingCache.loadSaved(AutoSavingCache.java:119) at org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:261) at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore (ColumnFamilyStore.java:415) at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore (ColumnFamilyStore.java:386) at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:309) at org.apache.cassandra.db.Keyspace.<init>(Keyspace.java:266) at org.apache.cassandra.db.Keyspace.open(Keyspace.java:110) at org.apache.cassandra.db.Keyspace.open(Keyspace.java:88) at org.apache.cassandra.db.SystemKeyspace.checkHealth(SystemKeyspace.java:536) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:246) at com.datastax.bdp.server.DseDaemon.setup(DseDaemon.java:376) at org.apache.cassandra.service.CassandraDaemon.activate (CassandraDaemon.java:480)


Exceptions – Nested Exceptionsorg.apache.thrift.transport.TTransportException: … at org.apache.thrift.transport.TIOStreamTransport.read at com.datastax.bdp.transport.server.TPreviewableTransport.readUntilEof at com.datastax.bdp.transport.server.TPreviewableTransport.preview at com.datastax.bdp.transport.server.TNegotiatingServerTransport.open at com.datastax.bdp.transport.server.TNegotiatingServerTransport$... at com.datastax.bdp.transport.server.TNegotiatingServerTransport$... at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run at java.util.concurrent.ThreadPoolExecutor.runWorker at java.util.concurrent.ThreadPoolExecutor$Worker.run at java.lang.Thread.run Caused by: java.net.SocketException: Connection reset at java.net.SocketInputStream.read at java.net.SocketInputStream.read at java.io.BufferedInputStream.fill at java.io.BufferedInputStream.read1 at java.io.BufferedInputStream.read at org.apache.thrift.transport.TIOStreamTransport.read ... 9 more


Exceptions – Error Messageorg.apache.thrift.transport.TTransportException: … at org.apache.thrift.transport.TIOStreamTransport.read at com.datastax.bdp.transport.server.TPreviewableTransport.readUntilEof at com.datastax.bdp.transport.server.TPreviewableTransport.preview at com.datastax.bdp.transport.server.TNegotiatingServerTransport.open at com.datastax.bdp.transport.server.TNegotiatingServerTransport$... at com.datastax.bdp.transport.server.TNegotiatingServerTransport$... at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run at java.util.concurrent.ThreadPoolExecutor.runWorker at java.util.concurrent.ThreadPoolExecutor$Worker.run at java.lang.Thread.run Caused by: java.net.SocketException: Connection reset at java.net.SocketInputStream.read at java.net.SocketInputStream.read at java.io.BufferedInputStream.fill at java.io.BufferedInputStream.read1 at java.io.BufferedInputStream.read at org.apache.thrift.transport.TIOStreamTransport.read ... 9 more


Exceptions - Organizationsorg.apache.thrift.transport.TTransportException: … at org.apache.thrift.transport.TIOStreamTransport.read at com.datastax.bdp.transport.server.TPreviewableTransport.readUntilEof at com.datastax.bdp.transport.server.TPreviewableTransport.preview at com.datastax.bdp.transport.server.TNegotiatingServerTransport.open at com.datastax.bdp.transport.server.TNegotiatingServerTransport$... at com.datastax.bdp.transport.server.TNegotiatingServerTransport$... at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run at java.util.concurrent.ThreadPoolExecutor.runWorker at java.util.concurrent.ThreadPoolExecutor$Worker.run at java.lang.Thread.run Caused by: java.net.SocketException: Connection reset at java.net.SocketInputStream.read at java.net.SocketInputStream.read at java.io.BufferedInputStream.fill at java.io.BufferedInputStream.read1 at java.io.BufferedInputStream.read at org.apache.thrift.transport.TIOStreamTransport.read ... 9 more


Exceptions - Subsystemsorg.apache.thrift.transport.TTransportException: … at org.apache.thrift.transport.TIOStreamTransport.read at com.datastax.bdp.transport.server.TPreviewableTransport.readUntilEof at com.datastax.bdp.transport.server.TPreviewableTransport.preview at com.datastax.bdp.transport.server.TNegotiatingServerTransport.open at com.datastax.bdp.transport.server.TNegotiatingServerTransport$... at com.datastax.bdp.transport.server.TNegotiatingServerTransport$... at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run at java.util.concurrent.ThreadPoolExecutor.runWorker at java.util.concurrent.ThreadPoolExecutor$Worker.run at java.lang.Thread.run Caused by: java.net.SocketException: Connection reset at java.net.SocketInputStream.read at java.net.SocketInputStream.read at java.io.BufferedInputStream.fill at java.io.BufferedInputStream.read1 at java.io.BufferedInputStream.read at org.apache.thrift.transport.TIOStreamTransport.read ... 9 more


Google Tips

106


Google TipsDO:

106


Google TipsDO:

• Use exception and several package+class+method names

106


Google TipsDO:

• Use exception and several package+class+method names• Use quotation marks around individual elements

106


Google TipsDO:


• Use “site:” to limit search to relevant web sites

106


Google TipsDO:


• Use “site:” to limit search to relevant web sites• Mailing list: site:www.mail-archive.com/[email protected]

106


Google TipsDO:


• Use “site:” to limit search to relevant web sites• Mailing list: site:www.mail-archive.com/[email protected]• JIRA: site:issues.apache.org/jira/browse/CASSANDRA

106


Google TipsDO:


• Use “site:” to limit search to relevant web sites• Mailing list: site:www.mail-archive.com/[email protected]• JIRA: site:issues.apache.org/jira/browse/CASSANDRA• StackOverflow: site:stackoverflow.com cassandra

106


Google TipsDO:



• Narrow or broaden as necessary

106


Google TipsDO:




DON’T:

106


Google TipsDO:




DON’T:

• Include source file + line number

106


Google TipsDO:




DON’T:

• Include source file + line number• Include specific numbers and string

106


Exceptions – Search Termsjava.io.EOFException at java.io.DataInputStream.readFully(DataInputStream.java:197) at java.io.DataInputStream.readFully(DataInputStream.java:169) at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:395) at org.apache.cassandra.service.CacheService$KeyCacheSerializer.deserialize (CacheService.java:356) at org.apache.cassandra.cache.AutoSavingCache.loadSaved(AutoSavingCache.java:119) at org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:261) at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore (ColumnFamilyStore.java:415) at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore (ColumnFamilyStore.java:386) at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:309) at org.apache.cassandra.db.Keyspace.<init>(Keyspace.java:266) at org.apache.cassandra.db.Keyspace.open(Keyspace.java:110) at org.apache.cassandra.db.Keyspace.open(Keyspace.java:88) at org.apache.cassandra.db.SystemKeyspace.checkHealth(SystemKeyspace.java:536) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:246) at com.datastax.bdp.server.DseDaemon.setup(DseDaemon.java:376) at org.apache.cassandra.service.CassandraDaemon.activate (CassandraDaemon.java:480)


Exceptions – Search Termsorg.apache.thrift.transport.TTransportException: … at org.apache.thrift.transport.TIOStreamTransport.read at com.datastax.bdp.transport.server.TPreviewableTransport.readUntilEof at com.datastax.bdp.transport.server.TPreviewableTransport.preview at com.datastax.bdp.transport.server.TNegotiatingServerTransport.open at com.datastax.bdp.transport.server.TNegotiatingServerTransport$... at com.datastax.bdp.transport.server.TNegotiatingServerTransport$... at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run at java.util.concurrent.ThreadPoolExecutor.runWorker at java.util.concurrent.ThreadPoolExecutor$Worker.run at java.lang.Thread.run Caused by: java.net.SocketException: Connection reset at java.net.SocketInputStream.read at java.net.SocketInputStream.read at java.io.BufferedInputStream.fill at java.io.BufferedInputStream.read1 at java.io.BufferedInputStream.read at org.apache.thrift.transport.TIOStreamTransport.read ... 9 more


Startup


Startup

What to look for… What it tells you…


Startup

What to look for… What it tells you…Logging initialized Node just restarted


Startup

What to look for… What it tells you…Logging initialized Node just restarted• DSE version: … • Cassandra version: …

Versions of major components


Startup



Loading settings from file: … Settings file locations


Startup



Loading settings from file: … Settings file locations• Data files directories: […] • Commit log directory: …

Data file and commit log location


Startup





JVM vendor/version: … JVM vendor and version


Startup





JVM vendor/version: … JVM vendor and version• Heap size: … • Par Eden Space Heap memory: … • Par Survivor Space Heap memory: … • CMS Old Gen Heap memory: … • CMS Perm Gen Non-heap memory:…

Heap settings for each generation


Startup







Classpath: … Classpath (jar files and directories)


Startup







Classpath: … Classpath (jar files and directories)JNA mlockall successful JNA is installed (make sure it is!)


Startup







Classpath: … Classpath (jar files and directories)JNA mlockall successful JNA is installed (make sure it is!)• Starting listening for CQL clients on ... • Listening for thrift clients…

Node is serving requests


Flushes

INFO [OptionalTasks:1] 2015-02-17 19:18:54,835 ColumnFamilyStore.java (line 794) Enqueuing flush of Memtable-pdps@127960523(91418/471789 serialized/live bytes, 9370 ops)

INFO [FlushWriter:30] 2015-02-17 19:18:54,836 Memtable.java (line 355) Writing Memtable-pdps@127960523(91418/471789 serialized/live bytes, 9370 ops)

INFO [FlushWriter:30] 2015-02-17 19:18:54,852 Memtable.java (line 395) Completed flushing /data/OpsCenter/pdps/OpsCenter-pdps-jb-3101-Data.db (26109 bytes) for commitlog position ReplayPosition(segmentId=1424226832202, position=38722)


Flushes





Compactions INFO [CompactionExecutor:54106] 2015-01-24 12:57:04,632 CompactionTask.java (line 105)

Compacting [SSTableReader(path='/data/system/hints/system-hints-ic-2498-Data.db'), SSTableReader(path='/data/system/hints/system-hints-ic-2493-Data.db'), SSTableReader(path='/data/system/hints/system-hints-ic-2499-Data.db'), SSTableReader(path='/data/system/hints/system-hints-ic-2497-Data.db'), SSTableReader(path='/data/system/hints/system-hints-ic-2495-Data.db'), SSTableReader(path='/data/system/hints/system-hints-ic-2500-Data.db'), SSTableReader(path='/data/system/hints/system-hints-ic-2494-Data.db'), SSTableReader(path=‘/data/system/hints/system-hints-ic-2496-Data.db’)]

INFO [CompactionExecutor:54106] 2015-01-24 12:57:04,635 CompactionController.java (line 156) Compacting large row system/hints:fa937b87-de5a-45e0-9b9d-379cc04b81b2 (487226212 bytes) incrementally

INFO [CompactionExecutor:54106] 2015-01-24 12:57:21,851 CompactionTask.java (line 262) Compacted 8 sstables to [/data/system/hints/system-hints-ic-2501,]. 359,979,223 bytes to 358,979,736 (~99% of original) in 17,219ms = 19.882093MB/s. 17 total rows, 6 unique. Row merge counts were {1:2, 2:2, 3:1, 4:0, 5:0, 6:0, 7:0, 8:1, }


Garbage Collections

INFO [ScheduledTasks:1] 2015-02-17 19:33:41,493 GCInspector.java (line 116) GC for ConcurrentMarkSweep: 91197 ms for 3 collections, 15035796304 used; max is 16965173248

INFO [ScheduledTasks:1] 2015-02-17 19:31:03,013 GCInspector.java (line 116) GC for ParNew: 908 ms for 2 collections, 4428318872 used; max is 16965173248

WARN [ScheduledTasks:1] 2013-02-20 11:49:12,484 GCInspector.java (line 142) Heap is 0.8275743567425272 full.

You may need to reduce memtable and/or cache sizes. Cassandra will now flush up to the two largest memtables to free up memory. Adjust flush_largest_memtables_at threshold in cassandra.yaml if you don't want Cassandra to do this automatically


Garbage Collections






Flapping INFO [GossipTasks:1] 2015-02-17 14:38:05,129 Gossiper.java (line 901)

InetAddress /10.98.12.216 is now DOWN

INFO [GossipTasks:1] 2015-02-17 14:38:39,768 Gossiper.java (line 901) InetAddress /10.100.12.130 is now DOWN

INFO [RequestResponseStage:18] 2015-02-17 14:39:14,169 Gossiper.java (line 887) InetAddress /10.98.12.216 is now UP








Hinted HandoffINFO [HintedHandoff:1] 2015-03-16 20:18:28,627 HintedHandOffManager.java (line 349)

Started hinted handoff for host: d1a57a91-7aef-4878-a056-88949920724c with IP: /54.153.39.203

INFO [HintedHandoff:1] 2015-03-16 20:18:37,292 HintedHandOffManager.java (line 381) Finished hinted handoff of 2492 rows to endpoint /54.153.39.203


RepairsINFO [AntiEntropySessions:1] 2015-03-16 17:24:44,097 RepairSession.java (line 246)

[repair #562f46d0-cc01-11e4-a84d-098a653a7013] new session: will sync /54.174.19.98, /54.153.108.157, /54.153.39.203, /54.174.245.247 on range (3074457345618258602,6148914691236517205] for Keyspace1.[Counter3, Standard1, SuperCounter1, Super1, Counter1]

INFO [RepairJobTask:2] 2015-03-16 17:24:48,102 RepairJob.java (line 161) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] requesting merkle trees for Standard1 (to [/54.153.108.157, /54.153.39.203, /54.174.245.247, /54.174.19.98])

INFO [AntiEntropyStage:1] 2015-03-16 17:24:52,749 RepairSession.java (line 166) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] Received merkle tree for Standard1 from /54.174.19.98

INFO [RepairJobTask:4] 2015-03-16 17:24:52,757 Differencer.java (line 67) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] Endpoints /54.153.39.203 and /54.174.245.247 are consistent for Standard1

INFO [RepairJobTask:6] 2015-03-16 17:24:55,438 Differencer.java (line 74) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] Endpoints /54.153.108.157 and /54.174.19.98 have 6222 range(s) out of sync for Standard1

INFO [RepairJobTask:6] 2015-03-16 17:24:55,441 StreamingRepairTask.java (line 64) [streaming task #562f46d0-cc01-11e4-a84d-098a653a7013] Performing streaming repair of 6222 ranges with /54.153.108.157

INFO [StreamReceiveTask:1] 2015-03-16 17:25:01,524 StreamingRepairTask.java (line 92) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] streaming task succeed, returning response to /54.174.19.98

INFO [AntiEntropyStage:1] 2015-03-16 17:25:01,805 RepairSession.java (line 223) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] Standard1 is fully synced

INFO [AntiEntropySessions:1] 2015-03-16 17:25:01,806 RepairSession.java (line 284) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] session completed successfully


Bringing it full circle…

166

1 Ask what changed




5 Share relevant information

6 Ask for help

Thanks!Q&A

Cassandra Day Atlanta 2015: Troubleshooting with Apache Cassandra

Technology

Transcript of Cassandra Day Atlanta 2015: Troubleshooting with Apache Cassandra