1

HBase TracingElliott Clark

2

Who Am I

• Elliott Clark• eclark@apache.org

• HBase PMC• HBase Committer• HTrace Mantainer• Cloudera Engineer

• CDH Distribution

3

What’s Going on During Failovers

4

Motivation

• Want to improve MTTR• First need to know what’s going on• Google’s Dapper

5

Options

• Log all the things• We already do that.• Our logs are way too verbose as it is.• Hard to combine multiple logs

• HRegionServer• HMaster• Client

• Guess• Zipkin• Not Invented Here

6

Spoiler:Guessing Didn’t Win

7

About Zipkin

• Created by Twitter• Thanks

• Modeled after Dapper• Scala• Finagle

• Functional• Netty• Non-blocking

• Thrift• Cool UI• Pluggable Storage Backend

8

Dapper’s Terminology

• Span• Segment of a remote call• Contains Annotations• Has a parent trace• Can have multiple children spans• Can have multiple annotations

• Annotations• Trace

• Grouping of spans

9

Can’t use Cassandra and look at myself in the mirror

10

Hooking up Zipkin, HTrace, and HBase

Implementation

11

Zipkin HBase Storage

• Replaces need for Cassandra• Functional• Created a non-blocking client shim• Fixed Key Length

• ID’s are long• Mapping table

• ID -> Name• Name -> ID• Inspiration from OpenTSDB

• Open source on Github

12

HTrace

• Way to instrument Java Code• Used In HBase trunk• Coming to an install near you in 0.96.0• Pluggable where it emits Spans

• To a file• To sys.out

13

HTrace and Zipkin

• Modularize HTrace• Create htrace-zipkin• Create ZipkinSpanReceiver

• Receives spans from HTrace instrumented code• Converts to Thrift objects• Sends to Zipkin

• Background Flush• Batching

14

Let’s hope this works.

Demo

15