HTrace: Tracing in HBase and HDFS (HBase Meetup)

Copyright © 2014 NTT DATA Corporation

10/15/2014Masatake Iwasaki

HTrace: Tracing in HBase and HDFS

Presentation slide for HBase Meetup on the Night before Strata/HW @ Google in Chelsea

2 Copyright © 2014 NTT DATA Corporation

What is HTrace?

https://github.com/cloudera/htrace

Tracing tool for parallel distributed systemlike Google's Dapper �

Effective for finding bottleneckEffective for code analysisLow Overhead


Tracing spans

Span represents traced processing unit and timeSpans have parent-‐‑‒child relationshipPassing tracing info along with RPC

Time

Span A trace id: 12345parent: rootnode 1

Span B trace id: 12345parent: Anode 2

Span C Span Dtrace id: 12345parent: B

trace id: 12345parent: Bnode 3

RPC

RPC RPC


Starting root span

FsShell shell = new FsShell();! conf.setQuietMode(false);! shell.setConf(conf);! int res = 0;! SpanReceiverHost.getInstance(new HdfsConfiguration());! TraceScope ts = null;! try {! ts = Trace.startSpan("FsShell", Sampler.ALWAYS);! res = ToolRunner.run(shell, argv);! } finally {! shell.close();! if (ts != null) ts.close();! }

org.htrace.Trace#startSpan create spannew trace ID is assigned when root span starts


Starting passive span

Starting child span only when there is parentFor server-‐‑‒side custom tracing span

if (Trace.isTracing()) {! traceScope = Trace.startSpan(method.getName());! }


Passing tracing info along with RPC

message RequestHeader {! optional uint32 call_id = 1;! optional RPCTInfo trace_info = 2;! optional string method_name = 3;! ...!}!!message RPCTInfo {! optional int64 trace_id = 1;! optional int64 parent_id = 2;!}!

RPC header has optional field for tracingRPC with tracing info start span on server-‐‑‒side


Span receivers

Each process loads receiver moduleReceivers receive spans from in-‐‑‒process queue Receivers send spans to collector asynchronously

Server

Client

Server

Collector/Sink

SpanReceiver

SpanReceiver

SpanReceiver

RPCRPC

Tracing Spans


Passing tracing info between threads

Ongoing tracing span is stored in ThreadLocalYou need to pass tracing info between threadsif (header.hasTraceInfo()) {! // If the incoming RPC included tracing info, always continue the trace !

TraceInfo parentSpan = new TraceInfo(header.getTraceInfo().getTraceId(),! header.getTraceInfo().getParentId());! traceSpan = Trace.startSpan(rpcRequest.toString(), parentSpan).detach();!}!Call call = new Call(header.getCallId(), header.getRetryCount(),! rpcRequest, this, ProtoUtil.convert(header.getRpcKind()),! header.getClientId().toByteArray(), traceSpan);!!...!!if (call.traceSpan != null) {! traceScope = Trace.continueSpan(call.traceSpan);!}!


JIRAs

Already available in HBase (HBASE-‐‑‒6449) HDFS (HDFS-‐‑‒5274)

Working on YARN (YARN-‐‑‒1418)


Configurations

Setting receiver class impl turns on tracingEach receiver impl has its own additional confs! <property>! <name>hbase.trace.spanreceiver.classes</name>! <value>org.htrace.impl.HBaseSpanReceiver</value>! </property> ! <property>! <name>hbase.htrace.hbase.collector-quorum</name>! <value>127.0.0.1</value>! </property>


Tracing from HBase shell

"trace" command start/stop tracing span# You need configuration on client node!

$ hbase shell! > trace 'start'! > create 'test', 'f'! > trace 'stop'


Example: Creating table in HBase


Example: Tracing of putting 200MB of a file to HDFS


Example: Tracing of getting 200MB of a file from HDFS


Example: Zipkin UI


Todo

Adding granular tracing spansSampling and filtering spansDynamic reconfiguration (HDFS-‐‑‒6956)sink and viewer with less dependency

HTrace: Tracing in HBase and HDFS (HBase Meetup)

Technology

Transcript of HTrace: Tracing in HBase and HDFS (HBase Meetup)