Amit Hurvitz, ISV Engineering, Oracle

DTrace, for Solaris (zones inside) and Java

Program Agenda

Introductory Demo What's Dtrace? Enabling Dtrace from Inside the Zone Java Statically Defined Tracing TCP Client and Server Flow Tracing - Demo Future Thoughts

Introductory Demo What's Dtrace? Enabling Dtrace from Inside the Zone Java Statically Defined Tracing TCP Client and Server Flow Tracing - Demo Future Thoughts

Introductory DemoIntroductory Demo

What's DTrace? - cont.

Zero performance impact when not in use Completely safe; no way to cause panics, crashes, data corruption or

pathological performance degradation Powerful data management primitives eliminate need for most post-

processing

DTrace – D Language

probe descriptionprobe description

/predicate//predicate/ {{ actionsactions }}

Probes Probes Which events we are interested in monitoring Predicates (optional) Predicates (optional) When do we want to monitor the events Actions (optional) Actions (optional) What do we want to do when the above happens

One liner # dtrace -n 'probeprobe/predicate//predicate/{actions}{actions}'

DTrace – Probes

Programmable sensors (points of instrumentation) made available by providers placed all over the Solaris system

provider:module:function:name tcp:ip:tcp_send:entry Syscall:::

Providers: syscall,io,pid,profile, hotspot, tcp, udp, ip, iscsi,...

Modules: nfs, zfs, cpc, …

Names: entry,return

Listing Probes # dtrace -l [-P provider | -m module | -f function name | -n name]

DTrace – Predicates and Actions

Predicates /cpu == 0/ /execname == “date”/ /ppid != 0 && arg0 != 0/

Actions Commands separated by “;” trace(execname) printf(“%s %s %s”, execname, probefunc, copyinstr(arg0));

Predefined Variables execname, probefunc, pid, ppid, cpu, timestamp,

arg0, arg1, ...

DTrace – Aggregations

Used to aggregate data and look for trends Has the general form:

@name[keys] = aggfunc(args)@name[keys] = aggfunc(args)

Aggregating functions count(), sum(), avg(), min(), max(), quantize(), lquantize()

Examples: % dtrace -n 'syscall::read*:entry{@[execname]=count();}' % dtrace -n 'syscall::read*:entry{@[execname,arg0]=count();}'

An Example – Off-CPU Tracing#!/usr/sbin/dtrace -sBEGIN{ start_timestamp = timestamp;}

sched:::off-cpu/pid == $1/{ self->ts = timestamp;}

sched:::on-cpu/self->ts/{ @[stack(), jstack(), "ns"] = sum(timestamp - self->ts); self->ts = 0;}

END{ printf("elapsed time: %d\n", timestamp - start_timestamp);}

Enabling DTrace in a Zone

Java Statically Defined Tracing (JSDT)

Insert your own DTrace probes in desired locations inside your methods

– Use them in conjunction with any other probes

Make points of interest in your application easily monitored

#!/usr/sbin/dtrace -s

MyProvider:::start{ self->start_time = timestamp;}

syscall:::entry/self->start_time/{ @[probefunc] = quantize();}

JSDT – define a Provider

Provider Interfacepublic interface MyProvider extends com.sun.tracing.Provider {

void methodEntry();void methodReturn();void start();void dataAdded(int x, int y);void myProbe(int intData, String stringData);

}

JSDT – insert probes to your Java code

Provider Interface

Call probes actions

Create the provider

import com.sun.tracing.*;

MyProvider provider;

public static void main(String args) { ProviderFactory factory = ProviderFactory.getDefaultFactory(); provider = factory.createProvider(MyProvider.class);}

public void method() { provider.methodEntry(); ... provider.myProbe(i, str); ... provider.methodReturn();}

Flow Trace- DemoFlow Trace- Demo

Next Thoughts

A special ‘Java-DTrace’ utility to do implicit instrumentation Probes look like native DTrace PID provider:

JDDT$target:class-name:method-name:entry JDDT$target:class-name:method-name:return

# jdtrace java-dtrace-script.d -p <process-id> jdtrace will take care of all required dynamic instrumentation

Clean instrumented code on script end Any suggestions?

Thanks!

Amit.Hurvitz@oracle.com