Solaris Kernel Debugging V1.0
-
Upload
jarod-wang -
Category
Technology
-
view
8.726 -
download
10
Transcript of Solaris Kernel Debugging V1.0
1
Oliver YangSoftware EngineerSun Mircosystem, Inc.
Solaris Kernel DebuggingMdb and DTrace
1
2
Agenda• Kernel Debug Overview• Modular Debugger - Mdb• Dynamic Tracing - DTrace • References
3
Skill Sets of Kernel Debugging• Key elements for kernel debugging
> Kernel source code– http://src.opensolaris.org/source/xref/onnv/o
nnv-gate/usr/src/> Kernel debugging tools> System Architecture
– x32/x64/SPARC> Programing skills
– C/Assembly/D/Shell/Awk/Sed/Perl
4
Kernel Debugging Tools• Debug In Code
> cmn_err(9F) - Kernel version of printf(3C)> ASSERT - Only effective in debug kernel
• In-situ kernel debuggers> Kmdb, SPARC OBP
• Run time tracing> DTrace, Lockstat, Kmem allocator...etc.
• Post-mortem debuggers> Mdb, ACT, SCAT
5
Difficulties of Kernel Debugging...• The problems you may encounter
> System Panic> System hang> Memory leaks & corruption > Performance issues> Any other functionality issues
• Some of hot bugs found on customer sites...> Can not debug on the non-production kernel> Can not debug on mission-critical machines> May not be deterministically reproduced> May only have the crash dumps
6
Agenda• Kernel Debug Overview• Modular Debugger - Mdb• Dynamic Tracing - DTrace • References
7
Mdb - The Modular Debugger • Mdb targets
> User processes> User process core files> Live kernel read only by /dev/kmem&/dev/ksyms> Live Kernel with execution control by kmdb> System crash dumps> User process images inside system crash dumps> ELF object files> Raw data files
8
Live Kernel Debug – Read Only• How to run it?
> mdb -k• What you can do?
> Inspect kernel data structures and kernel pages> /dev/kmem
Access kernel virtual address space excluding memorythat is associated with an I/O device
> /dev/ksymsAccess kernel symbols as kernel ELF definitions
9
Live Kernel Debug - Execution Control • How to run it?
> mdb -K> Boot system with kmdb loaded
– x86 “-k”option in grub menu– SPARC “-k or kmdb” option in OBP
• What you can do?> Instruction-level control of kernel threads
executing on each CPU> Setting breakpoint and single-step the kernel
and inspect data structures in real time
10
Live Kernel Debug - Execution Control • dcmds
> [addr]:b> [addr]:d> ::events or $b> :z> :c> :e> :s> [syscall]::sysbp> addr [,len]::wp
11
Post-mortem Debug - Crash Dumps• How to use it
> mdb unix.<n> vmcore.<n>• What you can do?
> Access kernel memory pages and user process images inside a system crash dump
> Inspect kernel/user process data structures and kernel/user process pages
12
Post-mortem Debug - Crash Dumps• You can get a crash dump by...
> A real panic> Reboot with -d> Enter kmdb, run $<systemdump> Deadman timer
– Setting snooping to 1 in /etc/system, reboot– Setting deadman_enabled to 1 via mdb -kw
• savecore(1M) & dumpadm(1M)Dump content: kernel pagesDump device: /dev/dsk/c0d0s1 (swap)Savecore directory: /var/crash/<hostname>Savecore enabled: yes
13
Modular Debugger Basic• General Dcmds
> ::help> ::dcmds> ::formats> ::dmods -l [module...]> ::log -e file> ::quit or $q
14
Modular Debugger Basic• Inspect memory and data structures
> addr[,b]::dump [-g sz] [-e]> addr::dis> addr::print type field> ::sizeof type> ::offsetof type field> ::enum enumname> addr::array [type count] [var]> addr::list type field [var]
15
Crash Dumps Analysis - Panic• Panic procedures
> Panic messages– Panic thread– Trap number– Pointer of trap frame– CPU registers– back trace
> Dump memory to dump device> Dump CPU registers to dump device> Reboot> Savecore (from dump device to file system)
16
Crash Dumps Analysis - Panic• dcmds
> ::satus> ::showrev> ::prtconf> ::modinfo> ::msgbuf> [addr]$c/::stack/::stackregs> [addr]::dis> ::regs> [rp]::print struct regs
• Know the ABIs of x32/x64/SPARC
17
Crash Dumps Analysis – Hang• What conditions cause hangs?
> Deadlock> Resources exhaustion> Hardware problems
• Debugging system hangs> Live debugging with kmdb> Forcing a crash dump and analysis with mdb
18
Crash Dumps Analysis – Hang• Dispatcher and kernel threads
> [id]::cpuinfo> ::cycinfo> [addr]::threadlist> [addr]::thread> [addr]::findstack> [addr]::mutex> [addr]::rwlock> [addr]::wchaninfo> [addr]::whatthread or ::kgrep
19
Crash Dumps Analysis – Hang• Kernel Memory
> ::memstat> ::findleaks> ::kmastat/::kmem_cache/::walk <cache name>> ::kmausers> ::vmem/::walk vmem_seg/::vmem_seg> [addr]::whatis> [addr]::bufctl> [addr]::allocdby/[addr]::freedby
• Some of dcmds need kmem allocator tracing> Setting kmem_flags = 0xf in /etc/system, reboot
20
Agenda• Kernel Debug Overview• Modular Debugger - Mdb• Dynamic Tracing - DTrace • References
21
Dynamic Tracing Framework• DTrace framework includes...
> Consumer programs running in user land– dtrace(1M)/intrstat(1M)/lockstat(1M)...
> Kernel modules that provide probes to gather tracing data– dtrace(7D) and providers: syscall/fbt/sdt/vminfo...
> A library interface that consumer programs use to access the DTrace facility by dtrace driver
22
DTrace Big Picture
23
Provider• How provider works
> Provider represents a methodology for instrumenting the system
> Provider covers a certain aspect of the system> Provider makes probes available to the DTrace
framework> DTrace informs providers when a probe is to be
enabled provider transfers• Using providers with different ways
> Watch code path– fbt/sdt/syscall/pid/fsinfo/io/vminfo/proc/sched, etc.
> Get statistical data– mib/lockstat/profile/sysinfo, etc.
24
ProvidersProvider Description
lock contention statistics or understand locking behaviorsprofile a time-based interrupt firing every fixed, specified interval
entry to and return from most functions in the Solaris kernelentry to and return from every system call in the systemlocations at that a programmer has formally designatedcorrespond to kernel statistics classified by the name sys
process creation and termination,sending and handling signalsrelated to CPU schedulingrelated to disk input and outputrelated to counters in MIB - management information basesentry and return of any function in a user process
lockstat
fbtsyscallsdtsysinfovminfo correspond to the vm kernel statisticsprocschediomibpid
25
Running DTrace• D scripts
> Run *.d scripts#!/usr/sbin/dtrace -sprobe/predicate/{ actions}
• Command line> Run dtrace command, see dtrace(1M)
dtrace -n probe'/predicate/{actions}'
26
Probe• provider:module:function:name
> Provider– The instrumentation method to be used.For example,
the syscall provider is used to monitor system calls while the io provider is used to monitor the disk io.
> Module– The kernel module you want to observe
> Function– The kernel function you want to observe
> Name– Represents the location in the function. For example,
use entry for name to instrument when you enter the function.
27
Probe
Probe Description Explanationfbt::bge_intr:entryentry into bge_intr functionsfbt::bge_*:entry entry into any kernel functions that starts with bge_fbt:bge::entry entry into any bge driver functionsfbt:::entry entry into any kernel functionsfbt::: all probes published by the fbt provider
• A probe...> Is defined as 4-attribute tuple> could be listed by dtrace -l [-f|-l|-m|-n|-P]> supports wildcards match
28
Predicate
Predicate ExplanationCPU == 0 true if the probe executes on cpu0
true if the process is not the scheduler
Pid == 1029 true if the pid of the process that caused the probe to fire is 1029
execname != “sched”
ppid !=0 && arg0 == 0 true if the parent process id is not 0 and first argument is 0
• A predicate...> could be any D expression, result is boolean> is true means the actions could be executed
29
Action• An Action...
> is executed when a probe fires> has two categories
– Data Recording Action/Destructive ActionAction Explanationtrace() trace the D expression resultsprintf() print something using C-style printf()printa() print the aggregationsustack() print the user stack tracestack() print the kernel stack tracetracemem() copy data from an address in memory to a bufferbreakpoint() a kernel breakpoint, causes system drop into kmdbpanic() cause a kernel panicchill() spin for the specified number of nanoseconds
30
Aggregation
Functions Explanationcount() times that the count function is calledsum() total value of the specified expressionsavg() arithmetic average of the specified expressionsmin() smallest value among the specified expressionsmax() largest value among the specified expressions
quantize()
lquantize()A linear frequency distribution of the values of the specified expressions that is sized by the specified rangeA power of 2 frequency distribution of the values of the specified expressions.
• Aggregation syntax> @name[ keys ] = aggfunc( args );
31
Variables> Scalar Variables
– Represent individual fixed-size data objects> Associative Arrays
– name [ key ] = expression ;> Thread-Local Variables
– self->[variable name]> Clause-Local Variables
– this->[variable name]> Built-in Variables
– pre-defined scalar global variables> External Variables
– the ”`” is a scoping operator for accessing variables that are defined in the OS, eg: `kmem_flags
32
Built-in VariablesType and Name Explanationint64_t arg0...arg9 The first 10 input argumentscpuinfo_t *curcpu The CPU information for the current CPU.processorid_t cpu The CPU identifier for the current CPU.kthread_t *curthread kthread_t address for current kernel threadpid_t pid The process ID of the current processpid_t ppid parent process ID of the current processuint_t ipl IPL on the current CPU at probe firing timeint errno Error value returned by the last system callstring execname name passed to exec(2) to execute the process
uint64_t timestamp
uint64_t vtimestamp
A nanosecond timestamp counter, it increments from an arbitrary point in the past and should only be used for relative computationsA nanosecond timestamp counter that is the time of the current thread has been running on a CPU, minus the time spent in predicates and actions
33
Agenda• Kernel Debug Overview• Modular Debugger - Mdb• Dynamic Tracing - DTrace • References
34
Documentations & Links - Mdb• Solaris Internals Second Edition
> www.solarisinternals.com• Solaris Modular Debugger Guide
> docs.sun.com/app/docs/doc/817-2543• OpenSolaris mdb community
> opensolaris.org/os/community/mdb• Crash Dump analysis
> opensolaris.org/os/community/documentation/files/book.pdf
35
Documentations & Links - DTrace• Solaris Internals Second Edition
> www.solarisinternals.com/wiki/index.php/DTrace_Topics
• Solaris Dynamic Tracing Guide> docs.sun.com/app/docs/doc/819-3620
• OpenSolaris DTrace community> opensolaris.org/os/community/dtrace
• DTrace Tools> www.brendangregg.com/dtrace.html