1

Hands-on Workshop on How ToDebug Codes at the Institute

H. Birali Runesha, Shuxia Zhang and Ben Lynch

(612) 626 0802 (help)help@msi.umn.edu

October 13, 2005

Outline

• Debuggers at the Institute

• Totalview– Starting TotalView

– The TotalView GUI and basic windows

– Data management and visualization

– Breakpoints and Action points

– Parallel debugging: MPI, OpenMP, etc.

2

Debuggers at the Institute• Write statements• dbx• pdbx• Xpdbx• xldb• Pedb• GNU gdb

• CVD• Intel IDB• DDD• Totalview• Other tools: Compiler

options, profilers,hardware counters

• dbx is a command line debugger for serial andparallel Fortran and C programs respectively

• Most useful Commands:stop at linenumber # set a breakpoint at the given linenumberstop in myroutine # set a breakpoint at the given routinerun < infile # run your program, reading from the file infilenext # to execute the next step of your codeprint var # see the value of a given variablewhatis var # determine the data type of a given variablecont # continue execution from where it stopped

• Compile your program with –g option• Start dbx with or without the executable as

an argument.• More info: man dbx, on dbx prompt use help• Available: unix systems: IBM, SGI and SUN

dbx

3

• Command-line parallel debugger built on dbx• IBM debugger for parallel programs on AIX systems and

accepts all flags supported by POE• Supports C, C++, Fortran 77, Fortran 90 programs• Compile the code with -g flag• Setup POE environment variables, e.g.:

setenv MP_BUFFER_MEM 28MB setenv MP_EAGER_LIMIT 4090kb• Run pdbx:

pdbx a.out [program_options] [poe options] or

pdbx -a poe process id [limited poe options]• Available: IBM Power3 and Power4

• More information: man pdbx

pdbx

• Parallel debugger with graphical interface• Load executable after starting xpdbx. Use

the File menu• Use X resources to customize the xpdbx

window– The default resources for xpdbx are listed in

/usr/lib/X11/app-defaults/Xpdbx• Needs the –rmpool option on the IBM

SP/Power4 for an interactive run• Can set break/trace points

xpdbx

4

• Serial and multi-thread debugger withgraphical interface

• xldb a.out– Window pops up with source, etc

• Group of blue bars at the top right– Click on bar to open window– To minimize window, click on bar at top to get

menu, click on “minimize”• To set breakpoint, click on source line• To navigate, see “commands” window

xldb

• Graphical user interface based parallel debuggerfor Fortran and C programs

• Same as xpdbx• pedb a.out

– Window pops up with source, etc

• To set breakpoint, double-click on source line. Todelete breakpoint, right-click icon in left margin,select “ delete”

• To navigate, use buttons below source listingStep over step into step return Continue halt play stop

• “tasks”(processes for MPI) are chosen usingbuttons in “task” window

• Use “Find” to find a specific variable

pedb

5

• gdb is a command line debugger for serialprograms

• Support C, C++, and Fortran77 programs• Available: linux and unix systems• Compile the code :

g77 –g my.f or gcc –g my.c• Start debugging:

gdb ./a.outor

gdb ./a.out core # to include the coreor

gdb ./a.out 1234 # to attach GDB to process 1234

GNU gdb

• Most useful Commands:quit: Exit from GDB.

break [file:]function: Set a breakpoint at function in file. run: Start your program bt: Backtrace: display the program stack. print expr: Display the value of an expression. c: Continue running your program (after stopping)

stop:set a breakpoint at line number or routine

• Getting help:– (gdb) help– http://sources.redhat.com/gdb/onlinedocs/gdb

_toc.html– man gdb

6

• CVD - Invokes the WorkShop Debugger• a source-level debugging tool with GUI• set various types of breakpoints, watchpoints, and so on.• view variables, expressions, structures, arrays, call tree• debug Ada, C, C++, Fortran 77, and Fortran 90 programs.

• Available on SGI system running Irix Operating systems(SGI workstations and Origin)

• To invoke the Debugger for a Fortran 77 program:$ f77 -g -o myprog myprog.f$ cvd myprog # serial job

or $ cvd myprog core # serial job with core dumpor $ cvd mpirun -args -np 2 a.out # For an mpi job

• More information:• Manpage: man cvd• http://techpubs.sgi.com/library/tpl/cgi-bin/init.cgi

CVD

CVD and Other Workshop Tools:• Build Analyzer (cvbuild) - Invokes the Build Analyzer that

provides a graphical display of build dependency information.• Parallel Analyzer (cvpav) - Invokes the Parallel Analyzer that

presents information about MP Fortran programs.• Performance Analyzer (cvperf) - Invokes the Performance

Analyzer that lets you specify the name of a performance taskto enable proper data collection for your experiment.

• Static Analyzer (cvstatic) - Invokes the Static Analyzer thatlets you display source code information about your code andfacilitates the setting of breakpoints.

• Tester (cvxcov) - Invokes the test coverage tool that lets youperform dynamic test coverage over any set of tests.

7

• Intel debugger are part of the compiler installation• Support C, C++, Fortran 77/90• Both dbx (default) and gdb like interfaces• Documentation:

– http://www.intel.com/software/products/compilers/docs/linux/idb_manual_l.html

• Debugging of optimized code generated by intel compilersdoes not work

• Does not support multithreaded/multiprocess applications.claims do support mpich programs (untested)

• Available on the SGI Altix and systems with Intel compilers

Intel IDB

• Graphical debugger• Part of Redhat distribution• Works best with gdb• Can hook in idb via: ddd --debugger “idb” -- dbx ./a.out• Multiple panes for

– Data/array browsing– Source code– Assembler code– Command line interface

• Use help menu for details

DDD

8

Other Debuggers

• Portland Group PGDBG graphical debugger• LF95: Fujitsu debugger FDB

• Write statements!• Editors

Totalview

9

Introduction• Debugger for Unix/Linux platform• Supports Fortran, C/C++ programs• Serial, multi-threaded, parallel programs

employing multiple processors• MPI, MPICH, OpenMP, PVM, SHMEM, Global

arrays, and UPC• User interface and debugging style remain

uniform and consistent across platforms andprogramming models.

Starting TotalView

• Compile with –g option• Starting Totalview on MSI machines:

module load totalview• On a new process: % totalview a.out -a • On a core file:

% totalview a.out core

• To attach to a running process:% totalview

10

o• File > New Program dialog Box• Short cut: ctrl-N

• F1 function keytells Totalview todisplay help infoabout the window

TotalView Windows

ProcessWindow

RootWindow

UnattachedProcessesWindow

DataWindows

•Menu driven•Alternative keyboard shortcuts for those who prefer them•All 3 mouse buttons are used

Test drive:

11

Root WindowProcess name

Number of threads

Expandlist

Process/thread status

Process Window

StackTrace pane

Sourcepane

Threadpane

Localvariablesfor theselectedframe

ActionPointspane

12

Current function and source fileGridded box isa possible sitefor a breakpoint

Selectto

setoneCurrent

point ofexecution

• Dive on a source word to get more information

• Select a line to use Run to selection command

• Select or dive on a line number to set an action point

Source Pane

Stack Trace Pane

Stack framevalue

• Select a line tochoose a new stackframe

• Just dive on aroutine andTotalView showsthe routine in theSource Pane andits variables in theStack Frame Pane.

13

Stack Frame Pane

Names ofarguments

Names offunction’slocal variables

Values of variablesStack Frame pane in the processwindow contains current call stack

Dive on a variable to bring up adata pane (and follow a pointer)

Starting, Stopping, & RestartingYour Program

• Method 1– Set a breakpoint– Select Go on the icon bar– The program starts executing and stop when it reaches

the breakpoint• Method 2

– Select step– Totalview starts and then stops it immediately before

the first statement in your main function• To stop a running program, select halt• To restart a program, select the Group > Restart

command

14

• Stepping through a Program– Use the Step and Next commands. Step and Next both

execute the current line. The difference between themis that if the line has a function call, Step dives into thefunction while Next executes it.

– If you want to get to a line without individually steppingeach intervening line, select the line, and then selectRun To.

• Stepping Out of a Function– If you've stepped into a function and want to pop out to

the statement that called it, select the Out command.

Data Management

15

Seeing a value of a variable• Method 1 Dive on a variable displayed in the

Source Pane. The variable appears in a VariableWindow.

• Method 2 Scroll to the variable in the StackFrame Pane. If you want to display a compoundvariable like an array or structure, dive on it.TotalView responds by displaying information in aVariable Window.

• Method 3 Use the View > Lookup Variablecommand. The variable can be global or local. Ifyou've defined the same variable in more than oneroutine, TotalView displays the variable containedin the current stack frame.

Displaying Data

Array Data Window• Array has an

actual type aswell as adeclaration type

• Arrays have aslice field thatyou can edit tospecify thedimensions todisplay

• You can updatedata by Selecting(left mousebutton) the valueand editing it

Variable Window

16

Array Slice

• An array sliceindicates whichportion of an arrayyou want to see. I.e.:see elements 3 to 8

• Fortran example-TotalView displays aone-dimensional arrayby using (:). It uses(:,:) if it is displayinga two-dimensionalarray, and so on

Data Array FilteringGo to theFilter line andtype a filterexpression

You canspecify theelements of anarray you’reinterested in

This examplerequests allelements withvalues >0.8

17

Data Array Filtering (cont.)

• Filter: 5:998 will display elements between5 and 998

• Filter:== $nan• Filter:== $inf

Visualization• In an array data pane select

Visualize from the menu

• The Visualizer is automaticallystarted

• Arrays of rank up to two may bevisualized

• Unlike data panes visualizationsare not automatically updated

• $visualize in EVAL points foranimated display

18

Action Points

Action Point Types“Action point” is a generic name for severaltypes of breakpoints:

– Breakpoint stops execution of processes andthreads that reach it

– Barrier breakpoint holds each process thatreaches it until all processes in the groupreach it

– Conditional breakpoint stops code if conditionis met

– Evaluation point is point at which user-definedcode fragments get executed

– Data watchpoint

19

Setting a Breakpoint at a Function• Method 1: If you can see the function, click on a

line number within the function.• Method 2: Use the View > Lookup Function

command to find the function. After the functiondisplays, select a line number.

• Method 3: Select Action Points > At Locationand then type the function's name in the dialogbox.

Setting a Data Watchpoint• A data watchpoint is a type of breakpoint that

monitors a variable's value. Unlike other actionpoints, it isn't set on a source line. Instead, it'sset on the memory location where your programstores the variable. When the contents of thismemory location change, TotalView stopsexecution and displays the source line thatchanged it. This means you should only setwatchpoints on global variables or on malloc'dmemory.

• Set a watchpoint by first diving on the variable todisplay its Variable Window, and then select theTools > Watchpoint command.

20

Printing Something at a Breakpoint• After creating a

breakpoint, right-click onthe sign, and then selectProperties from the popupmenu.

• In the Properties dialogbox, select Evaluate andthen type a printstatement. For example:

    printf("The value of foodivided by 20 is:%d\n",foo/20);

• When code is associatedwith a breakpoint, thebreakpoint is called aneval point.

Stopping Your Program Using a Condition• Create an eval point (this was discussed in the

previous section), entering a condition instead ofa statement. For example, here's a condition thatstops execution when the value of the variable i inthe eval point's line is equal to 30:

     if (i == 30) $stop• One of the big differences between an eval point

and a breakpoint is that execution doesn't stopunless you tell it to. That is, if you don't add a$stop to your eval point, TotalView just continuesexecuting.

•

21

Stopping Your Program Every "x" TimesTotalView Executes a Line

• Create an eval pointthat uses the $countstatement to tellTotalView how manytimes the line shouldexecute. For example:

     $count 20• $count is another of

TotalView's built-infunctions. $stop isimplied when you usethis statement.

Saving Your Action Points• You usually want this to happen

automatically and, unless someone haschanged things, TotalView automaticallysaves them. If it isn't saving them, go tothe Options Tab in the File > Preferencesdialog box and select Save preferencesfile on exit.

• While you're looking at preferences, youmight want to look at the other things youcan set.

22

Action Point List PaneEnabledbreakpoint

Disabledbreakpoint

Breakpointlocation

• Select on action point icon enables/disables it• Dive on action point refocuses source pane to

location; this behaves like a bookmark

Parallelism in TotalView

23

Supported Parallelism

TotalView supports all major parallelprogramming paradigms– MPI– OpenMP– Pthreads– PVM– HPF– Shmem

TotalView Multi-Process Functionality• TotalView performs actions on groups of

processes– Start / stop– Share breakpoints across executables– Take action on multiple processes when at

breakpoint– Attach to multiple running processes– Single step groups of processes– Set barrier breakpoints

• Easily view and manipulate variables processes– Laminated variables for threads and processes– Distributed HPF arrays

24

Debugging MPI Programs on the IBM SPwith Totalview

% totalview poe -a a.out

[program_args][PE_args]

• Automatically attaches to all of theMPI processes

• TotalView can display MPI message

queues

Displaying a variable in all processes or threads• Need to view or update the

value of a variable in all ofthe processes or threads atonce

1. First display an instance of thevariable in a Variable Window

2. View > Laminate > Process, orView > Laminate > Thread

3. View > Laminate > None,delaminates the window

• Example on the right is foran OpenMP

25

Other tools

• Profiler: Xprofiler:– GUI based IBM performance profiling tool– Graphical function call tree– Flat report: line by line CPU time– Compile and link your program with –pg– Run your code => produce gmon.out– Invoke xprofiler xprofiler executable [-s] gmon.out.

Other Tools

26

• Hardware counter: Hpm on the IBM systems– Hardware performance counters

• Cycles• Instructions• Float point and fixed point operations• Loads/stores• Cache misses• TLB misses• Etc

– Supports MPI, OpenMP and threaded applications– Multiple instrumentations and derived metrics

• MSI– www.msi.umn.edu– Help: help@,msi.umn.edu 612 626 0802

• Totalview– Visit www.etnus.com– Online documentation at

www.etnus.com/Support/docs/index.html