Case Study: PETSc ex19

Non-linear solver (snes) 2-D driven cavity code uses velocity-velocity formulation finite difference discretization on a structured grid

Instrumentation: MPI

MPI wrapper interposition library PDT (tau_instrumentor)

PETSc library sources Application

Selective instrumentation (tau_reduce)

Configuring TAU

Profiling with wallclock time (on a quad PIII Linux machine) % configure –mpiinc=/usr/local/packages/mpich/include

–mpilib=/usr/local/packages/mpich/lib –pdt=/usr/pkg/pdtoolkit/ -useropt=-O2 –LINUXTIMERS

Tracing % configure –mpiinc=/usr/local/packages/mpich/include

–mpilib=/usr/local/packages/mpich/lib –pdt=/usr/pkg/pdtoolkit –useropt=-O2 –LINUXTIMERS

Profiling with PAPI % configure –mpiinc=/usr/local/packages/mpich/include

–mpilib=/usr/local/packages/mpich/lib –pdt=/usr/pkg/pdtoolkit/ -useropt=-O2 –papi=/usr/local/packages/papi % setenv PAPI_EVENT PAPI_FP_INS % setenv PAPI_EVENT PAPI_L1_DCM

Compiling: TAU Makefiles

Include TAU Stub Makefile (<arch>/lib) in the user’s Makefile. Variables:

TAU_CXX Specify the C++ compiler used by TAU TAU_CC, TAU_F90 Specify the C, F90 compilers TAU_DEFS Defines used by TAU. Add to CFLAGS TAU_LDFLAGS Linker options. Add to LDFLAGS TAU_INCLUDE Header files include path. Add to CFLAGS TAU_LIBS Statically linked TAU library. Add to LIBS TAU_SHLIBS Dynamically linked TAU library TAU_MPI_LIBS TAU’s MPI wrapper library for C/C++ TAU_MPI_FLIBS TAU’s MPI wrapper library for F90 TAU_FORTRANLIBS Must be linked in with C++ linker for F90. TAU_DISABLE TAU’s dummy F90 stub library

Note: Not including TAU_DEFS in CFLAGS disables instrumentation in C/C++ programs (TAU_DISABLE for f90).

Profiling ex19 (wallclock time)

Profiling ex19 (wallclock time)

Tracing ex19

Callgraph Display for ex19

Using TAU with PAPI (PAPI_FP_INS)

Using TAU with PAPI (PAPI_L1_DCM)

Using TAU with PAPI (PAPI_FP_INS)

Using TAU with PAPI (PAPI_FP_INS)

Support Acknowledgement

TAU and PDT support: Department of Energy (DOE)

DOE 2000 ACTS contract DOE MICS contract DOE ASCI Level 3 (LANL, LLNL) U. of Utah DOE ASCI Level 1 subcontract

DARPA NSF National Young Investigator (NYI) award