Software Tools and ProcessesTraining and Discussion

October 16, 2003

3:00-4:30 p.m.

Jim Willenbring

Overview

• Our most underutilized tool: Bonsai.• Taking advantage of Mailman.• Meta-bugs and Bugzilla.• Taking another step with CVS.• COVTOOL – a code coverage tool.• Autoconf and Automake.• Handling test harness failures effectively.• How to add tests to the test harness.• Check-in test harness.• The future of the test harness.• Where we build.• Are we satisfied with our current tools?• Any new tools we should consider?

Our most underutilized tool: Bonsai

• http://software.sandia.gov/bonsai/• Provides access to the CVS history of a project

via a web-based interface.• Can search based on filename, directory, branch,

date, and username.• Can “diff” any two versions of a file.• CVS commit messages are shown.• Very useful when trouble-shooting.

• There are very few messages sent to most of the package-level mail lists.– Consider sending information concerning new

features to the announce list.

– Conduct design discussions via the developers list (quite a bit of this is done).

– Remember some of the great things about Mailman.• Automatically sends to interested people.• Self documenting

– Good for researching why past decisions were made.

– Great for SQA

Taking advantage of Mailman

Meta-bugs and Bugzilla

• When starting to implement a new feature, consider using “meta-bugs”.

– Easily organize the development of the feature.• Assign bugs to different people.

• Track progress.

– Documents the entire process.– Little additional effort – should issue the three bugs anyway.

Meta-bug

bugbug bug

Taking another step with CVS

• In some cases we can improve our commit log messages.– Many leave great commit logs.

– Say a bit about what you did.

– Reference Bugzilla bug number.• Easy way to link the details of changes to the commit

(which we can access through Bonsai).• Ties all of the documentation of our changes

together.

• Has anyone had trouble with losing commit logs?

COVTOOL – a code coverage tool

• COVTOOL can be downloaded from http://covtool.sourceforge.net/

• Well documented• Easy to install and use• Provides useful statistics• Example configure-invocation scripts for compiling with COVTOOL on a Linux machine can be found in Trilinos/config/

Autoconf and Automake

• Do users have a positive impression of our Autotools system?

• Are there specific improvements that we can make?– --enable(disable)-package-tests/examples options.

– Option to disable all default packages (can be overridden by enabling single packages).

Handling test harness failures effectively

• Brief overview of the test harness.• A new way to handle test harness failures.

– The increase in the number of Trilinos packages has been causing problems.

• Bugs have remained open long enough to cause problems for other developers.

– When a failure occurs, I will issue a bug and then disable the package.

– When the developer addresses the bug, I will again enable the package.

How to add tests to the test harness

• See Trilinos/testharness/HowToAddToTestHarness.• Epetra and Amesos utilize the test harness in slightly different

ways. Both are good examples.• Provide a script that invokes the test.

– Return zero/non-zero to indicate pass/fail.– See Trilinos/packages/epetra/test/scripts/daily/mpi/epetraTestAllMpi for a

good example.

• Based on how often the test should run and whether it should run in serial or MPI mode, place the script in the proper directory.

• The biggest challenges– The script needs to determine whether the test passed or

failed.

The Structure of the Test Harness

Test-harness Script that calls exe

Test exe

Check-in Test Harness

• Can be used before committing.• Runs the “daily” test harness test for all

packages whose tests are enabled.• Does not work on Cygwin.• Instructions for use are located directly inside the

check-in test harness file:

Trilinos/testharness/checkin-test-harness

Check-in Test Harness Setup

• Must have build trees directly off of the Trilinos directory.

• The build trees to be tested must be listed in Trilinos/testharness/subdir-list.

• Each build directory to be tested must contain an executable script called invoke-configure that contains the configure line for the build.

Check-in Test Harness Setup (cont.)

• For MPI builds, --enable-mpi must appear on the first line of the script.

• If using LAM, perform a lamboot.• When this setup is complete, you can simply type:

perl checkin-test-harness

The Future of the Test Harness

• Where should we run the test harness?– Recently there have been requests to run on all

platforms we support.

• How often should we run the test harness? – The four current platforms are set up to run “daily” tests

six days a week, and weekly tests on the 7th day.

– Some have suggested that once a week is sufficient for some platforms.

The Future of the Test Harness (cont.)

• Improvements– Structure of mail messages

– Process (as mentioned)

– Platforms (as mentioned)

– Including more packages

Where We Build

1. Linux gcc 2.x 3.x2. Solaris3. SGI 32/64 (on its way out?)4. DEC/OSF5. Cygwin Intel Compilers6. Cygwin gcc7. IBM AIX8. OSX9. CPLANT10. Janus11. Red Storm12. QT13. ICC

Are we satisfied with our current tools?

• General impressions:– Developers are happy with tools– Developers see the value in using the tools

New Tools?

• Should we consider using any new tools?

• Are there issues that we would like to address if we could find an appropriate tool?