National Alliance for Medical Image Computing http://na-mic.org

Testing with CTest and CDash

Julien JomierKitware Inc.

National Alliance for Medical Image Computing http://na-mic.org

Testing Overview

CVS/SVN

CDash

Web Browser

CTest

Developers

National Alliance for Medical Image Computing http://na-mic.org

Testing Terminology

Sites

dash5.kitware

iris.elmtech

Linux-g++

Builds

Linux-gcc3

Linux-gcc4

Dashboard

National Alliance for Medical Image Computing http://na-mic.org

Why Test Daily?

• Large code base too large for any single developer to understand

• Developers distributed around the world

• Identify problems as they occur

• Insure that object API remains unchanged

• Provide feedback to developers as they experiment with new implementations

National Alliance for Medical Image Computing http://na-mic.org

“If it’s not tested it doesn’t work”

• Dasboards and Software Process– Roots in GE Six Sigma initiative

• Cross platform testing

• Nightly regression testing Memory (Purify/ValGrind) testing

• Coverage testing

• Style checking

• Documentation evaluation

National Alliance for Medical Image Computing http://na-mic.org

How to write a good test?

• Use empirical values (5/2=2.5)

• Test most of the use cases (even the boring/most difficult ones)

• Make your tests as efficient as possible

• Have someone else write the test for your algorithm

National Alliance for Medical Image Computing http://na-mic.org

Regression Testing

• Compare generated image against standard “correct” image– pixel-by-pixel comparison– can use a threshold metric– adjusted for effects like dithering

National Alliance for Medical Image Computing http://na-mic.org

CTest

• Testing client

• Distributed with CMake

• Submit test results to the dashboard

• XML submission

• Independent of CMake

National Alliance for Medical Image Computing http://na-mic.org

CTest with CMakeCMakeLists.txtINCLUDE(CTest)ENABLE_TESTING()# Add a simple testADD_TEST(CDashTest1 CDashTest CDashTest) ADD_EXECUTABLE(CDashTest main.cxx)

CTestConfig.txtset(CTEST_PROJECT_NAME "MyProject")set(CTEST_NIGHTLY_START_TIME "21:00:00 EDT")

set(CTEST_DROP_METHOD "http")set(CTEST_DROP_SITE "localhost")set(CTEST_DROP_LOCATION "/CDash/submit.php?

project=MyProject")set(CTEST_DROP_SITE_CDASH TRUE)

National Alliance for Medical Image Computing http://na-mic.org

Testing Submission

XML Submission

XML Parsing

HTTP PUT

CTest

CDash

Backup XML File

Fill Database

Send Email

Update RSS Feed

Update.xml

Configure.xml

Build.xml

Test.xml

Coverage.xml

DynamicAnalysis.xml

Compiler/Tester

National Alliance for Medical Image Computing http://na-mic.org

CDash

• New generation of Dart• Written in Php/Javascript• MySQL• Open Source• Run on top of a web server• XSL implementation• Web 2.0

National Alliance for Medical Image Computing http://na-mic.org

CDash Features

• Ajax requests• Graphs: build, test times• Builds group• User management• Email notifications• Timing defects

National Alliance for Medical Image Computing http://na-mic.org

Coverage Support

• Email Low Coverage• Bullseye support

National Alliance for Medical Image Computing http://na-mic.org

Build Notes

• Share with other developers• Keep it simple

National Alliance for Medical Image Computing http://na-mic.org

My CDash

• Personal Page• Subscribe to project• View sites• Email preferences

National Alliance for Medical Image Computing http://na-mic.org

Project Roles

• Normal user• Site maintainer• Project administrator

National Alliance for Medical Image Computing http://na-mic.org

Claiming Sites

• Site information• Automatic (CTest 2.6)• Information history• Manual editing• Personal view/email

National Alliance for Medical Image Computing http://na-mic.org

CDash Misc.

• Automatic Geolocation from IP addresses

• Support for iPhone

National Alliance for Medical Image Computing http://na-mic.org

Installing CDash

• (L/W)AMP : Apache, MySQL, PHP- XAMPP, EasyPHP

• Download CDash (www.cdash.org) in your Apache web directory

• Edit config.php• Point Web Browser to CDash

- Setup the database- Create first project

National Alliance for Medical Image Computing http://na-mic.org

Style Checking

• KWStyle• C++ Style checker• Configuration via XML• HTML Generator (TBI in CDash) • Open Source• Written in C++

National Alliance for Medical Image Computing http://na-mic.org

Contribute• Website

- http://www.cdash.org- http://public.kitware.com/KWStyle

• Bug Tracker- http://www.cmake.org/Bug

• Mailing List- http://public.kitware.com/cgi-bin/mailman/listinfo/cdash

National Alliance for Medical Image Computing http://na-mic.org

Happy Testing!