Introduction to Software Testing

(Paul deGrandis)

[Reading assignment: Chapter 15, pp. 231-252 and notes by Paul deGrandis]

Overview

• Terms and Definitions

• Tool Overview

• Static Checkers

• Unit Testing

• Coverage and Profiling

• Memory Analysis

• Bug Reporting and Tracking

• Continuous Integration

NOTE

•All the material and examples from here on are generated using a single makefile.

•Log into tux, then:wget http://www.pauldee.org/vandvmakefile

mv vandvmakefile Makefile •Export your classpath:

export CLASSPATH=`pwd`/junit/junit-4.4.jar:`pwd`/emma/emma.jar:.

•To make and download all tools and examples:make

Tool Overview

•Verification• Static Checking

• Unit Tests for code correctness

• Memory and performance specifications

•Validation• Unit Tests for behavior and requirements

Static Checkers

•Does not execute the code

•Finds suspicious code or security vulnerabilities

•Tools

•C - SPLINT,GCC Warnings

•Java - FindBugs, PMD

Cyclone - a safe C

GNU Compiler Collection

FindBugs - Find Bugs in Java

Static Checkers - CScreenshots

Static Checkers - CDemo

• Installation of SPLINT

•Compiling using GCC Warning

•Running Splint

Static Checkers - CDemo

• Installation of SPLINT• Download

wget http://www.splint.org/downloads/splint-3.1.2.src.tgz

• Untartar -xzvf splint-3.1.2.src.tgz

• Make (during make, splint checks itself)cd splint-3.1.2/./configuremake

Static Checkers - CDemo

•Compiling using GCC Warnings

• see the screenshot three slides back

Static Checkers - CDemo

•Running SPLINT

• Simple and Basicsplint -preproc +weak +show-summary +stats ../src/*.c

• Dependable Softwaresplint -preproc +checks +show-summary +stats ../src/*.c

Static Checkers - JavaScreenshots

Static Checkers - JavaDemo

• Installation of FindBugs

•Running FindBugs

Static Checkers - JavaDemo

• Installation of FindBugs• Download

wget http://prdownloads.sourceforge.net/findbugs/findbugs-1.2.1.tar.gz

• Untartar -xvzf findbugs-1.2.1.tar.gz cd findbugs-1.2.1

Static Checkers - JavaDemo

•Running FindBugs

• Run It./findbugs

Unit Tests

• Validate a unit of code; smallest testable part

• Executes the code in a sandboxed environment

• Testing mostly for functional requirements• can also test some non-functional requirements

• Many approaches and schools of thought• History Based, Risk Based, Data Path, DOE

Unit Tests Frameworks-CScreenshots

Check Unit Test Framework

http://check.sourceforge.net

Demo

Unit Tests Frameworks-C

• Installation of Check

•Writing a Check unit test

•Compiling your test(s)

•Running your test(s)

http://check.sourceforge.net/doc/check.html/index.html

Demo

Unit Tests Frameworks-C

• Installation of Check• Download

wget http://downloads.sourceforge.net/check/check-0.9.5.tar.gz

• Untartar -xvzf check-0.9.5.tar.gz

• Make (notice the GCC warnings passed)cd check-0.9.5./configuremake

Demo

Unit Tests Frameworks-C

•Writing a Check unit test

• Write your test fixtures

• Write your test(s)

• Write your test suite

• Write your main method (execute your suite)

Demo

Unit Tests Frameworks-C

•Writing a Check unit test

• Write your test fixtures

Demo

Unit Tests Frameworks-C

•Writing a Check unit test

• Write your test(s)

Demo

Unit Tests Frameworks-C

•Writing a Check unit test

• Write your test suite

Demo

Unit Tests Frameworks-C

•Writing a Check unit test

• Write your main method (execute your suite)

Demo

Unit Tests Frameworks-C

•Compiling your test(s)

• See README in check/check-example/square

Demo

Unit Tests Frameworks-C

•Running your test(s)

• See README in check/check-example/square

Unit Tests Frameworks-JavaScreenshots

Demo

Unit Tests Frameworks-Java

• Installation of Junit

•Writing a Junit unit test

•Compiling your test(s)• Just use javac

•Running your test(s)• just use java

http://junit.sourceforge.net/ (look at the cookbook)

Demo

Unit Tests Frameworks-Java

• Installation of Junit• Download

wget http://downloads.sourceforge.net/junit/junit-4.4.jar

• Add the jar to your classpathexport CLASSPATH=$CLASSPATH:`pwd`/junit-4.4.jar:.

Demo

Unit Tests Frameworks-Java

•Writing a Junit unit test

•Write your test fixtures

•Write your test(s)

•Write your test suite

•Write your main method (execute your suite)

Demo

Unit Tests Frameworks-Java

•Writing a Junit unit test

• Write your test fixtures

Demo

Unit Tests Frameworks-Java

•Writing a Junit unit test

• Write your test(s)

Demo

Unit Tests Frameworks-Java

•Writing a Junit unit test

• Write your test suite(s)

Demo

Unit Tests Frameworks-Java

•Writing a Junit unit test

• Write your main method

Code Coverage

•Degree to how much code was tested (E.g., How confident are we in our tests and code)

•Criteria:• Statement - Has each line been executed

• Condition - Every evaluation (if/else, try/catch, switch, loop)

• Path - Every route of code

Code Coverage - CScreenshots

GCC - GCOV

DemoCode Coverage - C

See the Screenshot one slide back

Code Coverage - JavaScreenshots

EMMAhttp://emma.sourceforge.net/intro.html

DemoCode Coverage - Java

• Installation of Emma

• Downloadwget http://downloads.sourceforge.net/emma/emma-2.0.5312-lib.zip

• Unzipunzip emma-2.0.5312-lib.zip

• Instrument Your Codejava -cp emma.jar:./sample:../junit/junit-4.4.jar emma instr -outdir ./sampleout -cp ./sample/ SimpleTest

• Run Your Code and Generate a reportjava SimpleTestjava -cp emma.jar emma report -r html -in sampleout/coverage.em,sampleout/coverage.ec

Memory Analysis

•Sandboxes an application in a VM

•Uses:• Memory Leak - allocate memory, but don’t free

• Buffer Overflow - access memory outside of a fixed buffer

• Profiling - performance analysis

Memory Analysis - CScreenshots

Valgrind

DemoMemory Analysis - C

• Installation of Valgrind• Download

wget http://downloads.sourceforge.net/check/check-0.9.5.tar.gz

• Untartar -xvjf valgrind-3.2.3.tar.bz2

• Make (notice the GCC warnings passed)cd valgrind-3.2.3./configuremake

DemoMemory Analysis - C

•Running of Valgrind• Compile with debugging enabled

gcc -g myprog.c

• Run Valgrindvalgrind --leak-check=yes myprog arg1 arg2

• Full tutorial and infoformationhttp://valgrind.org/docs/manual/quick-start.html#quick-start.intro

Memory Analysis - JavaScreenshots

JConsole

Issue and Bug Tracking

•Organizes and tracks issues, enhancements, features, and bugs in a software system

•Usually contains a database backend and a web frontend

•Generate report of project and team metrics and productivity

Issue and Bug Tracking

•Trac (http://trac.edgewall.org/)

•Roundup (http://roundup.sourceforge.net/)

•Tracker (http://tracker.rubyforge.org/)

Continuous Integration

•Committing changes early and often

•Building and testing at least nightly

• If code breaks, revert

• Integration issues discovered early as developers work with the latest code

You now know …

• … static checkers• … unit testing tools• … coverage and profiling tools • … memory analysis tools• … Bug reporting and tracking tools