Software Testing Software Testing

testing.f12.ppt

CS 121“Ordering Chaos”

“Mike” Michael A. Erlinger

– 2 – CS 121

AdministriviaAdministrivia

Prepare for Sycamore on the 30Prepare for Sycamore on the 30thth….….

Need schedule of meeting with graders for User TestNeed schedule of meeting with graders for User Test

– 3 – CS 121

Software Testing: Convention WisdomSoftware Testing: Convention Wisdom

95% of errors are in 5% of the code95% of errors are in 5% of the code

…or maybe 80% of the errors are in 20% of the code…

Testing: Can I find that 5% or 20% via a test suite

– 4 – CS 121

Software TestingSoftware Testing

GoalsGoals

Types of testsTypes of tests

Levels of testsLevels of tests

Test measuresTest measures

Test planTest plan

– 5 – CS 121

GoalsGoals

Verification: Have we built the software right?Verification: Have we built the software right?

Validation: Have we built the right software?Validation: Have we built the right software?

Bug-free, meets specs

Meets customers’ needs

Are Verification and Validation different or variation of the same idea?My argument: “meeting specs” should equal “meeting customer needs”... not generally true (my kitchen, satellite systems)

– 6 – CS 121

Software TestingSoftware Testing

GoalsGoals

Types of testsTypes of tests

Levels of testsLevels of tests

Test measuresTest measures

Test planTest plan

most of our discussion focuses on verification

(more specifically bug testing)

– 7 – CS 121

Types of testsTypes of tests

Black boxBlack box

White boxWhite box

Gray boxGray box

– 8 – CS 121

Black Box TestsBlack Box Tests

input output

interface

1. Does it perform the specified functions?2. Does it handle obvious errors in input?3. Ariane5 – lousy error handling4. Classic ints vs floats, yards vs meters Black box should catch these if there is

adequate “test coverage”

– 9 – CS 121

Example: Ordered List of intsExample: Ordered List of ints

class OrdInts

create

getFirst

getNext

insert

delete

print

L=create()L.insert(5)L.insert(-1)L.insert(-1)p=L.getFirst()print (p)L.delete(p)p=L.getFirst()print(p)p=L.getNext(p)print(p)p=L.getNext(p)

-1-15error

– 10 – CS 121

Black box testsBlack box tests

Advantage: black box testerAdvantage: black box tester≠developer≠developer is unbiased by is unbiased by implementation details. e.g., Use Case testing, just work implementation details. e.g., Use Case testing, just work through all the Use Cases, building tests for eachthrough all the Use Cases, building tests for each

Disadvantage: black box tester is uninformed about Disadvantage: black box tester is uninformed about implementation detailsimplementation details unnecessary tests – test same thing in different way insufficient tests – can miss the extremes, especially if actual use

follows a different pattern – generally impossible to check all inputs

– 11 – CS 121

Black Box TestsBlack Box Tests

Input Code

choose good distribution of input – hope good distribution of code tested

x

x

x x

– 12 – CS 121

Unnecessary TestsUnnecessary Tests

Input Code

large range of input may exercise a small part of codee.g., operator test of satellite control stations, run through each

input and output light/key option. Testing same functions, whereas no one had a test for my map function.

– 13 – CS 121

Insufficient TestsInsufficient Tests

Input Code

a small range of input may exercise a large range of codebut can you ‘know’ this without knowing the code? Did we miss the 20%

– 14 – CS 121

Sufficient TestsSufficient Tests

Input Code

complexcode

a small range of input may exercise a small but important/error-prone region of code

– 15 – CS 121

White Box TestsWhite Box Tests

Based on codeBased on code

test 1

test 2

– 16 – CS 121

Example: ordered list of intsExample: ordered list of ints

class ordInts {class ordInts {

public: …public: …

private:private:

int vals[1000];int vals[1000];

int maxElements=1000;int maxElements=1000;

……

}}

– 17 – CS 121

White Box TestExample: ordered list of intsWhite Box TestExample: ordered list of ints

bool testMax(){ L=create(); num=maxElements; for (int i=0; i<=num; i++)

print iL.insert(i)

print maxElements;}

– 18 – CS 121

White Box TestsWhite Box TestsAdvantage:Advantage:

design tests to achieve good code coverage and avoid duplication can stress complicated, error-prone code Can be driven from Design, e.g., Use Cases or other Models can stress boundary values (fault injection)

Disadvantage:Disadvantage: tester=developer may have bias (team alternatives to test developer) if code changes, tests may have to be redesigned (is this bad?)

– 19 – CS 121

Gray Box TestsGray Box Tests

Look at code to design testsLook at code to design tests

But test through interfaceBut test through interface

Best of both worlds - maybeBest of both worlds - maybe

– 20 – CS 121

Example: ordered list of intsExample: ordered list of ints

class OrdInts

create

getFirst

getNext

insert

delete

print

L=create()L.insert(1)L.insert(2)L.insert(3)

L.insert(1001)p=L.getFirst()print(p)p=L.getNext(p)print p

123

…

……

– 21 – CS 121

Types of TestsTypes of TestsBlack box: test based on interface, through interfaceBlack box: test based on interface, through interface

Gray box: test based on code, through interfaceGray box: test based on code, through interface

White box: test based on code, through codeWhite box: test based on code, through code

Testing strategy can/should include all approaches!

My experience: Black Box = non developer, outside testing “idiots” who follow

specfications/requirements. in your case who?White Box = developer, part of development process in your case who?Gray Box = hated, non developer within developer organization in your case who?

– 22 – CS 121

Levels of TestsLevels of Tests

UnitUnit

IntegrationIntegration

SystemSystem

System IntegrationSystem Integration

white

black

– 23 – CS 121

Testing measures (white box)Testing measures (white box)

Code coverage – individual modulesCode coverage – individual modules

Path coverage – sequence diagramsPath coverage – sequence diagrams

Code coverage based on complexity – test of the risks, tricky Code coverage based on complexity – test of the risks, tricky part of code (e.g., Unix part of code (e.g., Unix ““you are not expected to understand you are not expected to understand thisthis”” code) code)

– 24 – CS 121

Code CoverageCode Coverage

How much of code is How much of code is ““testedtested”” by tests? by tests? manually profiling tools

Design new tests to extend coverageDesign new tests to extend coverage

Is 100% good? Is it possible to achieve?Is 100% good? Is it possible to achieve?

– 25 – CS 121

Path CoveragePath Coverage

How many execution paths have been exercised by tests?How many execution paths have been exercised by tests?

100% path coverage is usually impossible100% path coverage is usually impossible

Aim to cover Aim to cover commoncommon paths and error prone ( paths and error prone (complexcomplex) paths) paths

Aim to break code with tests – good testers are not liked by Aim to break code with tests – good testers are not liked by developers....developers....

– 26 – CS 121

Convention WisdomConvention Wisdom

95% of errors are in 5% of the code95% of errors are in 5% of the code

…or maybe 80% of the errors are in 20% of the code…

– 27 – CS 121

Code Complexity MeasuresCode Complexity Measures

cyclomatic complexity measure (McCabe)cyclomatic complexity measure (McCabe) measures number of linearly independent paths through

program ignores asynchronous interaction, fallibility of services etc.

developerdeveloper’’s best guess – what problems are likely and which s best guess – what problems are likely and which will be hardest to diagnosewill be hardest to diagnose

started with Knuth and others who gathered stats on programs.started with Knuth and others who gathered stats on programs.

– 28 – CS 121

Test planTest plan

Collection of tests: unit, integration, systemCollection of tests: unit, integration, system

Rationale for test: why these tests?Rationale for test: why these tests?

Strategy for developing/performing testsStrategy for developing/performing tests e.g., test units as developed, test integration at each build, run the

same tests over each build, etc. Final test plan for Version 1 should do all the unit tests, integration

tests, etc.

– 29 – CS 121

Testing StrategyTesting Strategy

TDD – test driven developmentTDD – test driven development

RegressionRegression

Test harnessTest harness

Bug trackingBug tracking

User testsUser tests

Write test BEFORE your write the code!

– 30 – CS 121

TDD – test DrivenTDD – test Driven

Unit tests are written first by the SEs. Then as code is written it Unit tests are written first by the SEs. Then as code is written it passes incrementally larger portions of the test suites. The passes incrementally larger portions of the test suites. The test suites are continuously updated as new failure test suites are continuously updated as new failure conditions and corner cases are discovered, and integrated conditions and corner cases are discovered, and integrated with regression tests. Unit tests are maintained like with regression tests. Unit tests are maintained like software and integrated into the build process. The goal is software and integrated into the build process. The goal is to achieve continuous deployment with frequent updates.to achieve continuous deployment with frequent updates.

– 31 – CS 121

Testing StrategyTesting Strategy

TDD – test driven developmentTDD – test driven development

RegressionRegression

Test harnessTest harness

Bug trackingBug tracking

User testsUser tests

AutomateBuild up number of depth of testsRun tests before you check in codeMy mess up historically

– 32 – CS 121

Testing StrategyTesting Strategy

TDD – test driven developmentTDD – test driven development

RegressionRegression

Test harnessTest harness

Bug trackingBug tracking

User testsUser tests Useful for interactive graphics applicationsautomated test framework, built one for satellite console to run through all console interactions

– 33 – CS 121

Test HarnessTest Harness

everything else

input output

inject test inputquery state

Test harnesses allow for the automation of tests. They can call functions with supplied parameters and print out and compare the results to the desired value. The test harness is a hook to the developed code, which can be tested using an automation framework.

– 34 – CS 121

StrategyStrategy

TDD – test driven developmentTDD – test driven development

RegressionRegression

Test harnessTest harness

Bug trackingBug tracking

User testUser test

Design system for trackingScrupulously record problems,their context, the effect, ideas on the cause,attempts to fix, create new tests as bugs uncovered

– 35 – CS 121

StrategyStrategy

TDD – test driven developmentTDD – test driven development

RegressionRegression

Test harnessTest harness

User testUser testValidationNonfunctional requirementsNot best for discovering bugsNeed to know how they generated bugHistorically, I have seen these used to Validate the final deliverable….too late….

– 36 – CS 121

Test PlanTest Plan

Collection of tests: unit, integration, systemCollection of tests: unit, integration, system

Rationale for test: why these tests?Rationale for test: why these tests?

Strategy for developing/performing testsStrategy for developing/performing tests

Be thoughtful in designingBe diligent in executing

– 37 – CS 121

Python & PyGamePython & PyGame

What tools are provided to do tests?What tools are provided to do tests?

What kind of tests are the tools suited to?What kind of tests are the tools suited to?

– 38 – CS 121

The EndThe End