Model-Based Testing - ChalmersModel-Based Testing (DIT848 / DAT261) Spring 2017 Lecture 3 White Box...

41
Model-Based Testing (DIT848 / DAT261) Spring 2017 Lecture 3 White Box Testing - Coverage Gerardo Schneider Dept. of Computer Science and Engineering Chalmers | University of Gothenburg Some slides based on material by Magnus Björk, Thomas Arts and Ian Somerville 1

Transcript of Model-Based Testing - ChalmersModel-Based Testing (DIT848 / DAT261) Spring 2017 Lecture 3 White Box...

Page 1: Model-Based Testing - ChalmersModel-Based Testing (DIT848 / DAT261) Spring 2017 Lecture 3 White Box Testing - Coverage Gerardo Schneider Dept. of Computer Science and EngineeringTEST

Model-Based Testing (DIT848 / DAT261)

Spring 2017 Lecture 3

White Box Testing - Coverage

Gerardo Schneider Dept. of Computer Science and Engineering

Chalmers | University of Gothenburg

Some slides based on material by Magnus Björk, Thomas Arts and Ian Somerville

1

Page 2: Model-Based Testing - ChalmersModel-Based Testing (DIT848 / DAT261) Spring 2017 Lecture 3 White Box Testing - Coverage Gerardo Schneider Dept. of Computer Science and EngineeringTEST

What have we seen so far?

�  V&V:Valida)on&Verifica)on�  TheVmodel

�  Testlevels

�  Blackboxtes)ng�  (Extended)FiniteStateMachines

Anyques)on?

2

Page 3: Model-Based Testing - ChalmersModel-Based Testing (DIT848 / DAT261) Spring 2017 Lecture 3 White Box Testing - Coverage Gerardo Schneider Dept. of Computer Science and EngineeringTEST

TEST

Today’s topic Verification Validation

Code is executed •  sequential •  concurrent •  distributed

user

Dynamic Static Artifact Inspection

manual automatic •  Static analysis •  Formal methods White box

Coverage

Black box …… Several

techniques

3

Page 4: Model-Based Testing - ChalmersModel-Based Testing (DIT848 / DAT261) Spring 2017 Lecture 3 White Box Testing - Coverage Gerardo Schneider Dept. of Computer Science and EngineeringTEST

White box testing

4

Page 5: Model-Based Testing - ChalmersModel-Based Testing (DIT848 / DAT261) Spring 2017 Lecture 3 White Box Testing - Coverage Gerardo Schneider Dept. of Computer Science and EngineeringTEST

Do I need more test cases?

�  IthinkIhavetestcasesforallaspectsofthespecifica)on,

�  I’veaddedtestcasesforboundaryvalues,�  …guessederrorvalues,�  …andperformed10.000randomtestcases.

Isthatenough?

5

Page 6: Model-Based Testing - ChalmersModel-Based Testing (DIT848 / DAT261) Spring 2017 Lecture 3 White Box Testing - Coverage Gerardo Schneider Dept. of Computer Science and EngineeringTEST

Do I need more test cases? �  Thebad:

�  Thereisnowaytoknowforcertain

�  Thegood:�  Therearetechniquesthatcanhelpus

�  Iden)fysomeaspectsthatmayotherwisegounno)ced� Givesomecriteriafor”enough”

Coverage techniques

6

Page 7: Model-Based Testing - ChalmersModel-Based Testing (DIT848 / DAT261) Spring 2017 Lecture 3 White Box Testing - Coverage Gerardo Schneider Dept. of Computer Science and EngineeringTEST

Black box and White box testing

Blackboxtes)ng:Testtac)cinwhichthetestobjectisaddressedasaboxonecannotopen.

Atestisperformedbysendingasequenceofinputvaluesandobservingtheoutputwithoutusinganyknowledgeaboutthetestobjectinternals.

Whiteboxtes)ng:Testtac)cinwhichthetestobjectisaddressedasaboxonecanopen

Atestisperformedbysendingasequenceofinputvaluesandobservingtheoutputandinternalswhileexplicitlyusingknowledgeaboutthetestobjectinternals

event

starts

Another event

event

event

event

software

software

case Prop of true -> …..; false ->….. end

Also called Structural testing or Glass box testing

Page 8: Model-Based Testing - ChalmersModel-Based Testing (DIT848 / DAT261) Spring 2017 Lecture 3 White Box Testing - Coverage Gerardo Schneider Dept. of Computer Science and EngineeringTEST

What white box testing is not

� Whiteboxtes)ngis(typically)NOT:�  Blackboxtestcasesthatrefertointernalconstructs

Id: calc.h/pressPlus/1 Purpose: verifying that the correct operation is stored Precondition: state is a CalcStatePtr pointing to a valid calculator state Action: call pressPlus(state) Expected outcome: state->op = Plus Refers to internal

representation, not interface •  Drawbacksoftestcaseslikethis:

–  Testproper)esnotinspecifica)on–  Failifinternalrepresenta)onischanged–  Andwhentheyfail,itmaybehardtounderstandwhy/where

•  …butsome)mestheymaybenecessary–  Unittes)ngofinternalfunc)ons 8

Page 9: Model-Based Testing - ChalmersModel-Based Testing (DIT848 / DAT261) Spring 2017 Lecture 3 White Box Testing - Coverage Gerardo Schneider Dept. of Computer Science and EngineeringTEST

What white box testing is �  ’Normal’whiteboxtes)ngis:

�  Blackboxtes)ng,combinedwithtoolsthatanalyzeimplementa)onofspecificproper)es

� Whiteboxtechniquescoveredinthislecture�  Codecoverageanalysis

� Aretherepartsofthecodethatarenotexecutedbyanytestcases?

� Usedtofindinadequaciesinthetestsuite�  Inthislecture:SomeexamplesinC(GCov)andfunc)onal

programming9

Page 10: Model-Based Testing - ChalmersModel-Based Testing (DIT848 / DAT261) Spring 2017 Lecture 3 White Box Testing - Coverage Gerardo Schneider Dept. of Computer Science and EngineeringTEST

Coverage checking

10

Page 11: Model-Based Testing - ChalmersModel-Based Testing (DIT848 / DAT261) Spring 2017 Lecture 3 White Box Testing - Coverage Gerardo Schneider Dept. of Computer Science and EngineeringTEST

Coverage checking

Thestructureoftheso\wareisusedtodeterminewhetherasetoftestsisasufficient/adequateone

Thus:

1) Decidewhichstructuretoconsider2) Decideuponcoveragecriteria3) Findasetoftestssuchthatthisstructureis

coveredaccordingtothedecidedcriteria11

Page 12: Model-Based Testing - ChalmersModel-Based Testing (DIT848 / DAT261) Spring 2017 Lecture 3 White Box Testing - Coverage Gerardo Schneider Dept. of Computer Science and EngineeringTEST

Common structures �  Func5oncoverage

�  Allfunc)onshavebeenexecuted

�  Entry/exitcoverage�  Allentryandexitpointsofallfunc)onshavebeenexecuted

�  Entrypoints:allcallstoafunc)on

�  Exitpoints:eachreturnstatement

�  Statementcoverage(linesofcode)�  Alllinesofcodehavebeenexecuted

�  Branchcoverage(condi)oncoverage)�  If:both“if”and“else”part,evenifnoelsepart�  Whileloop:bothwithtrueandfalsecondi)ons

�  Lazylogicalops(&&and||):firstargumentsbothtrueandfalse

�  Pathcoverage�  Allpossibleroutesthroughthecode(combina)onofbranches)

�  Infinitelymanyiftherearewhileloops(onlyfeasibleforsmallfunc)ons)

How to cover exceptions?

Not easy, due to their dynamicity

12

Page 13: Model-Based Testing - ChalmersModel-Based Testing (DIT848 / DAT261) Spring 2017 Lecture 3 White Box Testing - Coverage Gerardo Schneider Dept. of Computer Science and EngineeringTEST

Example (Coverage in Functional Prog.) �  Afunc)onfrom(simple)ATMsystem

�  Representa)onofamountofcashinmachine:�  [(100,23),(500,11)]meansthatmachinecontains:

�  23billsof100kr�  11billsof500kr

�  Wecallit“pair-notes”

�  Func)ontolookat:subtract�  subtractanumberofnotes-notesremainingintheATM�  subtract(<list_of_pair-notes_to_withdraw>,<list_of_pair-notes_in_Bank>)

13

Page 14: Model-Based Testing - ChalmersModel-Based Testing (DIT848 / DAT261) Spring 2017 Lecture 3 White Box Testing - Coverage Gerardo Schneider Dept. of Computer Science and EngineeringTEST

subtract([],Notes) ->

Notes;

subtract([{Value,Nr}|Rest],Notes) ->

subtract(Rest,subtract2(Value,Nr,Notes)).

subtract2(Value,N,[{Value,M}|Notes]) when M>=N ->

[{Value,M-N}];

subtract2(Value,N,[{V,M}|Notes]) ->

[{M,V}|subtract2(Value,N,Notes)].

Test case: subtract([{500,2}],[{100,100},{500,3}]).

Expected output: [{100,100},{500,1}]

Example (Coverage in Functional Prog.)

14

Page 15: Model-Based Testing - ChalmersModel-Based Testing (DIT848 / DAT261) Spring 2017 Lecture 3 White Box Testing - Coverage Gerardo Schneider Dept. of Computer Science and EngineeringTEST

subtract([],Notes) ->

Notes;

subtract([{Value,Nr}|Rest],Notes) ->

subtract(Rest,subtract2(Value,Nr,Notes)).

subtract2(Value,N,[{Value,M}|Notes]) when M>=N ->

[{Value,M-N}];

subtract2(Value,N,[{V,M}|Notes]) ->

[{M,V}|subtract2(Value,N,Notes)].

Test case: subtract([{500,2}],[{100,100},{500,3}]).

subtract([{500,2}],[{100,100},{500,3}]).

Example (Coverage in Functional Prog.)

15

Page 16: Model-Based Testing - ChalmersModel-Based Testing (DIT848 / DAT261) Spring 2017 Lecture 3 White Box Testing - Coverage Gerardo Schneider Dept. of Computer Science and EngineeringTEST

subtract([],Notes) ->

Notes;

subtract([{Value,Nr}|Rest],Notes) ->

subtract(Rest,subtract2(Value,Nr,Notes)).

subtract2(Value,N,[{Value,M}|Notes]) when M>=N ->

[{Value,M-N}];

subtract2(Value,N,[{V,M}|Notes]) ->

[{M,V}|subtract2(Value,N,Notes)].

Test case: subtract([{500,2}],[{100,100},{500,3}]).

subtract([],subtract2(500,2,[{100,100},{500,3}])).

Example (Coverage in Functional Prog.)

16

Page 17: Model-Based Testing - ChalmersModel-Based Testing (DIT848 / DAT261) Spring 2017 Lecture 3 White Box Testing - Coverage Gerardo Schneider Dept. of Computer Science and EngineeringTEST

subtract([],Notes) ->

Notes;

subtract([{Value,Nr}|Rest],Notes) ->

subtract(Rest,subtract2(Value,Nr,Notes)).

subtract2(Value,N,[{Value,M}|Notes]) when M>=N ->

[{Value,M-N}];

subtract2(Value,N,[{V,M}|Notes]) ->

[{M,V}|subtract2(Value,N,Notes)].

Test case: subtract([{500,2}],[{100,100},{500,3}]).

subtract([],subtract2(500,2,[{100,100},{500,3}])).

Example (Coverage in Functional Prog.)

17

Page 18: Model-Based Testing - ChalmersModel-Based Testing (DIT848 / DAT261) Spring 2017 Lecture 3 White Box Testing - Coverage Gerardo Schneider Dept. of Computer Science and EngineeringTEST

subtract([],Notes) ->

Notes;

subtract([{Value,Nr}|Rest],Notes) ->

subtract(Rest,subtract2(Value,Nr,Notes)).

subtract2(Value,N,[{Value,M}|Notes]) when M>=N ->

[{Value,M-N}];

subtract2(Value,N,[{V,M}|Notes]) ->

[{M,V}|subtract2(Value,N,Notes)].

Test case: subtract([{500,2}],[{100,100},{500,3}]).

subtract([],subtract2(500,2,[{100,100},{500,3}])).

Example (Coverage in Functional Prog.)

18

Page 19: Model-Based Testing - ChalmersModel-Based Testing (DIT848 / DAT261) Spring 2017 Lecture 3 White Box Testing - Coverage Gerardo Schneider Dept. of Computer Science and EngineeringTEST

subtract([],Notes) ->

Notes;

subtract([{Value,Nr}|Rest],Notes) ->

subtract(Rest,subtract2(Value,Nr,Notes)).

subtract2(Value,N,[{Value,M}|Notes]) when M>=N ->

[{Value,M-N}];

subtract2(Value,N,[{V,M}|Notes]) ->

[{M,V}|subtract2(Value,N,Notes)].

Test case: subtract([{500,2}],[{100,100},{500,3}]).

subtract([],[{100,100}|subtract2(500,2,[{500,3}])]).

Example (Coverage in Functional Prog.)

19

Page 20: Model-Based Testing - ChalmersModel-Based Testing (DIT848 / DAT261) Spring 2017 Lecture 3 White Box Testing - Coverage Gerardo Schneider Dept. of Computer Science and EngineeringTEST

subtract([],Notes) ->

Notes;

subtract([{Value,Nr}|Rest],Notes) ->

subtract(Rest,subtract2(Value,Nr,Notes)).

subtract2(Value,N,[{Value,M}|Notes]) when M>=N ->

[{Value,M-N}];

subtract2(Value,N,[{V,M}|Notes]) ->

[{M,V}|subtract2(Value,N,Notes)].

Test case: subtract([{500,2}],[{100,100},{500,3}]).

subtract([],[{100,100}|subtract2(500,2,[{500,3}])]).

Example (Coverage in Functional Prog.)

20

Page 21: Model-Based Testing - ChalmersModel-Based Testing (DIT848 / DAT261) Spring 2017 Lecture 3 White Box Testing - Coverage Gerardo Schneider Dept. of Computer Science and EngineeringTEST

subtract([],Notes) ->

Notes;

subtract([{Value,Nr}|Rest],Notes) ->

subtract(Rest,subtract2(Value,Nr,Notes)).

subtract2(Value,N,[{Value,M}|Notes]) when M>=N ->

[{Value,M-N}];

subtract2(Value,N,[{V,M}|Notes]) ->

[{M,V}|subtract2(Value,N,Notes)].

Test case: subtract([{500,2}],[{100,100},{500,3}]).

subtract([],[{100,100}|subtract2(500,2,[{500,3}])]).

Example (Coverage in Functional Prog.)

21

Page 22: Model-Based Testing - ChalmersModel-Based Testing (DIT848 / DAT261) Spring 2017 Lecture 3 White Box Testing - Coverage Gerardo Schneider Dept. of Computer Science and EngineeringTEST

subtract([],Notes) ->

Notes;

subtract([{Value,Nr}|Rest],Notes) ->

subtract(Rest,subtract2(Value,Nr,Notes)).

subtract2(Value,N,[{Value,M}|Notes]) when M>=N ->

[{Value,M-N}];

subtract2(Value,N,[{V,M}|Notes]) ->

[{M,V}|subtract2(Value,N,Notes)].

Test case: subtract([{500,2}],[{100,100},{500,3}]).

subtract([],[{100,100}|[{500,3-2}]]).

Example (Coverage in Functional Prog.)

22

Page 23: Model-Based Testing - ChalmersModel-Based Testing (DIT848 / DAT261) Spring 2017 Lecture 3 White Box Testing - Coverage Gerardo Schneider Dept. of Computer Science and EngineeringTEST

subtract([],Notes) ->

Notes;

subtract([{Value,Nr}|Rest],Notes) ->

subtract(Rest,subtract2(Value,Nr,Notes)).

subtract2(Value,N,[{Value,M}|Notes]) when M>=N ->

[{Value,M-N}];

subtract2(Value,N,[{V,M}|Notes]) ->

[{M,V}|subtract2(Value,N,Notes)].

Test case: subtract([{500,2}],[{100,100},{500,3}]).

Evaluates to

subtract([],[{100,100}|[{500,1}]]).

Example (Coverage in Functional Prog.)

23

Page 24: Model-Based Testing - ChalmersModel-Based Testing (DIT848 / DAT261) Spring 2017 Lecture 3 White Box Testing - Coverage Gerardo Schneider Dept. of Computer Science and EngineeringTEST

subtract([],Notes) ->

Notes;

subtract([{Value,Nr}|Rest],Notes) ->

subtract(Rest,subtract2(Value,Nr,Notes)).

subtract2(Value,N,[{Value,M}|Notes]) when M>=N ->

[{Value,M-N}];

subtract2(Value,N,[{V,M}|Notes]) ->

[{M,V}|subtract2(Value,N,Notes)].

Test case: subtract([{500,2}],[{100,100},{500,3}]).

output: [{100,100},{500,1}]

All statements and all branches have been executed. Expected output.

Example (Coverage in Functional Prog.)

24

Page 25: Model-Based Testing - ChalmersModel-Based Testing (DIT848 / DAT261) Spring 2017 Lecture 3 White Box Testing - Coverage Gerardo Schneider Dept. of Computer Science and EngineeringTEST

subtract([],Notes) ->

Notes;

subtract([{Value,Nr}|Rest],Notes) ->

subtract(Rest,subtract2(Value,Nr,Notes)).

subtract2(Value,N,[{Value,M}|Notes]) when M>=N ->

[{Value,M-N}];

subtract2(Value,N,[{V,M}|Notes]) ->

[{V,M}|subtract2(Value,N,Notes)].

Example (Coverage in Functional Prog.)

25

Arewehappy?Istheprogramcorrect?

Whathappenwiththefollowing?

Test case: subtract([{500,2}],[{100,5},{500,3}]).

Itwillnotwork!!Thecase[100,100]wasapar)cularcase;weinvertedvalues!

Page 26: Model-Based Testing - ChalmersModel-Based Testing (DIT848 / DAT261) Spring 2017 Lecture 3 White Box Testing - Coverage Gerardo Schneider Dept. of Computer Science and EngineeringTEST

subtract([],Notes) ->

Notes;

subtract([{Value,Nr}|Rest],Notes) ->

subtract(Rest,subtract2(Value,Nr,Notes)).

subtract2(Value,N,[{Value,M}|Notes]) when M>=N ->

[{Value,M-N}|Notes];

subtract2(Value,N,[{V,M}|Notes]) ->

[{M,V}|subtract2(Value,N,Notes)].

Example (Coverage in Functional Prog.)

26

Arewehappynow?Istheprogramcorrect?

Whathappenwiththefollowing?

Test case: subtract([{100,2}],[{100,100},{500,3}]).

Itwillnotwork!!Weare“loosing”thesuffixofthelist!

Page 27: Model-Based Testing - ChalmersModel-Based Testing (DIT848 / DAT261) Spring 2017 Lecture 3 White Box Testing - Coverage Gerardo Schneider Dept. of Computer Science and EngineeringTEST

Coverage (example in C) voidprintPos(intn){prinl("Thisis");if(n<0) prinl("not");prinl("aposi)veinteger.\n");return;

}Code originally from Wikipedia

Test case 1

Action: call printPos(-1)

Expected outcome:

”This is not a positive integer” (printed on stdout)

Coverage: 100% statement, 50% branch, 50% path

Test case 2

Action: call printPos(1)

Expected outcome:

”This is a positive integer” (printed on stdout)

Coverage: 100% statement, branch & path (including previous)

Test case 3

Action: call printPos(0)

Expected outcome:

”This is not a positive integer” (printed on stdout)

Boundary value

Should be: <=

Actually: else { }

Fails!

27

Are we happy with coverage?

Page 28: Model-Based Testing - ChalmersModel-Based Testing (DIT848 / DAT261) Spring 2017 Lecture 3 White Box Testing - Coverage Gerardo Schneider Dept. of Computer Science and EngineeringTEST

Group exercise �  Comeupwithpiecesofcode(inanylanguage)andafew

testcasessuchthatfollowingcondi)onsaremet,ormo)vatewhyitisimpossible:

1.  100%branchcoverage,lessthan100%pathcoverage

2.  100%pathcoverage,lessthan100%statementcoverage

3.  100%func)oncoverage,lessthan100%exitpointcoverage

Group work: 10 min 28

Page 29: Model-Based Testing - ChalmersModel-Based Testing (DIT848 / DAT261) Spring 2017 Lecture 3 White Box Testing - Coverage Gerardo Schneider Dept. of Computer Science and EngineeringTEST

Suggestions 1:100%branchcoverage,lessthan100%pathcoverage

voidfoo(intn){

if(n>0)

prinl(”Posi)ve\n”);

else

prinl(”Notposi)ve\n”);

if(n%2)

prinl(”Odd\n”);

else

prinl(”Even\n”);

}

Id: Test case 1: pos/odd Action: call foo(1) Expected outcome: ”Positive” and ”Odd”

Id: Test case 2: neg/even Action: call foo(-2) Expected outcome: ”Not positive” and ”Even”

Path positive/even not covered! 29

Page 30: Model-Based Testing - ChalmersModel-Based Testing (DIT848 / DAT261) Spring 2017 Lecture 3 White Box Testing - Coverage Gerardo Schneider Dept. of Computer Science and EngineeringTEST

2:100%pathcoverage,lessthan100%statementcoverage

intmain(void){

prinl(”Helloworld\n”);

return0;

prinl(”Unreachablecode\n”);}

Id: Test case 1 Action: run main Expected outcome: ”Hello world” printed

This statement is not covered!

Suggestions

30

Page 31: Model-Based Testing - ChalmersModel-Based Testing (DIT848 / DAT261) Spring 2017 Lecture 3 White Box Testing - Coverage Gerardo Schneider Dept. of Computer Science and EngineeringTEST

3:100%func5oncoverage,lessthan100%exitpointcoverage

intabs(intn){

if(n<0)

return–n;

returnn;

}

Id: abs/1 Action: call abs(-17) Expected outcome: returns 17

Didn’t cover this exit point

Suggestions

31

Page 32: Model-Based Testing - ChalmersModel-Based Testing (DIT848 / DAT261) Spring 2017 Lecture 3 White Box Testing - Coverage Gerardo Schneider Dept. of Computer Science and EngineeringTEST

White box test design Strategyforusingcoveragemeasure:

1.  Designtestcasesusingblackboxtestdesigntechniques

2.  Measurecodecoverage

3.  Designtestcasesbyinspec)ngthecodetocoverunexecutedcode

100%coveragedoesnotmeantherearenoerrorsle\!

So,codecoverageshouldbeseenascomplementarymethod–Itcannotdothethinkingforyou

However,coverageanalysiscatchesaspectsthatareotherwiseeasilyforgouen

32

Page 33: Model-Based Testing - ChalmersModel-Based Testing (DIT848 / DAT261) Spring 2017 Lecture 3 White Box Testing - Coverage Gerardo Schneider Dept. of Computer Science and EngineeringTEST

Adding test cases after coverage analysis

�  Thenewtestcasesshoulds)llbeblackboxtestcases,notreferringtothecode

Id: abs/2 Purpose: Execute abs on negative integer Action: call abs(-17) Expected outcome: Call returns 17

Id: abs/2 Purpose: Cover line 3 of abs Action: call abs(-17) Expected outcome: Line 3 executed

Good test case:

Bad test case: Refers to code

33

Page 34: Model-Based Testing - ChalmersModel-Based Testing (DIT848 / DAT261) Spring 2017 Lecture 3 White Box Testing - Coverage Gerardo Schneider Dept. of Computer Science and EngineeringTEST

Practical coverage analysis

Inordertomeasurecoverage,mostlanguagesrequireacompileflagtoenablekeepingtrackoflinenumbersduringexecu)on

Consequences:

�  Performancechanges,hence5mingrelatedfaultsmaybeundiscoverable

� Memoryrequirementschange,henceonemayexperienceproblemsrunninginembeddeddevices

Therearealotoftoolsavailableformanylanguages 34

Page 35: Model-Based Testing - ChalmersModel-Based Testing (DIT848 / DAT261) Spring 2017 Lecture 3 White Box Testing - Coverage Gerardo Schneider Dept. of Computer Science and EngineeringTEST

Coverage vs Profiling

Bothmethodscountexecu)onsofen))es,butpurposeisdifferent

�  Coveragetool:findoutwhichen))eshavebeenexecuted,toestablishconfidenceinverifica)on

�  Profiler:iden)fyboulenecksandhelpprogrammerimproveperformanceofso\ware

35

Page 36: Model-Based Testing - ChalmersModel-Based Testing (DIT848 / DAT261) Spring 2017 Lecture 3 White Box Testing - Coverage Gerardo Schneider Dept. of Computer Science and EngineeringTEST

Example: Gcov (C) The program avg (short for “average”) reads a text file, whose name is given as a command line argument, containing a number of integers, and reports the average value of all the integers. The program has been implemented in C (see below and next page), and the following small test suite has been developed by a programmer to start testing the system: Test case avg.1: Normal integers Prerequisites: The file avgtest1.txt contains “10 15 35” Action: Run ./avg avgtest1.txt Expected outcome: The program prints “The average is 20” Test case avg.2: Negative numbers Prerequisites: The file avgtest2.txt contains “-2 2 -6” Action: Run ./avg avgtest2.txt Expected outcome: The program prints “The average is -2” Executing this test suite together with gcov reveals that there is untested code, the tool giving the message “Lines executed: 63.33% of 30”. The actual output from gcov can be seen in next slide.

NOTE: The uncovered statements are those lines preceded with ####

36

Page 37: Model-Based Testing - ChalmersModel-Based Testing (DIT848 / DAT261) Spring 2017 Lecture 3 White Box Testing - Coverage Gerardo Schneider Dept. of Computer Science and EngineeringTEST

Example: GCov -: 1:#include <stdio.h> -: 2:#include <stdlib.h> -: 3: -: 4:// readInts: read a file containing integers, and return their -: 5:// sum and the number of integers read. -: 6: -: 7:#define READINTS_SUCCESS 0 // Indicates success -: 8:#define READINTS_FILEERR 1 // the file could not be read -: 9:#define READINTS_SYNTAXERR 2 // syntax error in file -: 10: 2: 11:int readInts(const char* filename, int* sumRslt, int* lengthRslt){ 2: 12: FILE* file = fopen(filename, "r"); 2: 13: if(!file) #####: 14: return READINTS_FILEERR; -: 15: 2: 16: *sumRslt=0; 2: 17: *lengthRslt=0; -: 18: while(1) { -: 19: int theInt; 8: 20: if(fscanf(file, "%d", &theInt) == 1) { -: 21: // Successfully read integer 6: 22: (*sumRslt) += theInt; 6: 23: (*lengthRslt)++; -: 24: } else { -: 25: // Could not read integer. End of file or syntax error? 2: 26: if(feof(file)) { -: 27: // End of file 2: 28: fclose(file); 2: 29: return READINTS_SUCCESS; -: 30: } else { -: 31: // Syntax error #####: 32: fclose(file); #####: 33: return READINTS_SYNTAXERR; -: 34: } -: 35: } 6: 36: } -: 37:} -: 38:

2: 39:int main(int argc, char**argv) { -: 40: int sum, length; -: 41: const char* filename; -: 42: 2: 43: if(argc < 2) { #####: 44: printf("Error: missing argument\n"); #####: 45: exit(EXIT_FAILURE); -: 46: } 2: 47: filename = argv[1]; -: 48: 2: 49: switch(readInts(filename, &sum, &length)) { -: 50: case READINTS_FILEERR: #####: 51: printf("Error reading file %s\n", filename); #####: 52: exit(EXIT_FAILURE); -: 53: -: 54: case READINTS_SYNTAXERR: #####: 55: printf("Syntax error in file %s\n", filename); #####: 56: exit(EXIT_FAILURE); -: 57: -: 58: case READINTS_SUCCESS: -: 59: default: -: 60: break; -: 61: } -: 62: 2: 63: if(length==0) { #####: 64: printf("Error: no integers found in file %s\n", filename); #####: 65: exit(EXIT_FAILURE); -: 66: } -: 67: 2: 68: printf("The average is %d\n", sum / length); -: 69: 2: 70: return EXIT_SUCCESS; -: 71:} 37

Page 38: Model-Based Testing - ChalmersModel-Based Testing (DIT848 / DAT261) Spring 2017 Lecture 3 White Box Testing - Coverage Gerardo Schneider Dept. of Computer Science and EngineeringTEST

Group exercise

�  Provideaddi)onaltestcasessothatallcasestogetheryield100%statementcoverage

�  Writecompletetestcasesasshowninthetestcasesabove,andindicatewhichlineseachtestcasecover

Group work: 10 min 38

Page 39: Model-Based Testing - ChalmersModel-Based Testing (DIT848 / DAT261) Spring 2017 Lecture 3 White Box Testing - Coverage Gerardo Schneider Dept. of Computer Science and EngineeringTEST

Exercise: Proposed solution --Tocoverl.64-65(avgtest3.txtisanemptyfile-Testcaseavg3:

Prerequisites:Thefileavgtest3.txtexistsbutisempty

Ac)on:./avgavgtest3.txt

Expectedoutcome:Anerrorisreported,sta)ngthattheinputisempty

-Tocoverl.32-33and55-56-Testcaseavg4:

Prerequisites:avgtest4.txtcontainsalistofnon-integers

Ac)on:./avgavgtest4.txt

Expectedoutcome:Anerrormessageisgiventhatthereisasyntaxerror

Any problem understanding the

solution? Try it yourself with GCoV!

- -Tocoverl.14and51-52-Testcaseavg5:

Prerequisites:Callthefunc)onwithanargument,notafile

Ac)on:-./avg"asdfdf"(or./avgnon_exis)ng_file.txt)

Expectedoutcome:Anerrorreadingfilecouldbegiven

-Tocoverl.44-45Testcaseavg5:

Prerequisites:None

Ac)on:./avg

Expectedoutcome:Errormissingargumentisgiven

39

Page 40: Model-Based Testing - ChalmersModel-Based Testing (DIT848 / DAT261) Spring 2017 Lecture 3 White Box Testing - Coverage Gerardo Schneider Dept. of Computer Science and EngineeringTEST

TEST

Terminology

Verification Validation

Code is executed

user

Dynamic Static Artifact Inspection

manual automatic White box

Coverage

Black box …… Several

techniques

40

Page 41: Model-Based Testing - ChalmersModel-Based Testing (DIT848 / DAT261) Spring 2017 Lecture 3 White Box Testing - Coverage Gerardo Schneider Dept. of Computer Science and EngineeringTEST

Tes)ng:TheBiggerPicture� MondayMarch27

�  NOTE:Lecturestartsat8:30!

Next lecture

41