Testing in C
Transcript of Testing in C
7/23/2019 Testing in C
http://slidepdf.com/reader/full/testing-in-c 1/41
1
Testing
The material for this lecture is drawn, in part, from
The Practice of Programming (Kernighan & Pike) Chapter 6
Jennifer Rexford
7/23/2019 Testing in C
http://slidepdf.com/reader/full/testing-in-c 2/41
2
For Your Amusement
“On two occasions I have been asked [by members of Parliament!],
Pray, r" #abba$e, if yo% &%t into the machine wron$ fi$%res, will
the ri$ht answers come o%t'( I am not able ri$htly to a&&rehend the
kind of conf%sion of ideas that co%ld &rovoke s%ch a )%estion"*
‒ +harles #abba$e
“Pro$ram testin$ can be )%ite effective for showin$ the &resence
of b%$s, b%t is ho&elessly inade)%ate for showin$ their absence"*
‒ ds$er -i.kstra
“#eware of b%$s in the above code/ I have only &roved it correct,not tried it"*
-onald 0n%th‒
7/23/2019 Testing in C
http://slidepdf.com/reader/full/testing-in-c 3/41
“Programming in the Large” Steps
-esi$n 1 Im&lement2 Pro$ram 1 &ro$rammin$ style
2 +ommon data str%ct%res and al$orithms
2 od%larity
2 #%ildin$ techni)%es 1 tools
-eb%$2 -eb%$$in$ techni)%es 1 tools
3est2 3estin$ techni)%es <-- 4e are here
aintain2 Performance im&rovement techni)%es 1 tools
3
7/23/2019 Testing in C
http://slidepdf.com/reader/full/testing-in-c 4/41
Goals of this Lecture
5el& yo% learn abo%t62 Internal testin$
2 xternal testin$
2 7eneral testin$ strate$ies
4hy'2 It(s hard to know if a 8lar$e9 &ro$ram works &ro&erly
2 : &ower &ro$rammer s&ends at least as much time composing
test code as he;she s&ends com&osin$ the code itself
2 : &ower &ro$rammer knows how to s&end that time wisely
4
7/23/2019 Testing in C
http://slidepdf.com/reader/full/testing-in-c 5/41
5
Program Verification
Ideally6 Prove that any $iven &ro$ram is correct
7eneral
Pro$ram
+hecker program.c
Ri$ht or 4ron$<&ecification
?
7/23/2019 Testing in C
http://slidepdf.com/reader/full/testing-in-c 6/41
6
Program Testing
Pragmatically6 +onvince yo%rself that a specific
&ro$ram proaly works
<&ecific
3estin$<trate$y program.c
Probably Ri$ht
or
+ertainly 4ron$
<&ecification
7/23/2019 Testing in C
http://slidepdf.com/reader/full/testing-in-c 7/41
Agenda
!"ternal testing2 #esigning data to test your program
Internal testin$
2 -esi$nin$ yo%r &ro$ram to test itself
7eneral testin$ strate$ies
7
7/23/2019 Testing in C
http://slidepdf.com/reader/full/testing-in-c 8/41
8
Statement Testing
8=9 Statement testin$
2 “3estin$ to satisfy the criterion that each statement in a &ro$ram be
exec%ted at least once d%rin$ &ro$ram testin$"*
2 >rom the Glossary of Computerized System and Software
Development Terminology
7/23/2019 Testing in C
http://slidepdf.com/reader/full/testing-in-c 9/41
9
Statement Testing !"ample
xam&le &se%docode6
if (condition1) statement1;else statement2 ;
…if (condition2 ) statement3 ;else statement4;…
Statement testin$6
<ho%ld make s%re both ifstatements and all ? nested
statements are exec%ted
5ow many &asses
thro%$h code are
re)%ired'
7/23/2019 Testing in C
http://slidepdf.com/reader/full/testing-in-c 10/41
10
Path Testing
8@9 Path testin$
2 “3estin$ to satisfy covera$e criteria that each lo$ical &ath thro%$h
the &ro$ram be tested" Often &aths thro%$h the &ro$ram are$ro%&ed into a finite set of classes" One &ath from each class is then
tested"*
2 >rom the Glossary of Computerized System and Software
Development Terminology
7/23/2019 Testing in C
http://slidepdf.com/reader/full/testing-in-c 11/41
11
Path Testing !"ample
xam&le &se%docode6
2 <im&le &ro$rams AB maybe reasonable
2 +om&lex &ro$ram AB combinatorial ex&losion!!!
2 Path test code fra$ments
if (condition1) statement1;else statement2 ;…if (condition2 ) statement3 ;else statement4;…
Path testin$6
<ho%ld make s%re all lo$ical
&aths are exec%ted
5ow many &asses
thro%$h code are
re)%ired'
7/23/2019 Testing in C
http://slidepdf.com/reader/full/testing-in-c 12/41
12
$oundary Testing
8C9 $oundary testin$ 8alias corner case testin$9
2 “: testin$ techni)%e %sin$ in&%t val%es at, .%st below, and .%st
above, the defined limits of an in&%t domain/ and with in&%t val%esca%sin$ o%t&%ts to be at, .%st below, and .%st above, the defined
limits of an o%t&%t domain"*
2 >rom the Glossary of Computerized System and Software
Development Terminology
7/23/2019 Testing in C
http://slidepdf.com/reader/full/testing-in-c 13/41
13
$oundary Testing !"ample
<&ecification62 Print the n elements of array a to stdout, in reverse order
:ttem&t6
void printBackwards(int a[] unsigned int n)! unsigned int i; for (i " n; i #" $; i--) printf(%&d'n% a[i]);
-oes it work' :&olo$ies for the
forward reference
to arrays
7/23/2019 Testing in C
http://slidepdf.com/reader/full/testing-in-c 14/41
14
Stress Testing
8?9 Stress testin$
2 “3estin$ cond%cted to eval%ate a system or com&onent at or beyond
the limits of its s&ecified re)%irements*
2 >rom the Glossary of Computerized System and Software
Development Terminology
7/23/2019 Testing in C
http://slidepdf.com/reader/full/testing-in-c 15/41
15
Stress Testing
<ho%ld stress the &ro$ram with res&ect to62 %uantity of data
2 Dar$e data sets
2 Variety of data
2 3ext%al data sets containin$ nonE:<+II chars
2 #inary data sets
2 Randomly $enerated data sets
<ho%ld %se com&%ter to $enerate in&%t sets2 :voids h%man biases
7/23/2019 Testing in C
http://slidepdf.com/reader/full/testing-in-c 16/41
16
Stress Testing !"ample &
<&ecification62 Print n%mber of characters in stdin
:ttem&t6
include <stdio.*#
int main(void)! c*ar c*ar+ount " $; w*ile (getc*ar() ," /) c*ar+ount00; printf(%&d'n% c*ar+ount); return $;
-oes it work'
7/23/2019 Testing in C
http://slidepdf.com/reader/full/testing-in-c 17/41
17
Stress Testing !"ample '
<&ecification62 Read a line from stdin
2 <tore as strin$ 8witho%t 1'n19 in array of len$th 2332456789:
:ttem&t6
int i; c*ar s[2332456789:]; for (i " $; i < 2332456789:-; i00) ! s[i] " getc*ar(); if ((s[i] "" /) (s[i] "" 1'n1)) =reak;
s[i] " 1'$1;
-oes it work'
7/23/2019 Testing in C
http://slidepdf.com/reader/full/testing-in-c 18/41
18
!"ternal Testing Summary
xternal testin$6 -esi$nin$ data to test yo%r &ro$ram
xternal testin$ taxonomy
8=9 <tatement testin$
8@9 Path testin$ 8C9 #o%ndary testin$
8?9 <tress testin$
7/23/2019 Testing in C
http://slidepdf.com/reader/full/testing-in-c 19/41
Agenda
xternal testin$2 -esi$nin$ data to test yo%r &ro$ram
Internal testing
2 #esigning your program to test itself
7eneral testin$ strate$ies
19
7/23/2019 Testing in C
http://slidepdf.com/reader/full/testing-in-c 20/41
20
Aside( The assert )acro
assert(int e>pr)2 If e>pr eval%ates to 3RF 8nonEGero96
2 -o nothin$
2 If e>pr eval%ates to >:D< 8Gero96
2 Print messa$e to stderr “assert at line x failed*
2 xit the &rocess
Fsef%l for internal testin$
7/23/2019 Testing in C
http://slidepdf.com/reader/full/testing-in-c 21/41
21
Aside( The assert )acro
-isablin$asserts2 3o disable asserts, define 7B@8H
2 In code6
2 Or when b%ildin$6
A------------------------------------AA mCprogram.c A
A------------------------------------Adefine 7B@8…A 2sserts are disa=led *ere. A…
D gccEF G 7B@8 mCprogram.c Go mCprogram
7/23/2019 Testing in C
http://slidepdf.com/reader/full/testing-in-c 22/41
22
Validating Parameters
8=9 alidate &arameters2 :t leadin$ ed$e of each f%nction, make s%re val%es of &arameters
are valid
int f(int i dou=le d)! assert(i has a reasonable value); assert(d has a reasonable value); …
7/23/2019 Testing in C
http://slidepdf.com/reader/full/testing-in-c 23/41
23
Validating Parameters
2 xam&le
A 3eturn t*e greatest common divisor of positive integers
i and H. A
int gcd(int i int H)! assert(i # $); assert(H # $); …
7/23/2019 Testing in C
http://slidepdf.com/reader/full/testing-in-c 24/41
24
*hec+ing In,ariants
8@9 +heck invariants2 :t leadin$ ed$e of f%nction, check as&ects of data str%ct%res that
sho%ld not vary/ maybe at trailin$ ed$e too
int isIalid (JC9Cpe o=Hect)
! … A +ode to c*eck invariants goes *ere. 3eturn (93@) if o=Hect passes all tests and $ (/26K) ot*erwise. A …
void mC/unction(JC9Cpe o=Hect)! assert(isIalid(o=Hect)); … A +ode to manipulate o=Hect goes *ere. A … assert(isIalid(o=Hect));
7/23/2019 Testing in C
http://slidepdf.com/reader/full/testing-in-c 25/41
25
*hec+ing In,ariants
2 xam&le
2 “#alanced binary search tree insertion* f%nction
2 :t leadin$ ed$e6
2 :re nodes sorted'
2 Is tree balanced'2 :t trailin$ ed$e6
2 :re nodes still sorted'
2 Is tree still balanced'
7/23/2019 Testing in C
http://slidepdf.com/reader/full/testing-in-c 26/41
26
*hec+ing -eturn Values
8C9 +heck f%nction ret%rn val%es2 +heck val%es ret%rned by called f%nctions
some3etIalue " f(some2rgs);if (some3etIalue "" =adIalue) A :andle t*e error A…
f(some2rgs);… Bad code (sometimes)
Good code
if (f(some2rgs) "" =adIalue) A :andle t*e error A…
Good code
7/23/2019 Testing in C
http://slidepdf.com/reader/full/testing-in-c 27/41
27
*hec+ing -eturn Values
2 xam&le6
2 scanf89 ret%rns n%mber of val%es read
2 +aller sho%ld check ret%rn val%e
int i H;
…if (scanf(%&d&d% Li LH) ," E) A :andle t*e error A
int i H;…scanf(%&d&d% Li LH);
Bad code
Good code
7/23/2019 Testing in C
http://slidepdf.com/reader/full/testing-in-c 28/41
28
*hec+ing -eturn Values
2 xam&le6
2 &rintf89 ret%rns n%mber of chars 8not val%es9 written
2 +an fail if writin$ to file and disk )%ota is exceeded
2 +aller sho%ld check ret%rn val%e'''
int i " $$$;…if (printf(%&d% i) ," M) A :andle t*e error A
int i " $$$;…
printf(%&d% i);Is this too
m%ch'
Bad code???
Good code???
7/23/2019 Testing in C
http://slidepdf.com/reader/full/testing-in-c 29/41
29
*hanging *ode Temporarily
8?9 +han$e code tem&orarily2 3em&orarily chan$e code to $enerate artificial bo%ndary or stress
tests
2 xam&le6 :rrayEbased sortin$ &ro$ram
2 3em&orarily make array very small
2 -oes the &ro$ram handle overflow'
7/23/2019 Testing in C
http://slidepdf.com/reader/full/testing-in-c 30/41
30
Lea,ing Testing *ode Intact
89 Deave testin$ code intact
2 -o not remove testin$ code when &ro$ram is finished
2 In the “real world* no &ro$ram ever is “finished*!!!
2 If testin$ code is inefficient6
2 mbed in calls of assert(), or
2 Fse ifdef…endif &re&rocessor directives
2 <ee :&&endix
7/23/2019 Testing in C
http://slidepdf.com/reader/full/testing-in-c 31/41
31
Internal Testing Summary
Internal testin$6 -esi$nin$ yo%r &ro$ram to test itself
Internal testin$ techni)%es8=9 alidatin$ &arameters
8@9 +heckin$ invariants8C9 +heckin$ f%nction ret%rn val%es
8?9 +han$in$ code tem&orarily
89 Deavin$ testin$ code intact
$e.are of conflict et.een
internal testing and code clarity
7/23/2019 Testing in C
http://slidepdf.com/reader/full/testing-in-c 32/41
Agenda
xternal testin$2 -esi$nin$ data to test yo%r &ro$ram
Internal testin$
2 -esi$nin$ yo%r &ro$ram to test itself
General testing strategies
32
7/23/2019 Testing in C
http://slidepdf.com/reader/full/testing-in-c 33/41
33
Automation
8=9 :%tomate the tests2 +reate scripts to test yo%r programs
2 +reate software clients to test yo%r modules
2 +om&are im&lementations 8when &ossible9
2 ake s%re inde&endent im&lementations behave the same
2 0now what o%t&%t to ex&ect 8when &ossible9
2 7enerate o%t&%t that is easy to reco$niGe as ri$ht or wron$
:%tomated testin$ can &rovide62 %ch better covera$e than man%al testin$
2 #on%s6 xam&les of ty&ical %se of yo%r code
7/23/2019 Testing in C
http://slidepdf.com/reader/full/testing-in-c 34/41
34
Testing Incrementally
8@9 3est incrementally2 3est as yo% com&ose code
2 :dd test cases as yo% com&ose new code
2 -o regression testing
2 :fter a b%$ fix, make s%re &ro$ram has not “re$ressed*
2 3hat is, make s%re &revio%sly workin$ code is not broken
2 Rer%n all test cases
2 Kote the val%e of a%tomation!!!
2 +reate scaffolds and stus as a&&ro&riateH
7/23/2019 Testing in C
http://slidepdf.com/reader/full/testing-in-c 35/41
35
Testing Incrementally
/unction E
/unction N /unction M
/unction
Scaffold6 3em&orary
code that calls code
that yo% care abo%t
Stu6 3em&orarycode that is called
by code that yo%
care abo%t
+ode that
yo% care abo%t
7/23/2019 Testing in C
http://slidepdf.com/reader/full/testing-in-c 36/41
36
$ug/#ri,en Testing
8C9 Det deb%$$in$ drive testin$
2 Reactive modeH
2 >ind a b%$ AB create a test case that catches it
2 Proactive modeH
2 -o fault in0ection
2 Intentionally 8tem&orarily!9 in.ect a b%$
2 ake s%re testin$ mechanism catches it2 3est the testin$!!!
7/23/2019 Testing in C
http://slidepdf.com/reader/full/testing-in-c 37/41
37
General Strategies Summary
7eneral testin$ strate$ies8=9 :%tomation
8@9 3estin$ incrementally
8C9 #%$Edriven testin$
7/23/2019 Testing in C
http://slidepdf.com/reader/full/testing-in-c 38/41
38
1ho #oes the Testing2
Pro$rammers2 1hite/o" testin$
2 Pro6 0now the code AB can test all statements;&aths;bo%ndaries
2 +on6 0now the code AB biased by code desi$n
L%ality :ss%rance 8L:9 en$ineers2 $lac+/o" testin$
2 Pro6 -o not know the code AB %nbiased by code desi$n
2 +on6 -o not know the code AB %nlikely to test all
statements;&aths;bo%ndaries
+%stomers2 Field testin$
2 Pros6 Fse code in %nex&ected ways/ “deb%$* s&ecs
2 +ons6 Often don(t like “&artici&atin$*/ diffic%lt to $enerate eno%$h
cases
7/23/2019 Testing in C
http://slidepdf.com/reader/full/testing-in-c 39/41
39
Summary
xternal testin$ taxonomy2 <tatement testin$
2 Path testin$
2 #o%ndary testin$
2 <tress testin$
Internal testin$ techni)%es2 alidatin$ &arameters
2 +heckin$ invariants
2 +heckin$ f%nction ret%rn val%es
2 +han$in$ code tem&orarily
2 Deavin$ testin$ code intact
7/23/2019 Testing in C
http://slidepdf.com/reader/full/testing-in-c 40/41
Summary 3cont45
7eneral testin$ strate$ies2 :%tomation
2 +om&arin$ im&lementations
2 0nowin$ what o%t&%t to ex&ect
2 3estin$ incrementally
2 Re$ression testin$
2 <caffolds and st%bs
2 #%$Edriven testin$
2>a%lt in.ection
3est the code M and the tests6
40
7/23/2019 Testing in C
http://slidepdf.com/reader/full/testing-in-c 41/41
41
Appendi"( ifdef
Fsin$ ifdef…endif
2 3o enable testin$ code6
2 3o disable testin$ code6
…ifdef 9K95/29@35OA +ode to test feature O goes *ere. A
endif…
D gccEF G 9K95/29@35O mCprog.c Go mCprog
myprog.c
D gccEF mCprog.c Go mCprog