Software Testing with QuickCheck Lecture 1 Properties and Generators.
Specification Based Testing with QuickCheck
Transcript of Specification Based Testing with QuickCheck
![Page 1: Specification Based Testing with QuickCheck](https://reader031.fdocuments.net/reader031/viewer/2022012101/6169f47c11a7b741a34d387c/html5/thumbnails/1.jpg)
Specification Based Testing with QuickCheck
John Hughes
Chalmers University/Quviq AB
![Page 2: Specification Based Testing with QuickCheck](https://reader031.fdocuments.net/reader031/viewer/2022012101/6169f47c11a7b741a34d387c/html5/thumbnails/2.jpg)
![Page 3: Specification Based Testing with QuickCheck](https://reader031.fdocuments.net/reader031/viewer/2022012101/6169f47c11a7b741a34d387c/html5/thumbnails/3.jpg)
What is QuickCheck?
• A library for writing and testing properties of program code
• Some code:
• A property:
![Page 4: Specification Based Testing with QuickCheck](https://reader031.fdocuments.net/reader031/viewer/2022012101/6169f47c11a7b741a34d387c/html5/thumbnails/4.jpg)
Properties as Code
A quantifier! A set!
A predicate!
A boolean-valued
expression!
A macro!
An ordinary function
definition!
A test data generator!
![Page 5: Specification Based Testing with QuickCheck](https://reader031.fdocuments.net/reader031/viewer/2022012101/6169f47c11a7b741a34d387c/html5/thumbnails/5.jpg)
DEMO
![Page 6: Specification Based Testing with QuickCheck](https://reader031.fdocuments.net/reader031/viewer/2022012101/6169f47c11a7b741a34d387c/html5/thumbnails/6.jpg)
QuickCheck in a Nutshell
Properties Test case Test case Test case Test case Test case
MinimalTest case
![Page 7: Specification Based Testing with QuickCheck](https://reader031.fdocuments.net/reader031/viewer/2022012101/6169f47c11a7b741a34d387c/html5/thumbnails/7.jpg)
QuickCheck Properties: things with a counterexample
<bool-exp>
?FORALL(<var>,<generator>,<property>)
?IMPLIES(<bool-exp>,<property>)
conjunction, disjunction
?EXISTS(<var>,<generator>,<property>)
![Page 8: Specification Based Testing with QuickCheck](https://reader031.fdocuments.net/reader031/viewer/2022012101/6169f47c11a7b741a34d387c/html5/thumbnails/8.jpg)
QuickCheck Generators
int(), bool(), real()…
choose(<int>,<int>)
{<generator>,<generator>…}
oneof(<list-of-generators>)
?LET(<var>,<generator>,<generator>)
![Page 9: Specification Based Testing with QuickCheck](https://reader031.fdocuments.net/reader031/viewer/2022012101/6169f47c11a7b741a34d387c/html5/thumbnails/9.jpg)
Example: Sorted Lists
sorted_list_int() -> ?LET(L,list(int()), sort(L)).
![Page 10: Specification Based Testing with QuickCheck](https://reader031.fdocuments.net/reader031/viewer/2022012101/6169f47c11a7b741a34d387c/html5/thumbnails/10.jpg)
Benefits
• Less time spent writing test code
– One property replaces many tests
• Better testing
– Lots of combinations you’d never test by hand
• Less time spent on diagnosis
– Failures minimized automagically
![Page 11: Specification Based Testing with QuickCheck](https://reader031.fdocuments.net/reader031/viewer/2022012101/6169f47c11a7b741a34d387c/html5/thumbnails/11.jpg)
An Experiment
Unit tests
Properties
![Page 12: Specification Based Testing with QuickCheck](https://reader031.fdocuments.net/reader031/viewer/2022012101/6169f47c11a7b741a34d387c/html5/thumbnails/12.jpg)
How good were the tests at finding bugs—in other students’ code?
0
1
2
3
4
5
6
1 2 3 4 5 6 7 8 9 10 11 12
Hunit
QuickCheck
Better
0 1 2 3 4 5 6 7 8 9 10 11
Unit tests
![Page 13: Specification Based Testing with QuickCheck](https://reader031.fdocuments.net/reader031/viewer/2022012101/6169f47c11a7b741a34d387c/html5/thumbnails/13.jpg)
Tests for Base 64 encoding
base64_encode(Config) when is_list(Config) ->
%% Two pads
<<"QWxhZGRpbjpvcGVuIHNlc2FtZQ==">> =
base64:encode("Aladdin:open sesame"),
%% One pad
<<"SGVsbG8gV29ybGQ=">> = base64:encode(<<"Hello World">>),
%% No pad
"QWxhZGRpbjpvcGVuIHNlc2Ft" =
base64:encode_to_string("Aladdin:open sesam"),
"MDEyMzQ1Njc4OSFAIzBeJiooKTs6PD4sLiBbXXt9" =
base64:encode_to_string(
<<"0123456789!@#0^&*();:<>,. []{}">>),
ok.
Test cases
Expected results
![Page 14: Specification Based Testing with QuickCheck](https://reader031.fdocuments.net/reader031/viewer/2022012101/6169f47c11a7b741a34d387c/html5/thumbnails/14.jpg)
Writing a Property
prop_base64() ->
?FORALL(Data,list(choose(0,255)),
base64:encode(Data) == ???).
![Page 15: Specification Based Testing with QuickCheck](https://reader031.fdocuments.net/reader031/viewer/2022012101/6169f47c11a7b741a34d387c/html5/thumbnails/15.jpg)
Round-trip Properties
prop_encode_decode() ->
?FORALL(L,list(choose(0,255)),
base64:decode(base64:encode(L))
== list_to_binary(L)).
-define(DECODE_MAP, {bad,bad,bad,bad,bad,bad,bad,bad,ws,ws,bad,bad,ws,bad,bad, bad,bad,bad,bad,bad,bad,bad,bad,bad,bad,bad,bad,bad,bad,bad,bad, ws,bad,bad,bad,bad,bad,bad,bad,bad,bad,bad,62,bad,bad,bad,63, 52,53,54,55,56,57,58,59,60,61,bad,bad,bad,eq,bad,bad, bad,0,1,2,3,4,5,6,7,8,9,10,11,12,13,14, 15,16,17,18,19,20,21,22,23,24,25,bad,bad,bad,bad,bad, bad,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40, 41,42,43,44,45,46,47,48,49,50,51,bad,bad,bad,bad,bad, bad,bad,bad,bad,bad,bad,bad,bad,bad,bad,bad,bad,bad,bad,bad,bad, bad,bad,bad,bad,bad,bad,bad,bad,bad,bad,bad,bad,bad,bad,bad,bad, bad,bad,bad,bad,bad,bad,bad,bad,bad,bad,bad,bad,bad,bad,bad,bad, bad,bad,bad,bad,bad,bad,bad,bad,bad,bad,bad,bad,bad,bad,bad,bad,
-define(DECODE_MAP, {bad,bad,bad,bad,bad,bad,bad,bad,ws,ws,bad,bad,ws,bad,bad, bad,bad,bad,bad,bad,bad,bad,bad,bad,bad,bad,bad,bad,bad,bad,bad, ws,bad,bad,bad,bad,bad,bad,bad,bad,bad,62,bad,bad,bad,bad,63, 52,53,54,55,56,57,58,59,60,61,bad,bad,bad,eq,bad,bad, bad,0,1,2,3,4,5,6,7,8,9,10,11,12,13,14, 15,16,17,18,19,20,21,22,23,24,25,bad,bad,bad,bad,bad, bad,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40, 41,42,43,44,45,46,47,48,49,50,51,bad,bad,bad,bad,bad, bad,bad,bad,bad,bad,bad,bad,bad,bad,bad,bad,bad,bad,bad,bad,bad, bad,bad,bad,bad,bad,bad,bad,bad,bad,bad,bad,bad,bad,bad,bad,bad, bad,bad,bad,bad,bad,bad,bad,bad,bad,bad,bad,bad,bad,bad,bad,bad, bad,bad,bad,bad,bad,bad,bad,bad,bad,bad,bad,bad,bad,bad,bad,bad,
NOT caught by the test suite
![Page 16: Specification Based Testing with QuickCheck](https://reader031.fdocuments.net/reader031/viewer/2022012101/6169f47c11a7b741a34d387c/html5/thumbnails/16.jpg)
Round-trip Properties
117> eqc:quickcheck(base64_eqc:prop_encode_decode()).
...................................Failed! Reason: {'EXIT',{badarg,43}}
After 36 tests.
[204,15,130]
Shrinking...(3 times)
Reason: {'EXIT',{badarg,43}}
[0,0,62]
prop_encode_decode() ->
?FORALL(L,list(choose(0,255)),
base64:decode(base64:encode(L))
== list_to_binary(L)).
The table entry we changed
![Page 17: Specification Based Testing with QuickCheck](https://reader031.fdocuments.net/reader031/viewer/2022012101/6169f47c11a7b741a34d387c/html5/thumbnails/17.jpg)
Round-trip Properties
What does this test?
• NOT a complete test—will not find a consistent misunderstanding of base64
• WILL find mistakes in encoder or decoder
Simple properties find a lot of bugs!
prop_encode_decode() ->
?FORALL(L,list(choose(0,255)),
base64:decode(base64:encode(L))
== list_to_binary(L)).
![Page 18: Specification Based Testing with QuickCheck](https://reader031.fdocuments.net/reader031/viewer/2022012101/6169f47c11a7b741a34d387c/html5/thumbnails/18.jpg)
Back to the tests…
base64_encode(Config) when is_list(Config) ->
%% Two pads
<<"QWxhZGRpbjpvcGVuIHNlc2FtZQ==">> =
base64:encode("Aladdin:open sesame"),
%% One pad
<<"SGVsbG8gV29ybGQ=">> = base64:encode(<<"Hello World">>),
%% No pad
"QWxhZGRpbjpvcGVuIHNlc2Ft" =
base64:encode_to_string("Aladdin:open sesam"),
"MDEyMzQ1Njc4OSFAIzBeJiooKTs6PD4sLiBbXXt9" =
base64:encode_to_string(
<<"0123456789!@#0^&*();:<>,. []{}">>),
ok.
Where did these come
from?
![Page 19: Specification Based Testing with QuickCheck](https://reader031.fdocuments.net/reader031/viewer/2022012101/6169f47c11a7b741a34d387c/html5/thumbnails/19.jpg)
Possibilities
• Someone converted the data by hand
• Another base64 encoder
• The same base64 encoder!
– Only tests that changes don’t affect the result, not that the result is right
Use the other encoder as an
oracle
Use an old version (or a
simpler version) as an oracle
![Page 20: Specification Based Testing with QuickCheck](https://reader031.fdocuments.net/reader031/viewer/2022012101/6169f47c11a7b741a34d387c/html5/thumbnails/20.jpg)
Commuting Diagram Properties
![Page 21: Specification Based Testing with QuickCheck](https://reader031.fdocuments.net/reader031/viewer/2022012101/6169f47c11a7b741a34d387c/html5/thumbnails/21.jpg)
Property Types in Class Examples
• Rex Page: 71 properties in University of Oklahoma courses in Software Engineering, Applied Logic (QuickCheck+ACL2)
Round trip
Commuting diagram
Other
![Page 22: Specification Based Testing with QuickCheck](https://reader031.fdocuments.net/reader031/viewer/2022012101/6169f47c11a7b741a34d387c/html5/thumbnails/22.jpg)
Time for some C code…
![Page 23: Specification Based Testing with QuickCheck](https://reader031.fdocuments.net/reader031/viewer/2022012101/6169f47c11a7b741a34d387c/html5/thumbnails/23.jpg)
Testing Stateful Code
API
Calls API
Calls API
Calls API
Calls
Model state
Model state
Model state
Model state
postconditions
A list of numbers!
![Page 24: Specification Based Testing with QuickCheck](https://reader031.fdocuments.net/reader031/viewer/2022012101/6169f47c11a7b741a34d387c/html5/thumbnails/24.jpg)
A QuickCheck Property
prop_q() ->
?FORALL(Cmds,commands(?MODULE),
begin
{H,S,Res} = run_commands(?MODULE,Cmds),
Res == ok)
end).
![Page 25: Specification Based Testing with QuickCheck](https://reader031.fdocuments.net/reader031/viewer/2022012101/6169f47c11a7b741a34d387c/html5/thumbnails/25.jpg)
Let’s run some tests…
![Page 26: Specification Based Testing with QuickCheck](https://reader031.fdocuments.net/reader031/viewer/2022012101/6169f47c11a7b741a34d387c/html5/thumbnails/26.jpg)
Exercises Practice
Small scale
Property-driven development
Trivial inputs
Large scale Testing legacy code Complex inputs
![Page 27: Specification Based Testing with QuickCheck](https://reader031.fdocuments.net/reader031/viewer/2022012101/6169f47c11a7b741a34d387c/html5/thumbnails/27.jpg)
Example: Ericsson Media Proxy
Megaco request Megaco
response Megaco request Megaco
response
Many, many parameters, can be 1—2 pages per message!
Lots of work to write
generators
State machine models fit the problem well
![Page 28: Specification Based Testing with QuickCheck](https://reader031.fdocuments.net/reader031/viewer/2022012101/6169f47c11a7b741a34d387c/html5/thumbnails/28.jpg)
Ericsson Media Proxy Bug
• Test adding and removing callers from a call
Add Add Sub Add Sub Add Sub
Call Full
![Page 29: Specification Based Testing with QuickCheck](https://reader031.fdocuments.net/reader031/viewer/2022012101/6169f47c11a7b741a34d387c/html5/thumbnails/29.jpg)
• Relational databases don’t scale to ”Big Data”
• ”noSQL” databases are a popular alternative
A highly scalable, reliable, available and low-latency distributed key-
value store
![Page 30: Specification Based Testing with QuickCheck](https://reader031.fdocuments.net/reader031/viewer/2022012101/6169f47c11a7b741a34d387c/html5/thumbnails/30.jpg)
Put and Get
0 put
get 0 1
put
get 1
![Page 31: Specification Based Testing with QuickCheck](https://reader031.fdocuments.net/reader031/viewer/2022012101/6169f47c11a7b741a34d387c/html5/thumbnails/31.jpg)
Conflicts
0 put
1 put
get {0,1}
2 put
2 get
QuickCheck model: record each client’s current view of the data; put
replaces that view
![Page 32: Specification Based Testing with QuickCheck](https://reader031.fdocuments.net/reader031/viewer/2022012101/6169f47c11a7b741a34d387c/html5/thumbnails/32.jpg)
Example
0 put
1 put
{0,1} get
2 put
3 put
??? get
{0,1,2,3} get
A vector clock optimisation…
QuickCheck model: client’s view is fresh or stale: updating a stale view
just adds to the conflicts…
![Page 33: Specification Based Testing with QuickCheck](https://reader031.fdocuments.net/reader031/viewer/2022012101/6169f47c11a7b741a34d387c/html5/thumbnails/33.jpg)
Example
0 put
get ??
get 0
get {0,0}
![Page 34: Specification Based Testing with QuickCheck](https://reader031.fdocuments.net/reader031/viewer/2022012101/6169f47c11a7b741a34d387c/html5/thumbnails/34.jpg)
Duplicate value explained
0 put
0
0
0
12:43:27
12:43:27
12:43:27
get 0
12:43:28
get {0,0}
![Page 35: Specification Based Testing with QuickCheck](https://reader031.fdocuments.net/reader031/viewer/2022012101/6169f47c11a7b741a34d387c/html5/thumbnails/35.jpg)
Eventual Consistency
• ”For any sequence of operations, with any node or network failures, Riak eventually reaches a consistent state”
– When is ”eventually”?
• For any sequence of operations sent to any subsets of server nodes (because of failures), completing all Riak’s repair operations results in a consistent state.
![Page 36: Specification Based Testing with QuickCheck](https://reader031.fdocuments.net/reader031/viewer/2022012101/6169f47c11a7b741a34d387c/html5/thumbnails/36.jpg)
AutoSAR
• Joint project with Quviq, SP, Volvo Cars, Mentor Graphics…
![Page 37: Specification Based Testing with QuickCheck](https://reader031.fdocuments.net/reader031/viewer/2022012101/6169f47c11a7b741a34d387c/html5/thumbnails/37.jpg)
![Page 38: Specification Based Testing with QuickCheck](https://reader031.fdocuments.net/reader031/viewer/2022012101/6169f47c11a7b741a34d387c/html5/thumbnails/38.jpg)
AutoSAR Basic Software
![Page 39: Specification Based Testing with QuickCheck](https://reader031.fdocuments.net/reader031/viewer/2022012101/6169f47c11a7b741a34d387c/html5/thumbnails/39.jpg)
The Story So Far…
• QuickCheck state-machine models for 3 AutoSAR clusters (Com/PDUR, CAN, FlexRay)
• Used to test software from 3 suppliers
• Bugs revealed in all!
– Plus reinterpretations of the standard
![Page 40: Specification Based Testing with QuickCheck](https://reader031.fdocuments.net/reader031/viewer/2022012101/6169f47c11a7b741a34d387c/html5/thumbnails/40.jpg)
![Page 41: Specification Based Testing with QuickCheck](https://reader031.fdocuments.net/reader031/viewer/2022012101/6169f47c11a7b741a34d387c/html5/thumbnails/41.jpg)
uint32
![Page 42: Specification Based Testing with QuickCheck](https://reader031.fdocuments.net/reader031/viewer/2022012101/6169f47c11a7b741a34d387c/html5/thumbnails/42.jpg)
COM Component
• 500 pages of standard
• 250 pages of C
• 25 pages of QuickCheck
![Page 43: Specification Based Testing with QuickCheck](https://reader031.fdocuments.net/reader031/viewer/2022012101/6169f47c11a7b741a34d387c/html5/thumbnails/43.jpg)
"We know there is a lurking bug somewhere in the dets code. We have got 'bad object' and 'premature eof' every other month the last year. We have not been able to track the bug down since the dets files is repaired automatically next time it is opened.“
Tobbe Törnqvist, Klarna, 2007
![Page 44: Specification Based Testing with QuickCheck](https://reader031.fdocuments.net/reader031/viewer/2022012101/6169f47c11a7b741a34d387c/html5/thumbnails/44.jpg)
What is it?
Application
Mnesia
Dets
File system
Invoicing services for web shops
Distributed database: transactions, distribution, replication
Tuple storage
>500 people in
5 years
Race conditions?
![Page 45: Specification Based Testing with QuickCheck](https://reader031.fdocuments.net/reader031/viewer/2022012101/6169f47c11a7b741a34d387c/html5/thumbnails/45.jpg)
Imagine Testing This…
dispenser:take_ticket() dispenser:reset()
![Page 46: Specification Based Testing with QuickCheck](https://reader031.fdocuments.net/reader031/viewer/2022012101/6169f47c11a7b741a34d387c/html5/thumbnails/46.jpg)
A Unit Test in Erlang
test_dispenser() ->
reset(),
take_ticket(),
take_ticket(),
take_ticket(),
reset(),
take_ticket().
ok =
1 =
2 =
3 =
ok =
1 =
Expected results
BUT…
![Page 47: Specification Based Testing with QuickCheck](https://reader031.fdocuments.net/reader031/viewer/2022012101/6169f47c11a7b741a34d387c/html5/thumbnails/47.jpg)
A Parallel Unit Test
• Three possible correct outcomes!
reset
take_ticket
take_ticket
take_ticket
1 2
3
1 3
2
1 2
1
ok
![Page 48: Specification Based Testing with QuickCheck](https://reader031.fdocuments.net/reader031/viewer/2022012101/6169f47c11a7b741a34d387c/html5/thumbnails/48.jpg)
Another Parallel Test
• 42 possible correct outcomes!
reset
take_ticket
take_ticket
take_ticket
take_ticket
reset
A killer app for
properties!
![Page 49: Specification Based Testing with QuickCheck](https://reader031.fdocuments.net/reader031/viewer/2022012101/6169f47c11a7b741a34d387c/html5/thumbnails/49.jpg)
Modelling the dispenser
reset take take take
0 0 1 2
ok 1 2 3
![Page 50: Specification Based Testing with QuickCheck](https://reader031.fdocuments.net/reader031/viewer/2022012101/6169f47c11a7b741a34d387c/html5/thumbnails/50.jpg)
The Model
• State transitions
• Postconditions
next_state(S,_V,{call,_,reset,_}) ->
0;
next_state(S,_V,{call,_,take_ticket,_}) ->
S+1.
postcondition(S,{call,_,take_ticket,_},Res) ->
Res == S+1;
![Page 51: Specification Based Testing with QuickCheck](https://reader031.fdocuments.net/reader031/viewer/2022012101/6169f47c11a7b741a34d387c/html5/thumbnails/51.jpg)
Parallel Test Cases
resetok
take1
take3
take2
0 0 1 2
ok 1 2 3
![Page 52: Specification Based Testing with QuickCheck](https://reader031.fdocuments.net/reader031/viewer/2022012101/6169f47c11a7b741a34d387c/html5/thumbnails/52.jpg)
prop_parallel() ->
?FORALL(Cmds,parallel_commands(?MODULE),
begin
start(),
{H,Par,Res} =
run_parallel_commands(?MODULE,Cmds),
Res == ok)
end)).
Generate parallel test cases
Run tests, check for a matching serialization
![Page 53: Specification Based Testing with QuickCheck](https://reader031.fdocuments.net/reader031/viewer/2022012101/6169f47c11a7b741a34d387c/html5/thumbnails/53.jpg)
DEMO
![Page 54: Specification Based Testing with QuickCheck](https://reader031.fdocuments.net/reader031/viewer/2022012101/6169f47c11a7b741a34d387c/html5/thumbnails/54.jpg)
Prefix: Parallel: 1. take_ticket() --> 1 2. take_ticket() --> 1 Result: no_possible_interleaving
take_ticket() -> N = read(), write(N+1), N+1.
![Page 55: Specification Based Testing with QuickCheck](https://reader031.fdocuments.net/reader031/viewer/2022012101/6169f47c11a7b741a34d387c/html5/thumbnails/55.jpg)
dets
• Tuple store:
{Key, Value1, Value2…}
• Operations:
– insert(Table,ListOfTuples)
– delete(Table,Key)
– insert_new(Table,ListOfTuples)
– …
• Model:
– List of tuples (almost)
![Page 56: Specification Based Testing with QuickCheck](https://reader031.fdocuments.net/reader031/viewer/2022012101/6169f47c11a7b741a34d387c/html5/thumbnails/56.jpg)
QuickCheck Specification
... …
... … <100 LOC
> 6,000 LOC
![Page 57: Specification Based Testing with QuickCheck](https://reader031.fdocuments.net/reader031/viewer/2022012101/6169f47c11a7b741a34d387c/html5/thumbnails/57.jpg)
DEMO
• Sequential tests to validate the model
• Parallel tests to find race conditions
![Page 58: Specification Based Testing with QuickCheck](https://reader031.fdocuments.net/reader031/viewer/2022012101/6169f47c11a7b741a34d387c/html5/thumbnails/58.jpg)
Bug #1
Prefix:
open_file(dets_table,[{type,bag}]) -->
dets_table
Parallel:
1. insert(dets_table,[]) --> ok
2. insert_new(dets_table,[]) --> ok
Result: no_possible_interleaving
insert_new(Name, Objects) -> Bool Types: Name = name() Objects = object() | [object()] Bool = bool()
![Page 59: Specification Based Testing with QuickCheck](https://reader031.fdocuments.net/reader031/viewer/2022012101/6169f47c11a7b741a34d387c/html5/thumbnails/59.jpg)
Bug #2
Prefix:
open_file(dets_table,[{type,set}]) --> dets_table
Parallel:
1. insert(dets_table,{0,0}) --> ok
2. insert_new(dets_table,{0,0}) --> …time out…
=ERROR REPORT==== 4-Oct-2010::17:08:21 === ** dets: Bug was found when accessing table dets_table
![Page 60: Specification Based Testing with QuickCheck](https://reader031.fdocuments.net/reader031/viewer/2022012101/6169f47c11a7b741a34d387c/html5/thumbnails/60.jpg)
Bug #3
Prefix:
open_file(dets_table,[{type,set}]) --> dets_table
Parallel:
1. open_file(dets_table,[{type,set}]) --> dets_table
2. insert(dets_table,{0,0}) --> ok
get_contents(dets_table) --> []
Result: no_possible_interleaving !
![Page 61: Specification Based Testing with QuickCheck](https://reader031.fdocuments.net/reader031/viewer/2022012101/6169f47c11a7b741a34d387c/html5/thumbnails/61.jpg)
What’s going on?
Dets server
Reordering and concurrency!
![Page 62: Specification Based Testing with QuickCheck](https://reader031.fdocuments.net/reader031/viewer/2022012101/6169f47c11a7b741a34d387c/html5/thumbnails/62.jpg)
Is the file corrupt?
![Page 63: Specification Based Testing with QuickCheck](https://reader031.fdocuments.net/reader031/viewer/2022012101/6169f47c11a7b741a34d387c/html5/thumbnails/63.jpg)
Bug #4
Prefix:
open_file(dets_table,[{type,bag}]) --> dets_table
close(dets_table) --> ok
open_file(dets_table,[{type,bag}]) --> dets_table
Parallel:
1. lookup(dets_table,0) --> []
2. insert(dets_table,{0,0}) --> ok
3. insert(dets_table,{0,0}) --> ok
Result: ok
premature eof
![Page 64: Specification Based Testing with QuickCheck](https://reader031.fdocuments.net/reader031/viewer/2022012101/6169f47c11a7b741a34d387c/html5/thumbnails/64.jpg)
Bug #5
Prefix:
open_file(dets_table,[{type,set}]) --> dets_table
insert(dets_table,[{1,0}]) --> ok
Parallel:
1. lookup(dets_table,0) --> []
delete(dets_table,1) --> ok
2. open_file(dets_table,[{type,set}]) --> dets_table
Result: ok
false
bad object
![Page 65: Specification Based Testing with QuickCheck](https://reader031.fdocuments.net/reader031/viewer/2022012101/6169f47c11a7b741a34d387c/html5/thumbnails/65.jpg)
"We know there is a lurking bug somewhere in the dets code. We have got 'bad object' and 'premature eof' every other month the last year.”
Tobbe Törnqvist, Klarna, 2007 Each bug fixed the day
after reporting the failing case
![Page 66: Specification Based Testing with QuickCheck](https://reader031.fdocuments.net/reader031/viewer/2022012101/6169f47c11a7b741a34d387c/html5/thumbnails/66.jpg)
How come? • The bugs weren’t found earlier?
– despite > 6 weeks of work
• Hypotheses – …files of over 1GB?
– …rehashing could be the problem?
– Diagnosing races in production is hopeless
• The bugs weren’t found in testing? – Unit tests for races are hard to write…so people
don’t!
– Races=feature interaction impractically many tests
![Page 67: Specification Based Testing with QuickCheck](https://reader031.fdocuments.net/reader031/viewer/2022012101/6169f47c11a7b741a34d387c/html5/thumbnails/67.jpg)
Race conditions should be found by unit testing with
generated tests
![Page 68: Specification Based Testing with QuickCheck](https://reader031.fdocuments.net/reader031/viewer/2022012101/6169f47c11a7b741a34d387c/html5/thumbnails/68.jpg)
Reflections
![Page 69: Specification Based Testing with QuickCheck](https://reader031.fdocuments.net/reader031/viewer/2022012101/6169f47c11a7b741a34d387c/html5/thumbnails/69.jpg)
The Initial Phases
• Lots of work to develop specification
– Understanding and generating test inputs
• Many errors to fix in the specification, due to…
– New code is buggy
– Misunderstandings of the informal spec
– Undocumented features of the system
– Undocumented limitations of the system
• ”happy case” programming
![Page 70: Specification Based Testing with QuickCheck](https://reader031.fdocuments.net/reader031/viewer/2022012101/6169f47c11a7b741a34d387c/html5/thumbnails/70.jpg)
Making Progress
• QuickCheck tends to find the same problem in every run
– There is a ”most likely bug”
– Other bugs usually shrink to the most likely one
• To make progress, the most likely bug must be excluded
– Bug preconditions document the limitations of the system
![Page 71: Specification Based Testing with QuickCheck](https://reader031.fdocuments.net/reader031/viewer/2022012101/6169f47c11a7b741a34d387c/html5/thumbnails/71.jpg)
The Payoff
• Once the spec is corrected, and limitations accounted for, real bugs start to appear
• Each extension to the spec yields a non-linear improvement in the variety of tests
• The same spec can find many, many bugs
![Page 72: Specification Based Testing with QuickCheck](https://reader031.fdocuments.net/reader031/viewer/2022012101/6169f47c11a7b741a34d387c/html5/thumbnails/72.jpg)
QuickCheck…
• …is very widely applicable
• …almost always finds bugs in real systems!
• …is particularly good at spotting interactions that conventional test cases miss
• …makes diagnosis simple by shrinking
• …makes testing more intellectually challenging and fun!!