Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3...
Transcript of Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3...
![Page 1: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/1.jpg)
Test Driven Development of Scientific Models
Tom Clune
Software Systems Support OfficeEarth Science Division
NASA Goddard Space Flight Center
May 1, 2012
Tom Clune (SSSO) TDD - NCAR May 1, 2012 1 / 61
https://ntrs.nasa.gov/search.jsp?R=20120009963 2020-04-27T17:33:24+00:00Z
![Page 2: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/2.jpg)
Outline
1 Introduction
2 Testing
3 Testing Frameworks
4 Test-Driven Development
5 TDD and Scientific/Technical Software
6 Example
7 pFUnit
Tom Clune (SSSO) TDD - Introduction - NCAR May 1, 2012 2 / 61
![Page 3: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/3.jpg)
The Tightrope Act
Software development should not feel like this
Tom Clune (SSSO) TDD - Introduction - NCAR May 1, 2012 3 / 61
![Page 4: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/4.jpg)
The Tightrope Act
... or even like this
Tom Clune (SSSO) TDD - Introduction - NCAR May 1, 2012 3 / 61
![Page 5: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/5.jpg)
The Tightrope Act
Hopefully something more like this
Tom Clune (SSSO) TDD - Introduction - NCAR May 1, 2012 3 / 61
![Page 6: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/6.jpg)
The Development Cycle
Extend
Fix
Port
Compiles?
Executes?
Looks ok?
Correct?
Tom Clune (SSSO) TDD - Introduction - NCAR May 1, 2012 4 / 61
![Page 7: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/7.jpg)
The Development Cycle
Extend
Fix
Port
Compiles?
Executes?
Looks ok?
Correct?
Tom Clune (SSSO) TDD - Introduction - NCAR May 1, 2012 4 / 61
![Page 8: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/8.jpg)
The Development Cycle
Extend
Fix
Port
Compiles?
Executes?
Looks ok?
Correct?
Tom Clune (SSSO) TDD - Introduction - NCAR May 1, 2012 4 / 61
![Page 9: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/9.jpg)
The Development Cycle
Extend
Fix
Port
Compiles?
Executes?
Looks ok?
Correct?
Tom Clune (SSSO) TDD - Introduction - NCAR May 1, 2012 4 / 61
![Page 10: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/10.jpg)
The Development Cycle
Extend
Fix
Port
Compiles?
Executes?
Looks ok?
Correct?
Tom Clune (SSSO) TDD - Introduction - NCAR May 1, 2012 4 / 61
![Page 11: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/11.jpg)
The Development Cycle
Extend
Fix
Port
Compiles?
Executes?
Looks ok?
Correct?
Tom Clune (SSSO) TDD - Introduction - NCAR May 1, 2012 4 / 61
![Page 12: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/12.jpg)
The Development Cycle
Extend
Fix
Port
Compiles?
Executes?
Looks ok?
Correct?
Tom Clune (SSSO) TDD - Introduction - NCAR May 1, 2012 4 / 61
![Page 13: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/13.jpg)
The Development Cycle
Extend
Fix
Port
Compiles?
Executes?
Looks ok?
Correct?
Tom Clune (SSSO) TDD - Introduction - NCAR May 1, 2012 4 / 61
![Page 14: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/14.jpg)
Natural Time Scales
DesignEdit sourceCompilationBatchwaiting in queueExecutionAnalysis
Tom Clune (SSSO) TDD - Introduction - NCAR May 1, 2012 5 / 61
![Page 15: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/15.jpg)
Some observations
Risk grows with magnitude of implementation step
Magnitude of implementation step grows with cost ofverification/validation
Conclusion:Optimize productivity by reducing cost of verification!
Tom Clune (SSSO) TDD - Introduction - NCAR May 1, 2012 6 / 61
![Page 16: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/16.jpg)
Some observations
Risk grows with magnitude of implementation step
Magnitude of implementation step grows with cost ofverification/validation
Conclusion:Optimize productivity by reducing cost of verification!
Tom Clune (SSSO) TDD - Introduction - NCAR May 1, 2012 6 / 61
![Page 17: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/17.jpg)
Outline
1 Introduction
2 Testing
3 Testing Frameworks
4 Test-Driven Development
5 TDD and Scientific/Technical Software
6 Example
7 pFUnit
Tom Clune (SSSO) TDD - Testing - NCAR May 1, 2012 7 / 61
![Page 18: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/18.jpg)
Testing
Tom Clune (SSSO) TDD - Testing - NCAR May 1, 2012 8 / 61
![Page 19: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/19.jpg)
Test Harness - work in safety
Collection of tests that constrain system
Detects unintended changes
Localizes defects
Improves developer confidence
Decreases risk from change
Tom Clune (SSSO) TDD - Testing - NCAR May 1, 2012 9 / 61
![Page 20: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/20.jpg)
Test Harness - work in safety
Collection of tests that constrain system
Detects unintended changes
Localizes defects
Improves developer confidence
Decreases risk from change
Tom Clune (SSSO) TDD - Testing - NCAR May 1, 2012 9 / 61
![Page 21: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/21.jpg)
Test Harness - work in safety
Collection of tests that constrain system
Detects unintended changes
Localizes defects
Improves developer confidence
Decreases risk from change
Tom Clune (SSSO) TDD - Testing - NCAR May 1, 2012 9 / 61
![Page 22: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/22.jpg)
Test Harness - work in safety
Collection of tests that constrain system
Detects unintended changes
Localizes defects
Improves developer confidence
Decreases risk from change
Tom Clune (SSSO) TDD - Testing - NCAR May 1, 2012 9 / 61
![Page 23: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/23.jpg)
Test Harness - work in safety
Collection of tests that constrain system
Detects unintended changes
Localizes defects
Improves developer confidence
Decreases risk from change
Tom Clune (SSSO) TDD - Testing - NCAR May 1, 2012 9 / 61
![Page 24: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/24.jpg)
Do you write legacy code?
“The main thing that distinguishes legacy code from non-legacy code istests, or rather a lack of tests.”
Michael FeathersWorking Effectively with Legacy Code
Lack of tests leads to fear of introducingsubtle bugs and/or changing thingsinadvertently.
Programming on a tightrope
This is also a barrier to involving pure
software engineers in the development of
our models.
Tom Clune (SSSO) TDD - Testing - NCAR May 1, 2012 10 / 61
![Page 25: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/25.jpg)
Do you write legacy code?
“The main thing that distinguishes legacy code from non-legacy code istests, or rather a lack of tests.”
Michael FeathersWorking Effectively with Legacy Code
Lack of tests leads to fear of introducingsubtle bugs and/or changing thingsinadvertently.
Programming on a tightrope
This is also a barrier to involving pure
software engineers in the development of
our models.
Tom Clune (SSSO) TDD - Testing - NCAR May 1, 2012 10 / 61
![Page 26: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/26.jpg)
Do you write legacy code?
“The main thing that distinguishes legacy code from non-legacy code istests, or rather a lack of tests.”
Michael FeathersWorking Effectively with Legacy Code
Lack of tests leads to fear of introducingsubtle bugs and/or changing thingsinadvertently.
Programming on a tightrope
This is also a barrier to involving pure
software engineers in the development of
our models.
Tom Clune (SSSO) TDD - Testing - NCAR May 1, 2012 10 / 61
![Page 27: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/27.jpg)
Do you write legacy code?
“The main thing that distinguishes legacy code from non-legacy code istests, or rather a lack of tests.”
Michael FeathersWorking Effectively with Legacy Code
Lack of tests leads to fear of introducingsubtle bugs and/or changing thingsinadvertently.
Programming on a tightrope
This is also a barrier to involving pure
software engineers in the development of
our models.
Tom Clune (SSSO) TDD - Testing - NCAR May 1, 2012 10 / 61
![Page 28: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/28.jpg)
Excuses, excuses ...
Takes too much time to write tests
Too difficult to maintain tests
It takes too long to run the tests
It is not my job
“Correct” behavior is unknown
http://java.dzone.com/articles/unit-test-excuses
- James Sugrue
Tom Clune (SSSO) TDD - Testing - NCAR May 1, 2012 11 / 61
![Page 29: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/29.jpg)
Excuses, excuses ...
Takes too much time to write tests
Too difficult to maintain tests
It takes too long to run the tests
It is not my job
“Correct” behavior is unknown
http://java.dzone.com/articles/unit-test-excuses
- James Sugrue
Tom Clune (SSSO) TDD - Testing - NCAR May 1, 2012 11 / 61
![Page 30: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/30.jpg)
Excuses, excuses ...
Takes too much time to write tests
Too difficult to maintain tests
It takes too long to run the tests
It is not my job
“Correct” behavior is unknown
http://java.dzone.com/articles/unit-test-excuses
- James Sugrue
Tom Clune (SSSO) TDD - Testing - NCAR May 1, 2012 11 / 61
![Page 31: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/31.jpg)
Excuses, excuses ...
Takes too much time to write tests
Too difficult to maintain tests
It takes too long to run the tests
It is not my job
“Correct” behavior is unknown
http://java.dzone.com/articles/unit-test-excuses
- James Sugrue
Tom Clune (SSSO) TDD - Testing - NCAR May 1, 2012 11 / 61
![Page 32: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/32.jpg)
Excuses, excuses ...
Takes too much time to write tests
Too difficult to maintain tests
It takes too long to run the tests
It is not my job
“Correct” behavior is unknown
http://java.dzone.com/articles/unit-test-excuses
- James Sugrue
Tom Clune (SSSO) TDD - Testing - NCAR May 1, 2012 11 / 61
![Page 33: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/33.jpg)
Excuses, excuses ...
Takes too much time to write tests
Too difficult to maintain tests
It takes too long to run the tests
It is not my job
“Correct” behavior is unknown
http://java.dzone.com/articles/unit-test-excuses
- James Sugrue
Tom Clune (SSSO) TDD - Testing - NCAR May 1, 2012 11 / 61
![Page 34: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/34.jpg)
Just what is a test anyway?
Tests can exist in many forms
Conditional termination:IF (PA( I , J)+PTOP.GT. 1 2 0 0 . ) &
c a l l s t o p m o d e l ( ’ADVECM: P r e s s u r e d i a g n o s t i c e r r o r ’ , 1 1 )
Diagnostic print statementp r i n t ∗ , ’ l o s s o f mass = ’ , d e l t a M a s s
Visualization of outputTemp1
5 10 15
10
20
30
40
50
60
Temp2
5 10 15
10
20
30
40
50
60
Difference
5 10 15
10
20
30
40
50
60
Student Version of MATLAB
Tom Clune (SSSO) TDD - Testing - NCAR May 1, 2012 13 / 61
![Page 35: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/35.jpg)
Analogy with Scientific Method?
Reality −→ RequirementsConstraints: theory and data −→ Constraints: testsFormulate hypothesis −→ Trial implementationPerform experiment −→ Run testsRefine hypothesis −→ Refine implementation
Tom Clune (SSSO) TDD - Testing - NCAR May 1, 2012 14 / 61
![Page 36: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/36.jpg)
Properties of good tests
Isolating
I Test failure indicates location in source code
Orthogonal
I Each defect results in failure of small number of tests
Complete
I Each bit of functionality covered by at least one test
Independent
I No side effectsI Test order does not matterI Corollary: cannot terminate execution
Frugal
I Run quicklyI Small memory, etc.
Automated and repeatableClear intent
Tom Clune (SSSO) TDD - Testing - NCAR May 1, 2012 15 / 61
![Page 37: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/37.jpg)
Properties of good tests
IsolatingI Test failure indicates location in source code
Orthogonal
I Each defect results in failure of small number of tests
Complete
I Each bit of functionality covered by at least one test
Independent
I No side effectsI Test order does not matterI Corollary: cannot terminate execution
Frugal
I Run quicklyI Small memory, etc.
Automated and repeatableClear intent
Tom Clune (SSSO) TDD - Testing - NCAR May 1, 2012 15 / 61
![Page 38: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/38.jpg)
Properties of good tests
IsolatingI Test failure indicates location in source code
OrthogonalI Each defect results in failure of small number of tests
Complete
I Each bit of functionality covered by at least one test
Independent
I No side effectsI Test order does not matterI Corollary: cannot terminate execution
Frugal
I Run quicklyI Small memory, etc.
Automated and repeatableClear intent
Tom Clune (SSSO) TDD - Testing - NCAR May 1, 2012 15 / 61
![Page 39: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/39.jpg)
Properties of good tests
IsolatingI Test failure indicates location in source code
OrthogonalI Each defect results in failure of small number of tests
CompleteI Each bit of functionality covered by at least one test
Independent
I No side effectsI Test order does not matterI Corollary: cannot terminate execution
Frugal
I Run quicklyI Small memory, etc.
Automated and repeatableClear intent
Tom Clune (SSSO) TDD - Testing - NCAR May 1, 2012 15 / 61
![Page 40: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/40.jpg)
Properties of good tests
IsolatingI Test failure indicates location in source code
OrthogonalI Each defect results in failure of small number of tests
CompleteI Each bit of functionality covered by at least one test
IndependentI No side effectsI Test order does not matterI Corollary: cannot terminate execution
Frugal
I Run quicklyI Small memory, etc.
Automated and repeatableClear intent
Tom Clune (SSSO) TDD - Testing - NCAR May 1, 2012 15 / 61
![Page 41: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/41.jpg)
Properties of good tests
IsolatingI Test failure indicates location in source code
OrthogonalI Each defect results in failure of small number of tests
CompleteI Each bit of functionality covered by at least one test
IndependentI No side effectsI Test order does not matterI Corollary: cannot terminate execution
FrugalI Run quicklyI Small memory, etc.
Automated and repeatableClear intent
Tom Clune (SSSO) TDD - Testing - NCAR May 1, 2012 15 / 61
![Page 42: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/42.jpg)
Properties of good tests
IsolatingI Test failure indicates location in source code
OrthogonalI Each defect results in failure of small number of tests
CompleteI Each bit of functionality covered by at least one test
IndependentI No side effectsI Test order does not matterI Corollary: cannot terminate execution
FrugalI Run quicklyI Small memory, etc.
Automated and repeatable
Clear intent
Tom Clune (SSSO) TDD - Testing - NCAR May 1, 2012 15 / 61
![Page 43: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/43.jpg)
Properties of good tests
IsolatingI Test failure indicates location in source code
OrthogonalI Each defect results in failure of small number of tests
CompleteI Each bit of functionality covered by at least one test
IndependentI No side effectsI Test order does not matterI Corollary: cannot terminate execution
FrugalI Run quicklyI Small memory, etc.
Automated and repeatableClear intent
Tom Clune (SSSO) TDD - Testing - NCAR May 1, 2012 15 / 61
![Page 44: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/44.jpg)
Anatomy of a Software Test Procedure
testTrajectory() ! s = 12 at2
a = 2.; t = 3.
s = trajectory(a, t)
call assertEqual (9., s)call assertEqual (9., trajectory (2.,3.))
! no op
Tom Clune (SSSO) TDD - Testing - NCAR May 1, 2012 17 / 61
![Page 45: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/45.jpg)
Anatomy of a Software Test Procedure
testTrajectory() ! s = 12 at2
a = 2.; t = 3.
s = trajectory(a, t)
call assertEqual (9., s)call assertEqual (9., trajectory (2.,3.))
! no op
Tom Clune (SSSO) TDD - Testing - NCAR May 1, 2012 17 / 61
![Page 46: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/46.jpg)
Anatomy of a Software Test Procedure
testTrajectory() ! s = 12 at2
a = 2.; t = 3.
s = trajectory(a, t)
call assertEqual (9., s)call assertEqual (9., trajectory (2.,3.))
! no op
Tom Clune (SSSO) TDD - Testing - NCAR May 1, 2012 17 / 61
![Page 47: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/47.jpg)
Anatomy of a Software Test Procedure
testTrajectory() ! s = 12 at2
a = 2.; t = 3.
s = trajectory(a, t)
call assertEqual (9., s)call assertEqual (9., trajectory (2.,3.))
! no op
Tom Clune (SSSO) TDD - Testing - NCAR May 1, 2012 17 / 61
![Page 48: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/48.jpg)
Anatomy of a Software Test Procedure
testTrajectory() ! s = 12 at2
a = 2.; t = 3.
s = trajectory(a, t)
call assertEqual (9., s)
call assertEqual (9., trajectory (2.,3.))
! no op
Tom Clune (SSSO) TDD - Testing - NCAR May 1, 2012 17 / 61
![Page 49: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/49.jpg)
Anatomy of a Software Test Procedure
testTrajectory() ! s = 12 at2
a = 2.; t = 3.
s = trajectory(a, t)
call assertEqual (9., s)
call assertEqual (9., trajectory (2.,3.))
! no op
Tom Clune (SSSO) TDD - Testing - NCAR May 1, 2012 17 / 61
![Page 50: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/50.jpg)
Anatomy of a Software Test Procedure
testTrajectory() ! s = 12 at2
a = 2.; t = 3.
s = trajectory(a, t)
call assertEqual (9., s)
call assertEqual (9., trajectory (2.,3.))
! no op
Tom Clune (SSSO) TDD - Testing - NCAR May 1, 2012 17 / 61
![Page 51: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/51.jpg)
Outline
1 Introduction
2 Testing
3 Testing Frameworks
4 Test-Driven Development
5 TDD and Scientific/Technical Software
6 Example
7 pFUnit
Tom Clune (SSSO) TDD - Testing Frameworks - NCAR May 1, 2012 18 / 61
![Page 52: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/52.jpg)
Testing Frameworks
Provide infrastructure to radically simplify:I Creating test routines (Test cases)I Running collections of tests (Test suites)I Summarizing results
Key feature is collection of assert methodsI Used to express expected results
c a l l a s s e r t E q u a l (1 20 , f a c t o r i a l ( 5 ) )
Generally specific to programming language (xUnit)I Java (JUnit)I Pnython (pyUnit)I C++ (cxxUnit, cppUnit)I Fortran (FRUIT, FUNIT, pFUnit)
Tom Clune (SSSO) TDD - Testing Frameworks - NCAR May 1, 2012 20 / 61
![Page 53: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/53.jpg)
GUI - JUnit in Eclipse
Tom Clune (SSSO) TDD - Testing Frameworks - NCAR May 1, 2012 21 / 61
![Page 54: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/54.jpg)
Outline
1 Introduction
2 Testing
3 Testing Frameworks
4 Test-Driven Development
5 TDD and Scientific/Technical Software
6 Example
7 pFUnit
Tom Clune (SSSO) TDD - Test-Driven Development - NCAR May 1, 2012 22 / 61
![Page 55: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/55.jpg)
(Somewhat) New Paradigm: TDD
Old paradigm:
Tests written by separate team (black box testing)
Tests written after implementation
Consequences:
Testing schedule compressed for release
Defects detected late in development ($$)
New paradigm
Developers write the tests (white box testing)
Tests written before production code
Enabled by emergence of strong unit testing frameworks
Tom Clune (SSSO) TDD - Test-Driven Development - NCAR May 1, 2012 23 / 61
![Page 56: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/56.jpg)
(Somewhat) New Paradigm: TDD
Old paradigm:
Tests written by separate team (black box testing)
Tests written after implementation
Consequences:
Testing schedule compressed for release
Defects detected late in development ($$)
New paradigm
Developers write the tests (white box testing)
Tests written before production code
Enabled by emergence of strong unit testing frameworks
Tom Clune (SSSO) TDD - Test-Driven Development - NCAR May 1, 2012 23 / 61
![Page 57: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/57.jpg)
(Somewhat) New Paradigm: TDD
Old paradigm:
Tests written by separate team (black box testing)
Tests written after implementation
Consequences:
Testing schedule compressed for release
Defects detected late in development ($$)
New paradigm
Developers write the tests (white box testing)
Tests written before production code
Enabled by emergence of strong unit testing frameworks
Tom Clune (SSSO) TDD - Test-Driven Development - NCAR May 1, 2012 23 / 61
![Page 58: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/58.jpg)
The TDD cycle
Tom Clune (SSSO) TDD - Test-Driven Development - NCAR May 1, 2012 24 / 61
![Page 59: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/59.jpg)
Benefits of TDD
High reliability
Excellent test coverage
Always “ready-to-ship”
Tests act as maintainable documentationI Test shows real use case scenarioI Test is maintained through TDD process
Less time spent debugging
Reduced stress / improved confidence
Productivity
Predictable schedule
Porting
Quality implementation?
Tom Clune (SSSO) TDD - Test-Driven Development - NCAR May 1, 2012 25 / 61
![Page 60: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/60.jpg)
Benefits of TDD
High reliability
Excellent test coverage
Always “ready-to-ship”
Tests act as maintainable documentationI Test shows real use case scenarioI Test is maintained through TDD process
Less time spent debugging
Reduced stress / improved confidence
Productivity
Predictable schedule
Porting
Quality implementation?
Tom Clune (SSSO) TDD - Test-Driven Development - NCAR May 1, 2012 25 / 61
![Page 61: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/61.jpg)
Benefits of TDD
High reliability
Excellent test coverage
Always “ready-to-ship”
Tests act as maintainable documentationI Test shows real use case scenarioI Test is maintained through TDD process
Less time spent debugging
Reduced stress / improved confidence
Productivity
Predictable schedule
Porting
Quality implementation?
Tom Clune (SSSO) TDD - Test-Driven Development - NCAR May 1, 2012 25 / 61
![Page 62: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/62.jpg)
Benefits of TDD
High reliability
Excellent test coverage
Always “ready-to-ship”
Tests act as maintainable documentationI Test shows real use case scenarioI Test is maintained through TDD process
Less time spent debugging
Reduced stress / improved confidence
Productivity
Predictable schedule
Porting
Quality implementation?
Tom Clune (SSSO) TDD - Test-Driven Development - NCAR May 1, 2012 25 / 61
![Page 63: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/63.jpg)
Benefits of TDD
High reliability
Excellent test coverage
Always “ready-to-ship”
Tests act as maintainable documentationI Test shows real use case scenarioI Test is maintained through TDD process
Less time spent debugging
Reduced stress / improved confidence
Productivity
Predictable schedule
Porting
Quality implementation?
Tom Clune (SSSO) TDD - Test-Driven Development - NCAR May 1, 2012 25 / 61
![Page 64: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/64.jpg)
Benefits of TDD
High reliability
Excellent test coverage
Always “ready-to-ship”
Tests act as maintainable documentationI Test shows real use case scenarioI Test is maintained through TDD process
Less time spent debugging
Reduced stress / improved confidence
Productivity
Predictable schedule
Porting
Quality implementation?
Tom Clune (SSSO) TDD - Test-Driven Development - NCAR May 1, 2012 25 / 61
![Page 65: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/65.jpg)
Benefits of TDD
High reliability
Excellent test coverage
Always “ready-to-ship”
Tests act as maintainable documentationI Test shows real use case scenarioI Test is maintained through TDD process
Less time spent debugging
Reduced stress / improved confidence
Productivity
Predictable schedule
Porting
Quality implementation?
Tom Clune (SSSO) TDD - Test-Driven Development - NCAR May 1, 2012 25 / 61
![Page 66: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/66.jpg)
Benefits of TDD
High reliability
Excellent test coverage
Always “ready-to-ship”
Tests act as maintainable documentationI Test shows real use case scenarioI Test is maintained through TDD process
Less time spent debugging
Reduced stress / improved confidence
Productivity
Predictable schedule
Porting
Quality implementation?
Tom Clune (SSSO) TDD - Test-Driven Development - NCAR May 1, 2012 25 / 61
![Page 67: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/67.jpg)
Benefits of TDD
High reliability
Excellent test coverage
Always “ready-to-ship”
Tests act as maintainable documentationI Test shows real use case scenarioI Test is maintained through TDD process
Less time spent debugging
Reduced stress / improved confidence
Productivity
Predictable schedule
Porting
Quality implementation?
Tom Clune (SSSO) TDD - Test-Driven Development - NCAR May 1, 2012 25 / 61
![Page 68: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/68.jpg)
Benefits of TDD
High reliability
Excellent test coverage
Always “ready-to-ship”
Tests act as maintainable documentationI Test shows real use case scenarioI Test is maintained through TDD process
Less time spent debugging
Reduced stress / improved confidence
Productivity
Predictable schedule
Porting
Quality implementation?
Tom Clune (SSSO) TDD - Test-Driven Development - NCAR May 1, 2012 25 / 61
![Page 69: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/69.jpg)
Benefits of TDD
High reliability
Excellent test coverage
Always “ready-to-ship”
Tests act as maintainable documentationI Test shows real use case scenarioI Test is maintained through TDD process
Less time spent debugging
Reduced stress / improved confidence
Productivity
Predictable schedule
Porting
Quality implementation?
Tom Clune (SSSO) TDD - Test-Driven Development - NCAR May 1, 2012 25 / 61
![Page 70: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/70.jpg)
Anecdotal Testimony
Many professional SEs are initially skepticalI High percentage refuse to go back to the old way after only a few days
of exposure.
Some projects drop bug tracking as unnecessary
Often difficult to sell to managementI “What? More lines of code?”
Tom Clune (SSSO) TDD - Test-Driven Development - NCAR May 1, 2012 26 / 61
![Page 71: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/71.jpg)
Not a panacea
Requires training, practice, and discipline
Need strong tools (framework + refactoring)
Does not invent new algorithms (e.g. FFT)I No such thing as magic
Maintaining tests difficult during a major re-engineering effort.
I But isnt the alternative is even worse?!!
Tom Clune (SSSO) TDD - Test-Driven Development - NCAR May 1, 2012 27 / 61
![Page 72: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/72.jpg)
Not a panacea
Requires training, practice, and discipline
Need strong tools (framework + refactoring)
Does not invent new algorithms (e.g. FFT)I No such thing as magic
Maintaining tests difficult during a major re-engineering effort.
I But isnt the alternative is even worse?!!
Tom Clune (SSSO) TDD - Test-Driven Development - NCAR May 1, 2012 27 / 61
![Page 73: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/73.jpg)
Not a panacea
Requires training, practice, and discipline
Need strong tools (framework + refactoring)
Does not invent new algorithms (e.g. FFT)
I No such thing as magic
Maintaining tests difficult during a major re-engineering effort.
I But isnt the alternative is even worse?!!
Tom Clune (SSSO) TDD - Test-Driven Development - NCAR May 1, 2012 27 / 61
![Page 74: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/74.jpg)
Not a panacea
Requires training, practice, and discipline
Need strong tools (framework + refactoring)
Does not invent new algorithms (e.g. FFT)I No such thing as magic
Maintaining tests difficult during a major re-engineering effort.
I But isnt the alternative is even worse?!!
Tom Clune (SSSO) TDD - Test-Driven Development - NCAR May 1, 2012 27 / 61
![Page 75: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/75.jpg)
Not a panacea
Requires training, practice, and discipline
Need strong tools (framework + refactoring)
Does not invent new algorithms (e.g. FFT)I No such thing as magic
Maintaining tests difficult during a major re-engineering effort.
I But isnt the alternative is even worse?!!
Tom Clune (SSSO) TDD - Test-Driven Development - NCAR May 1, 2012 27 / 61
![Page 76: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/76.jpg)
Not a panacea
Requires training, practice, and discipline
Need strong tools (framework + refactoring)
Does not invent new algorithms (e.g. FFT)I No such thing as magic
Maintaining tests difficult during a major re-engineering effort.I But isnt the alternative is even worse?!!
Tom Clune (SSSO) TDD - Test-Driven Development - NCAR May 1, 2012 27 / 61
![Page 77: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/77.jpg)
Outline
1 Introduction
2 Testing
3 Testing Frameworks
4 Test-Driven Development
5 TDD and Scientific/Technical Software
6 Example
7 pFUnit
Tom Clune (SSSO) TDD - TDD and Scientific/Technical Software - NCAR May 1, 2012 28 / 61
![Page 78: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/78.jpg)
The Challenge of Technical Software
Serious objections have been raised:
I Difficult to estimate errorF RoundoffF Truncation
I Stability/NonlinearityF Problems that occur only after long integrations
I Insufficient analytic casesI Test would just be re-expression of implementation
F Irreducible complexity?
These concerns largely revealI Lack of experience with software testingI Confusion between roles of verification vs validationI Burden of legacy software (long procedures; complex interfaces)
Tom Clune (SSSO) TDD - TDD and Scientific/Technical Software - NCAR May 1, 2012 29 / 61
![Page 79: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/79.jpg)
The Challenge of Technical Software
Serious objections have been raised:I Difficult to estimate error
F RoundoffF Truncation
I Stability/NonlinearityF Problems that occur only after long integrations
I Insufficient analytic casesI Test would just be re-expression of implementation
F Irreducible complexity?
These concerns largely revealI Lack of experience with software testingI Confusion between roles of verification vs validationI Burden of legacy software (long procedures; complex interfaces)
Tom Clune (SSSO) TDD - TDD and Scientific/Technical Software - NCAR May 1, 2012 29 / 61
![Page 80: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/80.jpg)
The Challenge of Technical Software
Serious objections have been raised:I Difficult to estimate error
F RoundoffF Truncation
I Stability/NonlinearityF Problems that occur only after long integrations
I Insufficient analytic casesI Test would just be re-expression of implementation
F Irreducible complexity?
These concerns largely revealI Lack of experience with software testingI Confusion between roles of verification vs validationI Burden of legacy software (long procedures; complex interfaces)
Tom Clune (SSSO) TDD - TDD and Scientific/Technical Software - NCAR May 1, 2012 29 / 61
![Page 81: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/81.jpg)
The Challenge of Technical Software
Serious objections have been raised:I Difficult to estimate error
F RoundoffF Truncation
I Stability/NonlinearityF Problems that occur only after long integrations
I Insufficient analytic cases
I Test would just be re-expression of implementationF Irreducible complexity?
These concerns largely revealI Lack of experience with software testingI Confusion between roles of verification vs validationI Burden of legacy software (long procedures; complex interfaces)
Tom Clune (SSSO) TDD - TDD and Scientific/Technical Software - NCAR May 1, 2012 29 / 61
![Page 82: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/82.jpg)
The Challenge of Technical Software
Serious objections have been raised:I Difficult to estimate error
F RoundoffF Truncation
I Stability/NonlinearityF Problems that occur only after long integrations
I Insufficient analytic casesI Test would just be re-expression of implementation
F Irreducible complexity?
These concerns largely revealI Lack of experience with software testingI Confusion between roles of verification vs validationI Burden of legacy software (long procedures; complex interfaces)
Tom Clune (SSSO) TDD - TDD and Scientific/Technical Software - NCAR May 1, 2012 29 / 61
![Page 83: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/83.jpg)
The Challenge of Technical Software
Serious objections have been raised:I Difficult to estimate error
F RoundoffF Truncation
I Stability/NonlinearityF Problems that occur only after long integrations
I Insufficient analytic casesI Test would just be re-expression of implementation
F Irreducible complexity?
These concerns largely reveal
I Lack of experience with software testingI Confusion between roles of verification vs validationI Burden of legacy software (long procedures; complex interfaces)
Tom Clune (SSSO) TDD - TDD and Scientific/Technical Software - NCAR May 1, 2012 29 / 61
![Page 84: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/84.jpg)
The Challenge of Technical Software
Serious objections have been raised:I Difficult to estimate error
F RoundoffF Truncation
I Stability/NonlinearityF Problems that occur only after long integrations
I Insufficient analytic casesI Test would just be re-expression of implementation
F Irreducible complexity?
These concerns largely revealI Lack of experience with software testing
I Confusion between roles of verification vs validationI Burden of legacy software (long procedures; complex interfaces)
Tom Clune (SSSO) TDD - TDD and Scientific/Technical Software - NCAR May 1, 2012 29 / 61
![Page 85: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/85.jpg)
The Challenge of Technical Software
Serious objections have been raised:I Difficult to estimate error
F RoundoffF Truncation
I Stability/NonlinearityF Problems that occur only after long integrations
I Insufficient analytic casesI Test would just be re-expression of implementation
F Irreducible complexity?
These concerns largely revealI Lack of experience with software testingI Confusion between roles of verification vs validation
I Burden of legacy software (long procedures; complex interfaces)
Tom Clune (SSSO) TDD - TDD and Scientific/Technical Software - NCAR May 1, 2012 29 / 61
![Page 86: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/86.jpg)
The Challenge of Technical Software
Serious objections have been raised:I Difficult to estimate error
F RoundoffF Truncation
I Stability/NonlinearityF Problems that occur only after long integrations
I Insufficient analytic casesI Test would just be re-expression of implementation
F Irreducible complexity?
These concerns largely revealI Lack of experience with software testingI Confusion between roles of verification vs validationI Burden of legacy software (long procedures; complex interfaces)
Tom Clune (SSSO) TDD - TDD and Scientific/Technical Software - NCAR May 1, 2012 29 / 61
![Page 87: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/87.jpg)
The Challenge of Technical Software
Serious objections have been raised:I Difficult to estimate error
F RoundoffF Truncation
I Stability/NonlinearityF Problems that occur only after long integrations
I Insufficient analytic casesI Test would just be re-expression of implementation
F Irreducible complexity?
These concerns largely revealI Lack of experience with software testingI Confusion between roles of verification vs validationI Burden of legacy software (long procedures; complex interfaces)
Tom Clune (SSSO) TDD - TDD and Scientific/Technical Software - NCAR May 1, 2012 29 / 61
![Page 88: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/88.jpg)
Software Testing vs Science/Validation
Software tests should only check implementation.
Only a subset tests will express external requirements (i.e.implementation independent)
Other tests will reflect implementation choices
Use “convenient” input values - not realistic values
Consider tests for an ODE integrator implemented with RK4
A generic test may be for a constant flow field - any integrator shouldget an “exact” answer
A RK4 specific test may provide an artificial “flow field” that returnsthe values 1.,2.,3.,4. on subsequent calls independent of thecoordinates
Tom Clune (SSSO) TDD - TDD and Scientific/Technical Software - NCAR May 1, 2012 30 / 61
![Page 89: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/89.jpg)
Test by Layers
Do test
Proper # of iterations
Pieces called in correct order
Passing of data betweencomponents
Do NOT test
Calculations inside components
Much easier to do in practice with objects than with procedures.
Tom Clune (SSSO) TDD - TDD and Scientific/Technical Software - NCAR May 1, 2012 31 / 61
![Page 90: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/90.jpg)
Numerical Tolerance
For testing numerical results, a good estimate for the tolerance isnecessary:
If the tolerance is too low, then the test may fail for uninterestingreasons.
If the tolerance is too high, then the test may have no teeth
Unfortunately ...
Error estimates are seldom available for complex algorithms
And of those, usually we just have an asymtotic form with unknownleading coefficient!
Tom Clune (SSSO) TDD - TDD and Scientific/Technical Software - NCAR May 1, 2012 32 / 61
![Page 91: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/91.jpg)
Numerical Tolerance
For testing numerical results, a good estimate for the tolerance isnecessary:
If the tolerance is too low, then the test may fail for uninterestingreasons.
If the tolerance is too high, then the test may have no teeth
Unfortunately ...
Error estimates are seldom available for complex algorithms
And of those, usually we just have an asymtotic form with unknownleading coefficient!
Tom Clune (SSSO) TDD - TDD and Scientific/Technical Software - NCAR May 1, 2012 32 / 61
![Page 92: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/92.jpg)
Numerical Tolerance
For testing numerical results, a good estimate for the tolerance isnecessary:
If the tolerance is too low, then the test may fail for uninterestingreasons.
If the tolerance is too high, then the test may have no teeth
Unfortunately ...
Error estimates are seldom available for complex algorithms
And of those, usually we just have an asymtotic form with unknownleading coefficient!
Tom Clune (SSSO) TDD - TDD and Scientific/Technical Software - NCAR May 1, 2012 32 / 61
![Page 93: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/93.jpg)
Numerical Tolerance
For testing numerical results, a good estimate for the tolerance isnecessary:
If the tolerance is too low, then the test may fail for uninterestingreasons.
If the tolerance is too high, then the test may have no teeth
Unfortunately ...
Error estimates are seldom available for complex algorithms
And of those, usually we just have an asymtotic form with unknownleading coefficient!
Tom Clune (SSSO) TDD - TDD and Scientific/Technical Software - NCAR May 1, 2012 32 / 61
![Page 94: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/94.jpg)
Numerical Tolerance
For testing numerical results, a good estimate for the tolerance isnecessary:
If the tolerance is too low, then the test may fail for uninterestingreasons.
If the tolerance is too high, then the test may have no teeth
Unfortunately ...
Error estimates are seldom available for complex algorithms
And of those, usually we just have an asymtotic form with unknownleading coefficient!
Tom Clune (SSSO) TDD - TDD and Scientific/Technical Software - NCAR May 1, 2012 32 / 61
![Page 95: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/95.jpg)
Numerical tolerance (cont’d)
Observations
1 machine epsilon is a good estimate for most short arithmeticexpressions
2 large errors arise in small expressions in fairly obvious places (1/∆)
3 larger errors are generally a result of composition of many operations
Conclusion: If we write software as a composition of distinct smallfunctions and subroutines, the errors can be reasonably bounded at eachstage
Tom Clune (SSSO) TDD - TDD and Scientific/Technical Software - NCAR May 1, 2012 33 / 61
![Page 96: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/96.jpg)
Numerical tolerance (cont’d)
Observations
1 machine epsilon is a good estimate for most short arithmeticexpressions
2 large errors arise in small expressions in fairly obvious places (1/∆)
3 larger errors are generally a result of composition of many operations
Conclusion: If we write software as a composition of distinct smallfunctions and subroutines, the errors can be reasonably bounded at eachstage
Tom Clune (SSSO) TDD - TDD and Scientific/Technical Software - NCAR May 1, 2012 33 / 61
![Page 97: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/97.jpg)
Numerical tolerance (cont’d)
Observations
1 machine epsilon is a good estimate for most short arithmeticexpressions
2 large errors arise in small expressions in fairly obvious places (1/∆)
3 larger errors are generally a result of composition of many operations
Conclusion: If we write software as a composition of distinct smallfunctions and subroutines, the errors can be reasonably bounded at eachstage
Tom Clune (SSSO) TDD - TDD and Scientific/Technical Software - NCAR May 1, 2012 33 / 61
![Page 98: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/98.jpg)
Numerical tolerance (cont’d)
Observations
1 machine epsilon is a good estimate for most short arithmeticexpressions
2 large errors arise in small expressions in fairly obvious places (1/∆)
3 larger errors are generally a result of composition of many operations
Conclusion: If we write software as a composition of distinct smallfunctions and subroutines, the errors can be reasonably bounded at eachstage
Tom Clune (SSSO) TDD - TDD and Scientific/Technical Software - NCAR May 1, 2012 33 / 61
![Page 99: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/99.jpg)
Numerical tolerance (cont’d)
Observations
1 machine epsilon is a good estimate for most short arithmeticexpressions
2 large errors arise in small expressions in fairly obvious places (1/∆)
3 larger errors are generally a result of composition of many operations
Conclusion: If we write software as a composition of distinct smallfunctions and subroutines, the errors can be reasonably bounded at eachstage
Tom Clune (SSSO) TDD - TDD and Scientific/Technical Software - NCAR May 1, 2012 33 / 61
![Page 100: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/100.jpg)
Numerical tolerance (cont’d)
Observations
1 machine epsilon is a good estimate for most short arithmeticexpressions
2 large errors arise in small expressions in fairly obvious places (1/∆)
3 larger errors are generally a result of composition of many operations
Conclusion: If we write software as a composition of distinct smallfunctions and subroutines, the errors can be reasonably bounded at eachstage
Tom Clune (SSSO) TDD - TDD and Scientific/Technical Software - NCAR May 1, 2012 33 / 61
![Page 101: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/101.jpg)
TDD and long integration
TDD does not directly relate to issues of stability
If long integration gets incorrect results:
1 Software defect: missing test2 Genuine science challenge
TDD can reduce the frequency at which long integrations areneeded/performed
Tom Clune (SSSO) TDD - TDD and Scientific/Technical Software - NCAR May 1, 2012 34 / 61
![Page 102: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/102.jpg)
TDD and long integration
TDD does not directly relate to issues of stability
If long integration gets incorrect results:1 Software defect: missing test
2 Genuine science challenge
TDD can reduce the frequency at which long integrations areneeded/performed
Tom Clune (SSSO) TDD - TDD and Scientific/Technical Software - NCAR May 1, 2012 34 / 61
![Page 103: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/103.jpg)
TDD and long integration
TDD does not directly relate to issues of stability
If long integration gets incorrect results:1 Software defect: missing test2 Genuine science challenge
TDD can reduce the frequency at which long integrations areneeded/performed
Tom Clune (SSSO) TDD - TDD and Scientific/Technical Software - NCAR May 1, 2012 34 / 61
![Page 104: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/104.jpg)
TDD and Lack of Analytic Results
Keep in mind: “How can you implement it if you cannot say what itshould do?”
Split into pieces - often each step has analytic solution
Choose input values that are convenient
Consider a trivial case:
c a l l a s s e r t E q u a l ( 3 . 1 4 1 5 9 2 6 5 , a r e a O f C i r c l e ( 1 . ) )c a l l a s s e r t E q u a l ( 6 . 2 8 . . . , a r e a O f C i r c l e ( 2 . ) )
What if instead the areaOfCircle() function accepted 2 arguments: “π”and r .
c a l l a s s e r t E q u a l ( 1 . , a r e a O f C i r c l e ( 1 . , 1 . ) )c a l l a s s e r t E q u a l ( 4 . , a r e a O f C i r c l e ( 1 . , 2 . ) )c a l l a s s e r t E q u a l ( 2 . , a r e a O f C i r c l e ( 2 . , 1 . ) )
Tom Clune (SSSO) TDD - TDD and Scientific/Technical Software - NCAR May 1, 2012 36 / 61
![Page 105: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/105.jpg)
TDD and irreducible complexity
Are the tests as complex as the implementation?
Short answer: No
Long answer: Well, they shouldn’t be ...I Unit tests use specific inputs - implementation handles generic caseI Each layer of algorithm is tested separatelyI Layers of the production code are coupled - huge complexityI Tests are decoupled - low complexity
Tom Clune (SSSO) TDD - TDD and Scientific/Technical Software - NCAR May 1, 2012 37 / 61
![Page 106: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/106.jpg)
TDD and irreducible complexity
Are the tests as complex as the implementation?
Short answer: No
Long answer: Well, they shouldn’t be ...
I Unit tests use specific inputs - implementation handles generic caseI Each layer of algorithm is tested separatelyI Layers of the production code are coupled - huge complexityI Tests are decoupled - low complexity
Tom Clune (SSSO) TDD - TDD and Scientific/Technical Software - NCAR May 1, 2012 37 / 61
![Page 107: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/107.jpg)
TDD and irreducible complexity
Are the tests as complex as the implementation?
Short answer: No
Long answer: Well, they shouldn’t be ...I Unit tests use specific inputs - implementation handles generic case
I Each layer of algorithm is tested separatelyI Layers of the production code are coupled - huge complexityI Tests are decoupled - low complexity
Tom Clune (SSSO) TDD - TDD and Scientific/Technical Software - NCAR May 1, 2012 37 / 61
![Page 108: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/108.jpg)
TDD and irreducible complexity
Are the tests as complex as the implementation?
Short answer: No
Long answer: Well, they shouldn’t be ...I Unit tests use specific inputs - implementation handles generic caseI Each layer of algorithm is tested separately
I Layers of the production code are coupled - huge complexityI Tests are decoupled - low complexity
Tom Clune (SSSO) TDD - TDD and Scientific/Technical Software - NCAR May 1, 2012 37 / 61
![Page 109: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/109.jpg)
TDD and irreducible complexity
Are the tests as complex as the implementation?
Short answer: No
Long answer: Well, they shouldn’t be ...I Unit tests use specific inputs - implementation handles generic caseI Each layer of algorithm is tested separatelyI Layers of the production code are coupled - huge complexity
I Tests are decoupled - low complexity
Tom Clune (SSSO) TDD - TDD and Scientific/Technical Software - NCAR May 1, 2012 37 / 61
![Page 110: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/110.jpg)
TDD and irreducible complexity
Are the tests as complex as the implementation?
Short answer: No
Long answer: Well, they shouldn’t be ...I Unit tests use specific inputs - implementation handles generic caseI Each layer of algorithm is tested separatelyI Layers of the production code are coupled - huge complexityI Tests are decoupled - low complexity
Tom Clune (SSSO) TDD - TDD and Scientific/Technical Software - NCAR May 1, 2012 37 / 61
![Page 111: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/111.jpg)
TDD and the Legacy Burden
TDD was created for developing new code, and does not directlyspeak to maintaining legacy code.
Adding new functionalityI Avoid wedging new loging directly into existing large procedureI Use TDD to develop separate facility for new computationI Just call the new procedure from the large legacy procedure
RefactoringI Use unit tests to constrain existing behaviorI Very difficult for large proceduresI Try to find small pieces to pull out into new procedures
Tom Clune (SSSO) TDD - TDD and Scientific/Technical Software - NCAR May 1, 2012 38 / 61
![Page 112: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/112.jpg)
TDD Best Practices
Small steps - each iteration � 10 minutes
Small, readable tests
Extremely fast execution - 1 ms/test or less
Ruthless refactoring
Verify that each test initially fails
Tom Clune (SSSO) TDD - TDD and Scientific/Technical Software - NCAR May 1, 2012 39 / 61
![Page 113: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/113.jpg)
TDD Best Practices
Small steps - each iteration � 10 minutes
Small, readable tests
Extremely fast execution - 1 ms/test or less
Ruthless refactoring
Verify that each test initially fails
Tom Clune (SSSO) TDD - TDD and Scientific/Technical Software - NCAR May 1, 2012 39 / 61
![Page 114: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/114.jpg)
TDD Best Practices
Small steps - each iteration � 10 minutes
Small, readable tests
Extremely fast execution - 1 ms/test or less
Ruthless refactoring
Verify that each test initially fails
Tom Clune (SSSO) TDD - TDD and Scientific/Technical Software - NCAR May 1, 2012 39 / 61
![Page 115: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/115.jpg)
TDD Best Practices
Small steps - each iteration � 10 minutes
Small, readable tests
Extremely fast execution - 1 ms/test or less
Ruthless refactoring
Verify that each test initially fails
Tom Clune (SSSO) TDD - TDD and Scientific/Technical Software - NCAR May 1, 2012 39 / 61
![Page 116: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/116.jpg)
TDD Best Practices
Small steps - each iteration � 10 minutes
Small, readable tests
Extremely fast execution - 1 ms/test or less
Ruthless refactoring
Verify that each test initially fails
Tom Clune (SSSO) TDD - TDD and Scientific/Technical Software - NCAR May 1, 2012 39 / 61
![Page 117: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/117.jpg)
TDD Best Practices
Small steps - each iteration � 10 minutes
Small, readable tests
Extremely fast execution - 1 ms/test or less
Ruthless refactoring
Verify that each test initially fails
Tom Clune (SSSO) TDD - TDD and Scientific/Technical Software - NCAR May 1, 2012 39 / 61
![Page 118: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/118.jpg)
TDD and Performance
Optimized algorithms may require many steps within a singleprocedure
TDD emphasizes small simple procedures
Such an approach may lead to slow execution
Solution: BootstrappingI Use initial solution as unit test for optimized solutionI Maintain both implementations
Tom Clune (SSSO) TDD - TDD and Scientific/Technical Software - NCAR May 1, 2012 40 / 61
![Page 119: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/119.jpg)
Experience to date
TDD has been used heavily within several projects at NASA
Mostly for “infrastructure” portions - relatively little numerical alg.
pFUnit
DYNAMO - spectral MHD code on shperical shell
GTRAJ - offline trajectory integration (C++)
Snowfake - virtual snowfakes; Multi-lattice Snowfake
Observations:
∼ 1:1 ratio of test code to source code
Works very well for infrastructure
Learning curveI 1-2 days for techniqueI Weeks-months to wean old habitsI Full benefit may require some sophistication
Tom Clune (SSSO) TDD - TDD and Scientific/Technical Software - NCAR May 1, 2012 41 / 61
![Page 120: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/120.jpg)
Outline
1 Introduction
2 Testing
3 Testing Frameworks
4 Test-Driven Development
5 TDD and Scientific/Technical Software
6 Example
7 pFUnit
Tom Clune (SSSO) TDD - Example - NCAR May 1, 2012 42 / 61
![Page 121: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/121.jpg)
Linear Interpolation
��
�
6y
-xx1 x2 x3 xnx
t tt t
t tt
Tom Clune (SSSO) TDD - Example - NCAR May 1, 2012 43 / 61
![Page 122: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/122.jpg)
Potential Tests
Bracketing: Find i such that xi <= x < xi+1
Computing node weights:
wa =xi+1 − x
xi+1 − xiwb = 1− wa
Compute weighted sum: y = waf (xi ) + wbf (xi+1)
Tom Clune (SSSO) TDD - Example - NCAR May 1, 2012 44 / 61
![Page 123: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/123.jpg)
Potential Tests
Bracketing: Find i such that xi <= x < xi+1
Computing node weights:
wa =xi+1 − x
xi+1 − xiwb = 1− wa
Compute weighted sum: y = waf (xi ) + wbf (xi+1)
Tom Clune (SSSO) TDD - Example - NCAR May 1, 2012 44 / 61
![Page 124: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/124.jpg)
Potential Tests
Bracketing: Find i such that xi <= x < xi+1
Computing node weights:
wa =xi+1 − x
xi+1 − xiwb = 1− wa
Compute weighted sum: y = waf (xi ) + wbf (xi+1)
Tom Clune (SSSO) TDD - Example - NCAR May 1, 2012 44 / 61
![Page 125: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/125.jpg)
Potential Tests
Bracketing: Find i such that xi <= x < xi+1
Computing node weights:
wa =xi+1 − x
xi+1 − xiwb = 1− wa
Compute weighted sum: y = waf (xi ) + wbf (xi+1)
Tom Clune (SSSO) TDD - Example - NCAR May 1, 2012 44 / 61
![Page 126: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/126.jpg)
Bracketing Tests
i n d e x = b r a c k e t ( nodes , x )
Case Preconditions Postconditionnodes x return
interior {x} = {1, 2, 3} x = 1.5 i = 1other interior {x} = {1, 2, 3} x = 2.5 i = 2at node {x} = {1, 2, 3} x = 2.0 i = 2 (?)at edge {x} = {1, 2, 3} x = 1.0 i = 1 (?)other edge {x} = {1, 2, 3} x = 3.0 i = 2 (????)out-of-bounds {x} = {1, 2, 3} x = 1.5 out-of-bounds errorout-of-order {x} = {1, 2, 3} x = 1.5 out-of-order error
Tom Clune (SSSO) TDD - Example - NCAR May 1, 2012 46 / 61
![Page 127: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/127.jpg)
Bracketing Tests
i n d e x = b r a c k e t ( nodes , x )
Case Preconditions Postconditionnodes x return
interior {x} = {1, 2, 3} x = 1.5 i = 1
other interior {x} = {1, 2, 3} x = 2.5 i = 2at node {x} = {1, 2, 3} x = 2.0 i = 2 (?)at edge {x} = {1, 2, 3} x = 1.0 i = 1 (?)other edge {x} = {1, 2, 3} x = 3.0 i = 2 (????)out-of-bounds {x} = {1, 2, 3} x = 1.5 out-of-bounds errorout-of-order {x} = {1, 2, 3} x = 1.5 out-of-order error
Tom Clune (SSSO) TDD - Example - NCAR May 1, 2012 46 / 61
![Page 128: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/128.jpg)
Bracketing Tests
i n d e x = b r a c k e t ( nodes , x )
Case Preconditions Postconditionnodes x return
interior {x} = {1, 2, 3} x = 1.5 i = 1other interior {x} = {1, 2, 3} x = 2.5 i = 2
at node {x} = {1, 2, 3} x = 2.0 i = 2 (?)at edge {x} = {1, 2, 3} x = 1.0 i = 1 (?)other edge {x} = {1, 2, 3} x = 3.0 i = 2 (????)out-of-bounds {x} = {1, 2, 3} x = 1.5 out-of-bounds errorout-of-order {x} = {1, 2, 3} x = 1.5 out-of-order error
Tom Clune (SSSO) TDD - Example - NCAR May 1, 2012 46 / 61
![Page 129: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/129.jpg)
Bracketing Tests
i n d e x = b r a c k e t ( nodes , x )
Case Preconditions Postconditionnodes x return
interior {x} = {1, 2, 3} x = 1.5 i = 1other interior {x} = {1, 2, 3} x = 2.5 i = 2at node {x} = {1, 2, 3} x = 2.0 i = 2 (?)
at edge {x} = {1, 2, 3} x = 1.0 i = 1 (?)other edge {x} = {1, 2, 3} x = 3.0 i = 2 (????)out-of-bounds {x} = {1, 2, 3} x = 1.5 out-of-bounds errorout-of-order {x} = {1, 2, 3} x = 1.5 out-of-order error
Tom Clune (SSSO) TDD - Example - NCAR May 1, 2012 46 / 61
![Page 130: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/130.jpg)
Bracketing Tests
i n d e x = b r a c k e t ( nodes , x )
Case Preconditions Postconditionnodes x return
interior {x} = {1, 2, 3} x = 1.5 i = 1other interior {x} = {1, 2, 3} x = 2.5 i = 2at node {x} = {1, 2, 3} x = 2.0 i = 2 (?)at edge {x} = {1, 2, 3} x = 1.0 i = 1 (?)
other edge {x} = {1, 2, 3} x = 3.0 i = 2 (????)out-of-bounds {x} = {1, 2, 3} x = 1.5 out-of-bounds errorout-of-order {x} = {1, 2, 3} x = 1.5 out-of-order error
Tom Clune (SSSO) TDD - Example - NCAR May 1, 2012 46 / 61
![Page 131: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/131.jpg)
Bracketing Tests
i n d e x = b r a c k e t ( nodes , x )
Case Preconditions Postconditionnodes x return
interior {x} = {1, 2, 3} x = 1.5 i = 1other interior {x} = {1, 2, 3} x = 2.5 i = 2at node {x} = {1, 2, 3} x = 2.0 i = 2 (?)at edge {x} = {1, 2, 3} x = 1.0 i = 1 (?)other edge {x} = {1, 2, 3} x = 3.0 i = 2 (????)
out-of-bounds {x} = {1, 2, 3} x = 1.5 out-of-bounds errorout-of-order {x} = {1, 2, 3} x = 1.5 out-of-order error
Tom Clune (SSSO) TDD - Example - NCAR May 1, 2012 46 / 61
![Page 132: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/132.jpg)
Bracketing Tests
i n d e x = b r a c k e t ( nodes , x )
Case Preconditions Postconditionnodes x return
interior {x} = {1, 2, 3} x = 1.5 i = 1other interior {x} = {1, 2, 3} x = 2.5 i = 2at node {x} = {1, 2, 3} x = 2.0 i = 2 (?)at edge {x} = {1, 2, 3} x = 1.0 i = 1 (?)other edge {x} = {1, 2, 3} x = 3.0 i = 2 (????)out-of-bounds {x} = {1, 2, 3} x = 1.5 out-of-bounds error
out-of-order {x} = {1, 2, 3} x = 1.5 out-of-order error
Tom Clune (SSSO) TDD - Example - NCAR May 1, 2012 46 / 61
![Page 133: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/133.jpg)
Bracketing Tests
i n d e x = b r a c k e t ( nodes , x )
Case Preconditions Postconditionnodes x return
interior {x} = {1, 2, 3} x = 1.5 i = 1other interior {x} = {1, 2, 3} x = 2.5 i = 2at node {x} = {1, 2, 3} x = 2.0 i = 2 (?)at edge {x} = {1, 2, 3} x = 1.0 i = 1 (?)other edge {x} = {1, 2, 3} x = 3.0 i = 2 (????)out-of-bounds {x} = {1, 2, 3} x = 1.5 out-of-bounds errorout-of-order {x} = {1, 2, 3} x = 1.5 out-of-order error
Tom Clune (SSSO) TDD - Example - NCAR May 1, 2012 46 / 61
![Page 134: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/134.jpg)
Example: Bracketing Test 1
Preconditions: {x} = {1, 2, 3}, x = 1.5
Postcondition: return 1
s u b r o u t i n e t e s t B r a c k e t 1 ( )nodes = [ 1 . , 2 . , 3 . ]i n d e x = g e t B r a c k e t ( nodes , 1 . 5 )c a l l a s s e r t E q u a l ( 1 , i n d e x )
end s u b r o u t i n e
s u b r o u t i n e t e s t B r a c k e t 1 ( )c a l l a s s e r t E q u a l ( 1 , g e t B r a c k e t ( [ 1 . , 2 . , 3 . ] , 1 . 5 ) )
end s u b r o u t i n e
f u n c t i o n g e t B r a c k e t ( nodes , x ) r e s u l t ( i n d e x )i n d e x = 1
end f u n c t i o n
Tom Clune (SSSO) TDD - Example - NCAR May 1, 2012 48 / 61
![Page 135: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/135.jpg)
Example: Bracketing Test 1
Preconditions: {x} = {1, 2, 3}, x = 1.5
Postcondition: return 1
s u b r o u t i n e t e s t B r a c k e t 1 ( )nodes = [ 1 . , 2 . , 3 . ]i n d e x = g e t B r a c k e t ( nodes , 1 . 5 )c a l l a s s e r t E q u a l ( 1 , i n d e x )
end s u b r o u t i n e
s u b r o u t i n e t e s t B r a c k e t 1 ( )c a l l a s s e r t E q u a l ( 1 , g e t B r a c k e t ( [ 1 . , 2 . , 3 . ] , 1 . 5 ) )
end s u b r o u t i n e
f u n c t i o n g e t B r a c k e t ( nodes , x ) r e s u l t ( i n d e x )i n d e x = 1
end f u n c t i o n
Tom Clune (SSSO) TDD - Example - NCAR May 1, 2012 48 / 61
![Page 136: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/136.jpg)
Example: Bracketing Test 1
Preconditions: {x} = {1, 2, 3}, x = 1.5
Postcondition: return 1
s u b r o u t i n e t e s t B r a c k e t 1 ( )nodes = [ 1 . , 2 . , 3 . ]i n d e x = g e t B r a c k e t ( nodes , 1 . 5 )c a l l a s s e r t E q u a l ( 1 , i n d e x )
end s u b r o u t i n e
s u b r o u t i n e t e s t B r a c k e t 1 ( )c a l l a s s e r t E q u a l ( 1 , g e t B r a c k e t ( [ 1 . , 2 . , 3 . ] , 1 . 5 ) )
end s u b r o u t i n e
f u n c t i o n g e t B r a c k e t ( nodes , x ) r e s u l t ( i n d e x )i n d e x = 1
end f u n c t i o n
Tom Clune (SSSO) TDD - Example - NCAR May 1, 2012 48 / 61
![Page 137: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/137.jpg)
Example: Bracketing Test 1
Preconditions: {x} = {1, 2, 3}, x = 1.5
Postcondition: return 1
s u b r o u t i n e t e s t B r a c k e t 1 ( )nodes = [ 1 . , 2 . , 3 . ]i n d e x = g e t B r a c k e t ( nodes , 1 . 5 )c a l l a s s e r t E q u a l ( 1 , i n d e x )
end s u b r o u t i n e
s u b r o u t i n e t e s t B r a c k e t 1 ( )c a l l a s s e r t E q u a l ( 1 , g e t B r a c k e t ( [ 1 . , 2 . , 3 . ] , 1 . 5 ) )
end s u b r o u t i n e
f u n c t i o n g e t B r a c k e t ( nodes , x ) r e s u l t ( i n d e x )i n d e x = 1
end f u n c t i o n
Tom Clune (SSSO) TDD - Example - NCAR May 1, 2012 48 / 61
![Page 138: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/138.jpg)
Example: Bracketing Test 2
Preconditions: {x} = {1, 2, 3}, x = 2.5
Postcondition: return 2
s u b r o u t i n e t e s t B r a c k e t 2 ( )nodes = [ 1 . , 2 . , 3 . ]i n d e x = g e t B r a c k e t ( nodes , 2 . 5 )c a l l a s s e r t E q u a l ( 2 , i n d e x )
end s u b r o u t i n e
f u n c t i o n g e t B r a c k e t ( nodes , x ) r e s u l t ( i n d e x )i f ( x > nodes ( 2 ) ) then
i n d e x = 2e l s e
i n d e x = 1end i f
end f u n c t i o n
Generalize ...
f u n c t i o n g e t B r a c k e t ( nodes , x ) r e s u l t ( i n d e x )
do i = 1 , s i z e ( nodes ) 1i f ( nodes ( i +1) > x ) i n d e x = i
end do
end f u n c t i o n
Tom Clune (SSSO) TDD - Example - NCAR May 1, 2012 50 / 61
![Page 139: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/139.jpg)
Example: Bracketing Test 2
Preconditions: {x} = {1, 2, 3}, x = 2.5
Postcondition: return 2
s u b r o u t i n e t e s t B r a c k e t 2 ( )nodes = [ 1 . , 2 . , 3 . ]i n d e x = g e t B r a c k e t ( nodes , 2 . 5 )c a l l a s s e r t E q u a l ( 2 , i n d e x )
end s u b r o u t i n e
f u n c t i o n g e t B r a c k e t ( nodes , x ) r e s u l t ( i n d e x )i f ( x > nodes ( 2 ) ) then
i n d e x = 2e l s e
i n d e x = 1end i f
end f u n c t i o n
Generalize ...
f u n c t i o n g e t B r a c k e t ( nodes , x ) r e s u l t ( i n d e x )
do i = 1 , s i z e ( nodes ) 1i f ( nodes ( i +1) > x ) i n d e x = i
end do
end f u n c t i o n
Tom Clune (SSSO) TDD - Example - NCAR May 1, 2012 50 / 61
![Page 140: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/140.jpg)
Example: Bracketing Test 2
Preconditions: {x} = {1, 2, 3}, x = 2.5
Postcondition: return 2
s u b r o u t i n e t e s t B r a c k e t 2 ( )nodes = [ 1 . , 2 . , 3 . ]i n d e x = g e t B r a c k e t ( nodes , 2 . 5 )c a l l a s s e r t E q u a l ( 2 , i n d e x )
end s u b r o u t i n e
f u n c t i o n g e t B r a c k e t ( nodes , x ) r e s u l t ( i n d e x )i f ( x > nodes ( 2 ) ) then
i n d e x = 2e l s e
i n d e x = 1end i f
end f u n c t i o n
Generalize ...
f u n c t i o n g e t B r a c k e t ( nodes , x ) r e s u l t ( i n d e x )
do i = 1 , s i z e ( nodes ) 1i f ( nodes ( i +1) > x ) i n d e x = i
end do
end f u n c t i o n
Tom Clune (SSSO) TDD - Example - NCAR May 1, 2012 50 / 61
![Page 141: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/141.jpg)
Example: Bracketing Test 2
Preconditions: {x} = {1, 2, 3}, x = 2.5
Postcondition: return 2
s u b r o u t i n e t e s t B r a c k e t 2 ( )nodes = [ 1 . , 2 . , 3 . ]i n d e x = g e t B r a c k e t ( nodes , 2 . 5 )c a l l a s s e r t E q u a l ( 2 , i n d e x )
end s u b r o u t i n e
f u n c t i o n g e t B r a c k e t ( nodes , x ) r e s u l t ( i n d e x )i f ( x > nodes ( 2 ) ) then
i n d e x = 2e l s e
i n d e x = 1end i f
end f u n c t i o n
Generalize ...
f u n c t i o n g e t B r a c k e t ( nodes , x ) r e s u l t ( i n d e x )
do i = 1 , s i z e ( nodes ) 1i f ( nodes ( i +1) > x ) i n d e x = i
end do
end f u n c t i o n
Tom Clune (SSSO) TDD - Example - NCAR May 1, 2012 50 / 61
![Page 142: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/142.jpg)
Tests for Computing Weights
i n d e x = b r a c k e t ( nodes , x )
Case Preconditions Postconditioninterval x weights
lower bound [1., 2.] x = 1.0 w = [1.0, 0.0]upper bound [1., 2.] x = 1.0 w = [0.0, 1.0]interior [1., 2.] x = 1.5 w = [0.5, 0.5]big interval slope [1., 3.] x = 1.5 w = [0.75, 0.25]degenerate [1., 1.] x = 1.0 degenerate errorout-of-bounds [1., 2.] x = 0.5 out-of-bounds error
Tom Clune (SSSO) TDD - Example - NCAR May 1, 2012 52 / 61
![Page 143: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/143.jpg)
Tests for Computing Weights
i n d e x = b r a c k e t ( nodes , x )
Case Preconditions Postconditioninterval x weights
lower bound [1., 2.] x = 1.0 w = [1.0, 0.0]
upper bound [1., 2.] x = 1.0 w = [0.0, 1.0]interior [1., 2.] x = 1.5 w = [0.5, 0.5]big interval slope [1., 3.] x = 1.5 w = [0.75, 0.25]degenerate [1., 1.] x = 1.0 degenerate errorout-of-bounds [1., 2.] x = 0.5 out-of-bounds error
Tom Clune (SSSO) TDD - Example - NCAR May 1, 2012 52 / 61
![Page 144: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/144.jpg)
Tests for Computing Weights
i n d e x = b r a c k e t ( nodes , x )
Case Preconditions Postconditioninterval x weights
lower bound [1., 2.] x = 1.0 w = [1.0, 0.0]upper bound [1., 2.] x = 1.0 w = [0.0, 1.0]
interior [1., 2.] x = 1.5 w = [0.5, 0.5]big interval slope [1., 3.] x = 1.5 w = [0.75, 0.25]degenerate [1., 1.] x = 1.0 degenerate errorout-of-bounds [1., 2.] x = 0.5 out-of-bounds error
Tom Clune (SSSO) TDD - Example - NCAR May 1, 2012 52 / 61
![Page 145: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/145.jpg)
Tests for Computing Weights
i n d e x = b r a c k e t ( nodes , x )
Case Preconditions Postconditioninterval x weights
lower bound [1., 2.] x = 1.0 w = [1.0, 0.0]upper bound [1., 2.] x = 1.0 w = [0.0, 1.0]interior [1., 2.] x = 1.5 w = [0.5, 0.5]
big interval slope [1., 3.] x = 1.5 w = [0.75, 0.25]degenerate [1., 1.] x = 1.0 degenerate errorout-of-bounds [1., 2.] x = 0.5 out-of-bounds error
Tom Clune (SSSO) TDD - Example - NCAR May 1, 2012 52 / 61
![Page 146: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/146.jpg)
Tests for Computing Weights
i n d e x = b r a c k e t ( nodes , x )
Case Preconditions Postconditioninterval x weights
lower bound [1., 2.] x = 1.0 w = [1.0, 0.0]upper bound [1., 2.] x = 1.0 w = [0.0, 1.0]interior [1., 2.] x = 1.5 w = [0.5, 0.5]big interval slope [1., 3.] x = 1.5 w = [0.75, 0.25]
degenerate [1., 1.] x = 1.0 degenerate errorout-of-bounds [1., 2.] x = 0.5 out-of-bounds error
Tom Clune (SSSO) TDD - Example - NCAR May 1, 2012 52 / 61
![Page 147: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/147.jpg)
Tests for Computing Weights
i n d e x = b r a c k e t ( nodes , x )
Case Preconditions Postconditioninterval x weights
lower bound [1., 2.] x = 1.0 w = [1.0, 0.0]upper bound [1., 2.] x = 1.0 w = [0.0, 1.0]interior [1., 2.] x = 1.5 w = [0.5, 0.5]big interval slope [1., 3.] x = 1.5 w = [0.75, 0.25]degenerate [1., 1.] x = 1.0 degenerate error
out-of-bounds [1., 2.] x = 0.5 out-of-bounds error
Tom Clune (SSSO) TDD - Example - NCAR May 1, 2012 52 / 61
![Page 148: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/148.jpg)
Tests for Computing Weights
i n d e x = b r a c k e t ( nodes , x )
Case Preconditions Postconditioninterval x weights
lower bound [1., 2.] x = 1.0 w = [1.0, 0.0]upper bound [1., 2.] x = 1.0 w = [0.0, 1.0]interior [1., 2.] x = 1.5 w = [0.5, 0.5]big interval slope [1., 3.] x = 1.5 w = [0.75, 0.25]degenerate [1., 1.] x = 1.0 degenerate errorout-of-bounds [1., 2.] x = 0.5 out-of-bounds error
Tom Clune (SSSO) TDD - Example - NCAR May 1, 2012 52 / 61
![Page 149: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/149.jpg)
Example: Weights Test 1
Precondition: [a, b] = [1., 2.], x = 1.0
Postcondition: w = {1.0, 0.0}
s u b r o u t i n e t e s t W e i g h t 1 ( )r e a l : : i n t e r v a l ( 2 ) , w e i g h t s ( 2 )r e a l : : xi n t e r v a l = [ 1 . , 2 . ]w e i g h t s = computeWeights ( i n t e r v a l , 1 . 0 )c a l l a s s e r t E q u a l ( [ 1 . 0 , 0 . 0 ] , w e i g h t s )
end s u b r o u t i n e t e s t W e i g h t 1
r e a l f u n c t i o n computeWeights ( i n t e r v a l , x ) r e s u l t ( w e i g h t s )r e a l , i n t e n t ( i n ) : : i n t e r v a l ( 2 )r e a l , i n t e n t ( i n ) : : xw e i g h t s = [ 1 . 0 , 0 . 0 ]
end f u n c t i o n
Tom Clune (SSSO) TDD - Example - NCAR May 1, 2012 54 / 61
![Page 150: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/150.jpg)
Example: Tying it together
Precondition:I {(x , y)i} = {(1, 1), (2, 1), (4, 1)}I x = 3
Postcondition: y = 1.
s u b r o u t i n e t e s t I n t e r p o l a t e C o n s t a n t Y ( )r e a l : : nodes ( 2 , 3 )nodes = r e s h a p e ( [ [ 1 , 1 ] , [ 2 , 1 ] , [ 4 , 1 ] ] , shape = [ 2 , 3 ] )c a l l a s s e r t E q u a l ( 1 . 0 , i n t e r p o l a t e ( nodes , 3 . 0 ) )
end s u b r o u t i n e t e s t I n t e r p o l a t e 1
f u n c t i o n i n t e r p o l a t e ( nodes , x )r e a l , i n t e n t ( i n ) : : nodes ( : , : )y = 1
end f u n c t i o n i n t e r p o l a t e
Tom Clune (SSSO) TDD - Example - NCAR May 1, 2012 56 / 61
![Page 151: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/151.jpg)
Example: Tying it together
Precondition:I {(x , y)i} = {(1, 1), (2, 3), (4, 1)}I x = 3
Postcondition: y = 2.
s u b r o u t i n e t e s t I n t e r p o l a t e 1 ( )r e a l : : nodes ( 2 , 3 )nodes = r e s h a p e ( [ [ 1 , 1 ] , [ 2 , 3 ] , [ 4 , 1 ] ] , shape = [ 2 , 3 ] )c a l l a s s e r t E q u a l ( 1 . 0 , i n t e r p o l a t e ( nodes , 3 . 0 ) )
end s u b r o u t i n e t e s t I n t e r p o l a t e 1
f u n c t i o n i n t e r p o l a t e ( nodes , x ) r e s u l t ( y )i n t e g e r : : ir e a l : : w e i g h t s ( 2 ) , xAtEndPoints ( 2 ) , y A t E n d p o i n t s ( 2 )
i = g e t B r a c k e t ( nodes ( 1 , : ) , x )
xAtEndPoints = nodes ( 1 , i ) ! used d e r i v e d type ?y A t E n d p o i n t s = nodes ( 2 , i )w e i g h t s = computeWeights ( nodes ( 1 , [ i , i +1]) , x )
y = sum ( w e i g h t s ∗ y A t E n d p o i n t s )end f u n c t i o n i n t e r p o l a t e
Tom Clune (SSSO) TDD - Example - NCAR May 1, 2012 58 / 61
![Page 152: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/152.jpg)
Outline
1 Introduction
2 Testing
3 Testing Frameworks
4 Test-Driven Development
5 TDD and Scientific/Technical Software
6 Example
7 pFUnit
Tom Clune (SSSO) TDD - pFUnit - NCAR May 1, 2012 59 / 61
![Page 153: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/153.jpg)
pFUnit - Fortran Unit testing framework
Tests written in Fortran
Supports testing of parallel (MPI) algorithms
Support for multi-dimensional array assertions
Written in standard F95 (plus a tiny bit of F2003)
Developed using TDD
Tutorial in the afternoon sessioon
Tom Clune (SSSO) TDD - pFUnit - NCAR May 1, 2012 60 / 61
![Page 154: Test Driven Development of Scientific Models · 2013-04-10 · Outline 1 Introduction 2 Testing 3 Testing Frameworks 4 Test-Driven Development 5 TDD and Scienti c/Technical Software](https://reader034.fdocuments.net/reader034/viewer/2022042205/5ea71789be7fea66fd0bb084/html5/thumbnails/154.jpg)
References
pFUnit: http://sourceforge.net/projects/pfunit/
Tutorial materialsI https://modelingguru.nasa.gov/docs/DOC-1982I https://modelingguru.nasa.gov/docs/DOC-1983I https://modelingguru.nasa.gov/docs/DOC-1984
TDD Bloghttps://modelingguru.nasa.gov/blogs/modelingwithtdd
Test-Driven Development: By Example - Kent Beck
Mller and Padberg,”About the Return on Investment of Test-DrivenDevelopment,” http://www.ipd.uka.de/mitarbeiter/muellerm/
publications/edser03.pdf
Refactoring: Improving the Design of Existing Code - Martin Fowler
JUnit http://junit.sourceforge.net/
These slides https://modelingguru.nasa.gov/docs/DOC-2222
Tom Clune (SSSO) TDD - pFUnit - NCAR May 1, 2012 61 / 61