Progress in Testing Component-Based Software Craig H. Wittenberg Component Applications Group...
-
Upload
elijah-white -
Category
Documents
-
view
225 -
download
1
Transcript of Progress in Testing Component-Based Software Craig H. Wittenberg Component Applications Group...
![Page 1: Progress in Testing Component-Based Software Craig H. Wittenberg Component Applications Group Microsoft Research.](https://reader036.fdocuments.net/reader036/viewer/2022081519/56649ef75503460f94c0abf5/html5/thumbnails/1.jpg)
Progress in Testing Component-Based Software
Craig H. Wittenberg
Component Applications Group
Microsoft Research
![Page 2: Progress in Testing Component-Based Software Craig H. Wittenberg Component Applications Group Microsoft Research.](https://reader036.fdocuments.net/reader036/viewer/2022081519/56649ef75503460f94c0abf5/html5/thumbnails/2.jpg)
Aug. 23, 2000 ISSTA 2000 2
Talk Outline
Our project
Demos
Component and composition
Progress in testing components
Areas for further research
![Page 3: Progress in Testing Component-Based Software Craig H. Wittenberg Component Applications Group Microsoft Research.](https://reader036.fdocuments.net/reader036/viewer/2022081519/56649ef75503460f94c0abf5/html5/thumbnails/3.jpg)
Aug. 23, 2000 ISSTA 2000 3
High Level Goals
A system of flexible components for a family of products
Faster and cheaper application assembly and evolution
Reliable, efficient, and consistent applications working together
![Page 4: Progress in Testing Component-Based Software Craig H. Wittenberg Component Applications Group Microsoft Research.](https://reader036.fdocuments.net/reader036/viewer/2022081519/56649ef75503460f94c0abf5/html5/thumbnails/4.jpg)
Aug. 23, 2000 ISSTA 2000 4
Octarine Demo
Prototype written in C
Looks familiar
Adding Bullets, Music
Followed by architectural drawings
![Page 5: Progress in Testing Component-Based Software Craig H. Wittenberg Component Applications Group Microsoft Research.](https://reader036.fdocuments.net/reader036/viewer/2022081519/56649ef75503460f94c0abf5/html5/thumbnails/5.jpg)
Aug. 23, 2000 ISSTA 2000 5
Octarine Text Architecture
ISequence Story ISequence Text
ISequence Music
Table
Text
ISequence
![Page 6: Progress in Testing Component-Based Software Craig H. Wittenberg Component Applications Group Microsoft Research.](https://reader036.fdocuments.net/reader036/viewer/2022081519/56649ef75503460f94c0abf5/html5/thumbnails/6.jpg)
Aug. 23, 2000 ISSTA 2000 6
Adding Bullets/Numbering
ISequence Story ISequence Text
ISequence Music
Table
Text
ISequence
ISequence Bullets/Numbering
ISequence
Text
![Page 7: Progress in Testing Component-Based Software Craig H. Wittenberg Component Applications Group Microsoft Research.](https://reader036.fdocuments.net/reader036/viewer/2022081519/56649ef75503460f94c0abf5/html5/thumbnails/7.jpg)
Aug. 23, 2000 ISSTA 2000 7
Landscape Demo
More recentMany small componentsStatic and dynamic connectionsMultiple uses of: Text, Layout, Grid, Data pipes
Architecture for many UisSmall devicesNon-visual interfaces
![Page 8: Progress in Testing Component-Based Software Craig H. Wittenberg Component Applications Group Microsoft Research.](https://reader036.fdocuments.net/reader036/viewer/2022081519/56649ef75503460f94c0abf5/html5/thumbnails/8.jpg)
Aug. 23, 2000 ISSTA 2000 8
Our Approach
Factoring and composition
Precise interface specs
Significant design and prototype step
Huge emphasis on testing
Control inter-component dependencies
Tools support (design, build, and run time)
Critical mass of designs, components, frameworks, large demos
![Page 9: Progress in Testing Component-Based Software Craig H. Wittenberg Component Applications Group Microsoft Research.](https://reader036.fdocuments.net/reader036/viewer/2022081519/56649ef75503460f94c0abf5/html5/thumbnails/9.jpg)
Aug. 23, 2000 ISSTA 2000 9
Component
Key points: Contracts, Composition, Dependencies, Deployment
No observable global state
“A software component is a unit of composition with contractually specified interfaces and explicit context dependencies only. A software component can be deployed independently and is subject to composition by third parties.”
Szyperski, Component Software, pg. 34.
![Page 10: Progress in Testing Component-Based Software Craig H. Wittenberg Component Applications Group Microsoft Research.](https://reader036.fdocuments.net/reader036/viewer/2022081519/56649ef75503460f94c0abf5/html5/thumbnails/10.jpg)
Aug. 23, 2000 ISSTA 2000 10
Component DLL
Binary package plus docsNo top-level APIsContains one or more classes Implementation of contracts only No implementation inheritance Realize behavior of component Instances of classes are connected Dependencies upon
other classes services from other instances
![Page 11: Progress in Testing Component-Based Software Craig H. Wittenberg Component Applications Group Microsoft Research.](https://reader036.fdocuments.net/reader036/viewer/2022081519/56649ef75503460f94c0abf5/html5/thumbnails/11.jpg)
Aug. 23, 2000 ISSTA 2000 11
An interface has: Name (compile time and runtime) Methods with pre/post conditions Abstract model with invariants State table
Specified separately from implementationImplementation does the obvious and: maps model variables to internal state
Never changed once published
Precise Interface Specs a.k.a. Software Contracts
![Page 12: Progress in Testing Component-Based Software Craig H. Wittenberg Component Applications Group Microsoft Research.](https://reader036.fdocuments.net/reader036/viewer/2022081519/56649ef75503460f94c0abf5/html5/thumbnails/12.jpg)
Aug. 23, 2000 ISSTA 2000 12
ILight Methods and Model
ILight, {2461A1A0-639E-11d2-874D-00AA0060FCA9}
TurnOn();TurnOff();SetIntensity(int desired);
Modelboolean on; int intensity; // range is 0..100.
![Page 13: Progress in Testing Component-Based Software Craig H. Wittenberg Component Applications Group Microsoft Research.](https://reader036.fdocuments.net/reader036/viewer/2022081519/56649ef75503460f94c0abf5/html5/thumbnails/13.jpg)
Aug. 23, 2000 ISSTA 2000 13
ILight Pre/Post Conditions
TurnOn():Post: on == true.
TurnOff():Post: on == false.
SetIntensity(desired):Pre: desired >=0 && desired <= 100.Post: intensity == desired.
![Page 14: Progress in Testing Component-Based Software Craig H. Wittenberg Component Applications Group Microsoft Research.](https://reader036.fdocuments.net/reader036/viewer/2022081519/56649ef75503460f94c0abf5/html5/thumbnails/14.jpg)
Aug. 23, 2000 ISSTA 2000 14
A Light
The Light class implements ILight
Contract-specific and implementation-specific model mapping
LightILight
Map ILight
model to Impl.
![Page 15: Progress in Testing Component-Based Software Craig H. Wittenberg Component Applications Group Microsoft Research.](https://reader036.fdocuments.net/reader036/viewer/2022081519/56649ef75503460f94c0abf5/html5/thumbnails/15.jpg)
Aug. 23, 2000 ISSTA 2000 15
Record Filter
Filter processes one set of records into another
Record Filter
Plants
Filter Spec
IString
IRecordSetIRecordSet
Map IRecordSet model to
Impl.
![Page 16: Progress in Testing Component-Based Software Craig H. Wittenberg Component Applications Group Microsoft Research.](https://reader036.fdocuments.net/reader036/viewer/2022081519/56649ef75503460f94c0abf5/html5/thumbnails/16.jpg)
Aug. 23, 2000 ISSTA 2000 16
Light Switch
Converts calls to push() into on() and off()
Light Switch
A lightIPushIt ILight
Map IPushIt model to
Impl.
![Page 17: Progress in Testing Component-Based Software Craig H. Wittenberg Component Applications Group Microsoft Research.](https://reader036.fdocuments.net/reader036/viewer/2022081519/56649ef75503460f94c0abf5/html5/thumbnails/17.jpg)
Aug. 23, 2000 ISSTA 2000 17
Closure
Executes closure over parameters data or connected objects
ClosureParameters
IDoIt IUnknown
Map IDoIt model to
Impl.
……
…
![Page 18: Progress in Testing Component-Based Software Craig H. Wittenberg Component Applications Group Microsoft Research.](https://reader036.fdocuments.net/reader036/viewer/2022081519/56649ef75503460f94c0abf5/html5/thumbnails/18.jpg)
Aug. 23, 2000 ISSTA 2000 18
Composition
Composites looks like components Can test small pieces Can also use large ones
Separate structure from algorithms Definition/analysis of relationships Eliminates explicit dependencies
No glue as usually meant connection glue is factored out algorithmic glue is encapsulated
![Page 19: Progress in Testing Component-Based Software Craig H. Wittenberg Component Applications Group Microsoft Research.](https://reader036.fdocuments.net/reader036/viewer/2022081519/56649ef75503460f94c0abf5/html5/thumbnails/19.jpg)
Aug. 23, 2000 ISSTA 2000 19
Composition time
Structure may be decided at Design time
Fully static composition Parameterized composition
Runtime Dynamic composition described as changes to existing
structure May be downloaded as add-on software package
Haven’t fully explored tradeoffs; tending towards more runtime composition
![Page 20: Progress in Testing Component-Based Software Craig H. Wittenberg Component Applications Group Microsoft Research.](https://reader036.fdocuments.net/reader036/viewer/2022081519/56649ef75503460f94c0abf5/html5/thumbnails/20.jpg)
Aug. 23, 2000 ISSTA 2000 20
Example of Composition
Button to turn on light
Each piece has one+ contracts
Button LightIButtonEvents
ILightButton-to-Light Adapter
Desc.: Button, Adapter+stuff, Light, B->A, A->L
IPushIt
![Page 21: Progress in Testing Component-Based Software Craig H. Wittenberg Component Applications Group Microsoft Research.](https://reader036.fdocuments.net/reader036/viewer/2022081519/56649ef75503460f94c0abf5/html5/thumbnails/21.jpg)
Aug. 23, 2000 ISSTA 2000 21
Encapsulation of the Switch
Button LightIButtonEvents
ILightEdge Conn.
Button-to-Light Adapter
Edge Connector for lookup/connection(role)
Desc.: Button, Adapter+stuff, Light, B->A, A->L, EC(out B, in A)
IPushIt
![Page 22: Progress in Testing Component-Based Software Craig H. Wittenberg Component Applications Group Microsoft Research.](https://reader036.fdocuments.net/reader036/viewer/2022081519/56649ef75503460f94c0abf5/html5/thumbnails/22.jpg)
Aug. 23, 2000 ISSTA 2000 22
Granularity of classes
Leaf classesUp to 1500 lines of C
Probably less in Java or C#1-30 classes per component DLL
Static compositesNormal range: 5-10 instancesPackaged like a leaf classConnection description very cheap
![Page 23: Progress in Testing Component-Based Software Craig H. Wittenberg Component Applications Group Microsoft Research.](https://reader036.fdocuments.net/reader036/viewer/2022081519/56649ef75503460f94c0abf5/html5/thumbnails/23.jpg)
Aug. 23, 2000 ISSTA 2000 23
Testing Open Systems
Late compositionUsually after component shippedRobustness of composite related to piecesSystem testing may be done by the user Need to support in-the-field debugging
Unit testing is criticalMust strive for more than the 80% caseFinding evidence of correctness
![Page 24: Progress in Testing Component-Based Software Craig H. Wittenberg Component Applications Group Microsoft Research.](https://reader036.fdocuments.net/reader036/viewer/2022081519/56649ef75503460f94c0abf5/html5/thumbnails/24.jpg)
Aug. 23, 2000 ISSTA 2000 24
Testing Components
Verify contract as implemented Check pre/post conditions Expose contract model variables Composites have contracts too
Verify contracts consumed Addresses some integration issues Fault injection
Test and measure the binary formAllow feedback to improve contract, impl., etc.
![Page 25: Progress in Testing Component-Based Software Craig H. Wittenberg Component Applications Group Microsoft Research.](https://reader036.fdocuments.net/reader036/viewer/2022081519/56649ef75503460f94c0abf5/html5/thumbnails/25.jpg)
Aug. 23, 2000 ISSTA 2000 25
What is a robust component?
Beizer on statement and branch coverage: “This is the weakest criterion in the family: testing less than this for new software is unconscionable and should be criminalized.”For us: Contract coverage 100% coverage at the object level
Coverage of all branches in sub-expressions Ad hoc approach to selecting paths
Code review
![Page 26: Progress in Testing Component-Based Software Craig H. Wittenberg Component Applications Group Microsoft Research.](https://reader036.fdocuments.net/reader036/viewer/2022081519/56649ef75503460f94c0abf5/html5/thumbnails/26.jpg)
Aug. 23, 2000 ISSTA 2000 26
Test Sequencer
Contract Scenarios
Contract Scenarios
Test Subject
State Mapping Objects
State Mapping Objects
Testing Framework
Dependencies
Pre/Post Checks
(delegator)
Pre/Post Checks
(delegator)
![Page 27: Progress in Testing Component-Based Software Craig H. Wittenberg Component Applications Group Microsoft Research.](https://reader036.fdocuments.net/reader036/viewer/2022081519/56649ef75503460f94c0abf5/html5/thumbnails/27.jpg)
Aug. 23, 2000 ISSTA 2000 27
Testing a Light
ILight Scenarios
LightMap
ILight model to
Impl.
ILightPre/Post Checks
Test Sequencer
![Page 28: Progress in Testing Component-Based Software Craig H. Wittenberg Component Applications Group Microsoft Research.](https://reader036.fdocuments.net/reader036/viewer/2022081519/56649ef75503460f94c0abf5/html5/thumbnails/28.jpg)
Aug. 23, 2000 ISSTA 2000 28
Testing a Switch
Switch Scenarios
SwitchMap
model to Impl.
SwitchPre/Post Checks
Test Sequencer
LightILight
Checks
![Page 29: Progress in Testing Component-Based Software Craig H. Wittenberg Component Applications Group Microsoft Research.](https://reader036.fdocuments.net/reader036/viewer/2022081519/56649ef75503460f94c0abf5/html5/thumbnails/29.jpg)
Aug. 23, 2000 ISSTA 2000 29
Production Process
One small team per componentDeliverables include Code for one component DLLDocumentationAny new or updated tests
Different levels of hardnessFactored, versioned build systemAudit at end
![Page 30: Progress in Testing Component-Based Software Craig H. Wittenberg Component Applications Group Microsoft Research.](https://reader036.fdocuments.net/reader036/viewer/2022081519/56649ef75503460f94c0abf5/html5/thumbnails/30.jpg)
Aug. 23, 2000 ISSTA 2000 30
Revisions
Bugs still happen
Usually modify and release new component
In rare cases, release in place
![Page 31: Progress in Testing Component-Based Software Craig H. Wittenberg Component Applications Group Microsoft Research.](https://reader036.fdocuments.net/reader036/viewer/2022081519/56649ef75503460f94c0abf5/html5/thumbnails/31.jpg)
Aug. 23, 2000 ISSTA 2000 31
Cost of Development
KLOC #People #Months #Classes Ship
TestComposition RT 2 5 5 4 9Connection Helper 1 2 1 1 6Collections 3 6 6 6 45Test framework 1 24 8Deployment RT 3.5 26 31 42 63
Totals 145 43 53 131
KLOC / year “shipped” = 4.4
KLOC / class = 1.2
![Page 32: Progress in Testing Component-Based Software Craig H. Wittenberg Component Applications Group Microsoft Research.](https://reader036.fdocuments.net/reader036/viewer/2022081519/56649ef75503460f94c0abf5/html5/thumbnails/32.jpg)
Aug. 23, 2000 ISSTA 2000 32
The Road to 100% Coverage
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
90.00%
100.00%
110.00%
9/21
/199
9
10/1
5/19
99
10/2
1/19
99
10/2
8/19
99
11/3
0/19
99
12/9
/199
9
12/1
4/19
99
12/2
0/19
99
1/7/
2000
1/13
/200
0
1/21
/200
0
1/28
/200
0
2/3/
2000
2/11
/200
0
![Page 33: Progress in Testing Component-Based Software Craig H. Wittenberg Component Applications Group Microsoft Research.](https://reader036.fdocuments.net/reader036/viewer/2022081519/56649ef75503460f94c0abf5/html5/thumbnails/33.jpg)
Aug. 23, 2000 ISSTA 2000 33
Interesting Bugs
Code and test made same bug
Coverage of last branch yielded bugs, twice
Post-condition in test was incomplete
Several difficult MT bugs
Shutdown of complex structures
![Page 34: Progress in Testing Component-Based Software Craig H. Wittenberg Component Applications Group Microsoft Research.](https://reader036.fdocuments.net/reader036/viewer/2022081519/56649ef75503460f94c0abf5/html5/thumbnails/34.jpg)
Aug. 23, 2000 ISSTA 2000 34
Future Work
Reducing Cost
Testing Composites
Increasing Coverage
Contract Coverage
Contract Specification
![Page 35: Progress in Testing Component-Based Software Craig H. Wittenberg Component Applications Group Microsoft Research.](https://reader036.fdocuments.net/reader036/viewer/2022081519/56649ef75503460f94c0abf5/html5/thumbnails/35.jpg)
Aug. 23, 2000 ISSTA 2000 35
Reducing Cost
Generate pre and post condition delegators Involves specification in contract and in
implementation
Apply consistency checks to contractsSelect test cases based on coverage goalsTesting generics
![Page 36: Progress in Testing Component-Based Software Craig H. Wittenberg Component Applications Group Microsoft Research.](https://reader036.fdocuments.net/reader036/viewer/2022081519/56649ef75503460f94c0abf5/html5/thumbnails/36.jpg)
Aug. 23, 2000 ISSTA 2000 36
Reducing Cost, cont.
Test specification language
Multi-threaded componentsTrack low level state and locks held for
detecting race conditions
Compiler-generated codeReducing noise in output
Use a GC runtime
![Page 37: Progress in Testing Component-Based Software Craig H. Wittenberg Component Applications Group Microsoft Research.](https://reader036.fdocuments.net/reader036/viewer/2022081519/56649ef75503460f94c0abf5/html5/thumbnails/37.jpg)
Aug. 23, 2000 ISSTA 2000 37
Testing Compositions
Static ones are like a componentUse composition description to helpLook for new paths not executed by unit testDomain-specific GUI (composition of bits on screen) Network-based (different failure modes) Hardware/software composites
Define contract for composite and try to show correspondence with structure
![Page 38: Progress in Testing Component-Based Software Craig H. Wittenberg Component Applications Group Microsoft Research.](https://reader036.fdocuments.net/reader036/viewer/2022081519/56649ef75503460f94c0abf5/html5/thumbnails/38.jpg)
Aug. 23, 2000 ISSTA 2000 38
Achieving More Coverage
Loops
More paths through cleanup code
Choosing other paths to measureE.g., D-U paths in code or in state diagram
![Page 39: Progress in Testing Component-Based Software Craig H. Wittenberg Component Applications Group Microsoft Research.](https://reader036.fdocuments.net/reader036/viewer/2022081519/56649ef75503460f94c0abf5/html5/thumbnails/39.jpg)
Aug. 23, 2000 ISSTA 2000 39
Contract Coverage
Sequencing testsState walking; choosing paths is keyGenetic algorithm
Test case correlation Track variations generated from specs with
actual tests
![Page 40: Progress in Testing Component-Based Software Craig H. Wittenberg Component Applications Group Microsoft Research.](https://reader036.fdocuments.net/reader036/viewer/2022081519/56649ef75503460f94c0abf5/html5/thumbnails/40.jpg)
Aug. 23, 2000 ISSTA 2000 40
Contract Specification Issues
Reentrancy, eventsNeed a description of the state of an
instance when an event handler is calledE.g., when button fires events to adapter
State mutation described in specCan use that to help direct MT testing
![Page 41: Progress in Testing Component-Based Software Craig H. Wittenberg Component Applications Group Microsoft Research.](https://reader036.fdocuments.net/reader036/viewer/2022081519/56649ef75503460f94c0abf5/html5/thumbnails/41.jpg)
Aug. 23, 2000 ISSTA 2000 41
Related Work
Brooks, Wirth, Parnas: various
Gall: Systemantics
Petroski: Designs and Errors
Williams: Inheritance + QI
Meyer: Design by Contract
Kiczales: Open Implementation
Harel, Wegner: Turing revisited
![Page 42: Progress in Testing Component-Based Software Craig H. Wittenberg Component Applications Group Microsoft Research.](https://reader036.fdocuments.net/reader036/viewer/2022081519/56649ef75503460f94c0abf5/html5/thumbnails/42.jpg)
Aug. 23, 2000 ISSTA 2000 42
Related Work, cont.
CMU, Nierstrasz: Composition
Gamma, et. al., Coplien: Patterns
D’Souza & Wills: Catalysis
Beizer, RST Corp: Testing
Beck and others: Extreme Prog.
Holmes, et. all: Synchronization
Szyperski: Beyond OOP
![Page 43: Progress in Testing Component-Based Software Craig H. Wittenberg Component Applications Group Microsoft Research.](https://reader036.fdocuments.net/reader036/viewer/2022081519/56649ef75503460f94c0abf5/html5/thumbnails/43.jpg)
Aug. 23, 2000 ISSTA 2000 43
Summary
Complex, robust, evolving software systems
Component and composite approach
Contracts and testing
Need better tools!
http://research.microsoft.com/comapps