Effects of Using Examples on Structural Model Comprehension:
A Controlled Experiment
Dina Zayan, Michał Antkiewicz, Krzysztof Czarnecki
2014.06.06
http://gsd.uwaterloo.ca http://necsis.ca
2
Many problems in SE are related to domain knowledge transfer
3
For example…
• Business analysts communicating to other stakeholders who are novices
• Industry standard which contains domain models applied by novices
• Specifications written by experts given to suppliers
4
Why do we have these problems?
Domain Knowledge
Complex in nature
Difficult to articulate
Difficult to communicate
5
Let’s run a mini experiment
6
In a “second degree price discrimination” strategy, price varies according to the quantity demanded. Larger quantities are available at a lower unit price. Additionally, sellers applying
this strategy are not able to differentiate between different types of consumers. Thus, the suppliers will provide incentives for the
consumers to differentiate themselves according to preference. This also applies to
product quality.
This is an Abstraction
Airlines often offer multiple classes of seats on flights, such as first class and economy class.
This is an Example
7
Examples + Abstractions
Effective Knowledge Transfer
Results in Cognitive Psychology and Software Engineering
8
Modeling
Model = Abstractions + Examples
Example-Driven
9
Of Course Examples Help!
+ Capture intuition+ Relate to existing knowledge+ Show the intended usage of an abstraction
10
However…
1. When exactly do examples help?2. How much can they help?3. What types of examples help?4. Which stakeholders benefit the most
from examples?5. Where is a PROOF that they help?
11
Provide the Missing Empirical Evidence and Guidelines
Objective
12
“Analyze the effects of using explicit examples
on structural model comprehension
in the context of knowledge transfer to model receptors
who are novices to the application domain.”
Controlled Experiment
13
Two Groups of Participants
EDM Control
Model Abstractions
Model Abstractions
Experimental Procedures & Requirements
List
Experimental Procedures & Requirements
List
Examples!
(13) (12)
14
Instrumentation & Actors Map
Experimental Procedures &
Requirements List
Model Abstractions
SME Model Creator
Examples!
15
Treatments and Metrics
Sobeys Object Diagram
EDM
Control
16
1723 classes, 34 associations, and 8 constraints
18
(EDM Only)
4 positive and 2 negative examples
19
Treatments and Metrics
Sobeys Object Diagram
1) Diagram Completeness
EDM
Control
20
Diagram Completeness
• Mean (Control) = 49.2 %• Mean (EDM) = 88.1 %• p-value < 0.0002• Maximum score for
Control is lower than the minimum score for the EDM
+ 37% increase
21
Treatments and Metrics
Sobeys Object Diagram
1) Diagram Completeness2) Diagram Mistakes
EDM
Control
22
Diagram Mistakes• Mean (Control) = 15.1
mistakes• Mean (EDM) = 3.5
mistakes• p-value < 0.0053• Maximum number of
mistakes for EDM is equal to the minimum number of mistakes for Control
- 80% decrease
23
Treatments and Metrics
Sobeys Object Diagram
1) Diagram Completeness2) Diagram Mistakes3) Perceived Difficulty4) Perceived Confidence
EDM
Control
24
Participants Rated Themselves
Difficulty
1 (very easy) to 5 (very difficult)
– Control: 3.83– EDM: 2.46
-27% decrease
Confidence
1 (unsure) to 5 (very sure)
– Control: 2.33– EDM: 4.08
+ 35% increase
25
Treatments and Metrics
Study Questions
Sobeys Object Diagram
1) Diagram Completeness2) Diagram Mistakes3) Perceived Difficulty4) Perceived Confidence
EDM
Control
26
27
28
Treatments and Metrics
Study Questions
Sobeys Object Diagram
1) Diagram Completeness2) Diagram Mistakes3) Perceived Difficulty4) Perceived Confidence
1) Task 2 Completeness
EDM
Control
29
Task 2 Completeness
• Mean (Control) = 13.6• Mean (EDM) = 21.2• p-value < 0.0012
+ 31% increase
Max 24
30
Treatments and Metrics
Study Questions
Sobeys Object Diagram
1) Diagram Completeness2) Diagram Mistakes3) Perceived Difficulty4) Perceived Confidence
1) Task 2 Completeness2) Efficiency
EDM
Control
31
Task 2 Efficiency
• Mean (Control) = 0.7• Mean (EDM) = 1.2• p-value = 0.00082
+ 71% increase
32
Treatments and Metrics
Study Questions
Sobeys Object Diagram
1) Diagram Completeness2) Diagram Mistakes3) Perceived Difficulty4) Perceived Confidence
1) Task 2 Completeness2) Efficiency3) Perceived Difficulty4) Perceived Confidence
EDM
Control
33
Participants Rated Themselves
Difficulty
1 (very easy) to 5 (very difficult)
– Control: 3.25– EDM: 2.08
-23.5% decrease
Confidence
1 (unsure) to 5 (very sure)
– Control: 3.00– EDM: 4.15
+ 23% increase
34
Treatments and Metrics
Study Questions
Sobeys Object Diagram
Participants Questions
1) Diagram Completeness2) Diagram Mistakes3) Perceived Difficulty4) Perceived Confidence
1) Task 2 Completeness2) Efficiency3) Perceived Difficulty4) Perceived Confidence
1) Number of domain-related questions
EDM
Control
35
Number of Domain-Related Questions
• Control: 10 questions/session
• EDM: 1 question/session
- 90% reduction
36
Treatments and Metrics
Study Questions
Sobeys Object Diagram
Participants Questions
Feedback
1) Diagram Completeness2) Diagram Mistakes3) Perceived Difficulty4) Perceived Confidence
1) Task 2 Completeness2) Efficiency3) Perceived Difficulty4) Perceived Confidence
1) Number of Domain-related Questions
1) UML Class Diagram2) Examples
EDM
Control
37
Understanding Without Examples?
EDM: “The class diagram was huge, I didn’t know where
to start reading to understand, so I looked at the examples one by one while looking at the model abstractions simultaneously.”
38
Classes Most Difficult To Comprehend
Class Control EDM Mistakes Control/EDM
Misses Control/EDM
BonusMechanic ✔ ✔ 5/2 4/0Redemption Mechanic
✔ ✔ 9/0 3/0
“Partner” Offer ✔ x 6/0 3/0Bill ✔ ✔ 9/3 0/0
39
40
How Were the Examples Helpful?
• Example 2: • Regular product vs. product with an offer associated• Application of the BonusMechanic class
• Example 4: • Propagation of points across classes and the
sequence for accumulating points• Account’s “pointsBalance” updated with Bill’s
“totalPoints” attribute
41
42
How Were the Examples Helpful?
• Example 5:• Redemption process
• Example 6:• Violation of the constraint prohibiting accumulation and
redemption for the same bill instance
43
Are Both Positive and Negative Examples Needed?
• “Yes” 12/13 EDM participants
• No comments about the number and content of examples
44
Discussion
46
Examples are mostly needed by Novices
• FixedPriceOff vs. FixedPercentOff– familiar
• RedemptionMechanic, RedemptionChannel, and BonusMechanic – foreign
47
Having a Variety of Examples
Useful when1. Having more than one association linked to a
single class
48
49
Having a Variety of Examples
Useful when1. Having more than one association linked to a
single class2. Complex constraints
50
51
Having a Variety of Examples
Useful when1. Having more than one association linked to a
single class2. Complex constraints3. Propagation of attributes through classes
52
53
Having a Variety of Examples
Useful when1. Having more than one association linked to a
single class2. Complex constraints3. Propagation of attributes through classes4. A domain concept that is spread over several
classes and associations (Points Accumulation)
54
EDM participant who preferred valid examples only answered the related question incorrectly
55
Partial Examples
• 12 participants– partial examples, no complete object diagram
• One participant– partial examples and complete object diagram
56
Conclusions
57
Quantitative Analysis
Dependent Variable
Significant? (p-value)
Hyptheses Testing
% Improvement due to the use of
examplesDiagram Completeness
Yes (<0.0002) Accept H1 +39%
Diagram Mistakes
Yes (<0.0053) Accept H2 -80%
Task 2 Completeness
Yes (<0.0012) Accept H3 +31%
Task 2 Efficiency
Yes (=0.00082) Accept H4 +71%
58
Qualitative Analysis
Dependent Variable % Improvement due to the use of examples
Perceived difficulty in creating the object diagram.
-27%
Perceived difficulty in answering Task 2 questions.
-23%
Confidence in their created object diagrams.
+35%
Confidence in the answers they provided for Task 2.
+23%
59
Limitations & Future Work
• Partial examples as views on a complete example
• Variety of examples vs. positive only
• Cost of creating and maintaining the examples vs. the benefit
• Empirical studies in an industrial context
Thank You!
Replication packageis available for download
http://gsd.uwaterloo.ca http://necsis.ca
Examples!
61
Example-Driven Modeling Using
• Unified syntax for abstractions and examples + redefinition
• Partial examples + automatic completion and constraint checking
• Specialization + extension
http://clafer.org
Top Related