Design Principles of Advanced Task Elicitation Systems
-
date post
21-Oct-2014 -
Category
Documents
-
view
822 -
download
1
description
Transcript of Design Principles of Advanced Task Elicitation Systems
Chair of Information Systems IV (ERIS) Institute for Enterprise Systems (InES)
Karlsruhe, November 30th 2012
Prof. Dr. Alexander MädcheChair of Information Systems IV, Business School and Institute for Enterprise Systems (InES), University of Mannheimhttp://eris.bwl.uni-mannheim.dehttp://ines.uni-mannheim.de
Design Principles of Advanced Task Elicitation Systems
(*) Joint work with: H. Meth, Y. Li, B. Mueller.
(*)
2Agenda
Agenda
1 Introduction
2 Related Work
3 Methodology
4 Exploring and Evaluating Design Principles
5 Discussion, Future Research & Summary
Motivation 3
Failure rate of software development projects is still high.
Driven by private life software usage the user expectations are growing.
Understanding the requirements remains the major challenge: 35 % of requirements change throughout the software
lifecycle (Jones, 2008) 45 % of delivered features are never used.
(Standish Report, 2009) 82 % of projects cited incomplete and unstable requirements
as the number one reason for failure (Taylor, 2000)
Analysis Phase
Engineering Phase
Continuous stakeholder integration, cross-functional teams as well as incremental & artifact-driven development
State-of-the-Art in Software Development 4
Analysis Phase
Analysis Phase
Human ComputerInteraction
Requirements Engineering Software
Engineering
IS Development
Focus of this talk 5
Approximately 80% of the requirements are recorded in natural language (Mich et al. 2004; Neill and Laplante 2003): Interview transcripts, Workshop nemos, Narrative scenarios
In large-scale development, manual requirements elicitation is known to be time-consuming, error-prone, and monotonous.
The study by Mich et al. (2004) on current elicitation practices explicates the need for advanced support with specific focus on automation.
6Agenda
Agenda
1 Introduction and Motivation
2 Related Work
3 Methodology
4 Exploring and Evaluating Design Principles
5 Discussion, Future Research & Summary
Basic Definitions 7
Requirements elicitation is the process of discovering requirements through direct interaction with stakeholders or analysis of documents or other sources of information (Ratchev et al. 2003).
A core activity in this process is the identification of relevant tasks to be supported by the software, referred to as task elicitation (also task analysis) (Lemaigre et al. 2008; Paterno 2002).
Task elicitation aims at capturing the interaction between user and system on a detailed level, differentiating between actors, activity, and data (Tam et al. 1998).
Related Work
Various attempts for advancing task elicitation by specialized task elicitation systems (TES) have been made, two major research streams:
8
Requirements Engineering • Identification of abstractions (Gacitua et al. 2011; Goldin and
Berry 1997; Kof 2004; Rayson et al. 2000) • Identification and classification of requirements (Cleland-Huang
et al. 2007; Casamayor et al. 2010; Kiyavitskaya and Zannone 2008)
• Create requirements and design model (Ambriola and Gervasi 2006)
Human Computer Interaction• Automate task elicitation with artifacts, e.g. U-TEL (Tam et al.
1998) or the model elicitation tool (Lemaigre et al. 2008)
1
2
Pattern: Leverage
automation techniques
and knowledge
bases
Related Work
Existing work has three major shortcomings: Manual creation of knowledge bases Lacking systematic empirical evaluation of productivity effects Limited explanation of artifact’s conceptualization
Research Question addressing this gap: Which design principles of task elicitation systems
improve task elicitation productivity over manual task elicitation?
9
10Agenda
Agenda
1 Introduction and Motivation
2 Related Work
3 Methodology
4 Exploring and Evaluating Design Principles
5 Discussion, Future Research & Summary
Methodology 11
Research question aims at the acquisition of theoretical design knowledge about task elicitation systems.
Design Science Research as proposed by March & Smith (1995) is an applicable and appropriate approach to address the research question.
Hevner et al (2004)
Research Design 12
Awareness of Problem
Suggestion
Development
Evaluation
Conclusion
Operation and Goal
Knowledge
General Design Science Cycle Cycle1 Cycle2 Cycle3
ExperimentEvaluation
Artifact FinalVersion
Design Principles
Expert Evaluation Focus: Usefulness
Artifact Concept Version
(Click-Through)
Analysis & Conceptualization
Literature Review,Expert Interviews
Expert Evaluation Focus: Ease of use
Artifact Prototype Version (First
Implementation)
Literature Review,Expert Feedback
(Meth et al. 2012a)
DSR project builds and evaluates an artifact to support task elicitation from natural language documents, guided by the Design Science framework suggested by Kuechler & Vaishnavi (2008):
13Agenda
Agenda
1 Introduction and Motivation
2 Related Work
3 Methodology
4 Exploring and Evaluating Design Principles
5 Discussion, Future Research & Summary
Justificatory Knowledge 14
The tool-supported task elicitation process can been seen as a series of advice-giving and advice-taking tasks (Bonaccio and Dalal 2006). An increase of the advisor’s advice accuracy has been found to
result in an increasing decision accuracy (of the advice-taker). Productivity improvement will only occur if the quality of approved requirements (the decision which has been taken) improves.
The underlying knowledge base influences the results of the advice-giving process (Casamayor et al. 2010): Leverage existing knowledge and enable continuous evolution of
knowledge base.
Conceptualization 15
DP1. Semi-Automatic Task
Elicitation
DP2. Usage of imported and
retrieved knowledge
DR1. Increase quality of approved
requirements
DR2. Decrease Elicitation Effort
DR3. Increase quality of underlying knowledge
DR4. Decrease knowledge creation and maintenance
efforts
DF1. Pre-Processing & Elicitation Algorithms
DF2. One-click Task Element Highlighting
DF3. Integrated Knowledge Base
DF4. Supervised Knowledge
Supplementation
Mapping Design-Requirements (DR) to Design Principles (DP) to Design Features (DF):
Conceptual Architecture 16
Knowledge Base
Imported Knowledge
Requirements Engineer
Manual Knowledge Creation
KnowledgeEngineer
Manual Elicitation
AutomaticElicitation
ElicitationAlgorithm
Retrieved Knowledge
Pre-ProcessingAlgorithm
Natural language
documentsText brick
Text brick
Category
Category
Text brick
Text brick
Category
Category
Automatic Knowledge
Creation
Text brick
POS Tag
Text brick
POS Tag
POS Tag
POS Tag
Artifact REMINER: Semi-Automatic Task Elicitation 17
(Meth et al. 2012a)
DP1. Semi-Automatic
Task Elicitation
DP2. Usage of imported
and retrieved
knowledge
MR1. Enable automatic task elicitation within natural language
documents
MR2. Allow manual
adaptions of automatically elicited tasks
MR3. Require minimal efforts to
build up task knowledge
MR4. Support simple
supplementation of domain-
specific knowledge
DF1. One-click Task Element Highlighting
DF2. Natural Language
Processing Capabilities
DF3. Knowledge Upload Capability
DF4. Knowledge Retrieval and Re-
Use
Online available at: http://www.reminer.com/
Artifact REMINER: Imported and Retrieved Knowledge 18
Retrieve & Re-Use
Upload
DP1. Semi-Automatic Task
Elicitation
DP2. Usage of imported and
retrieved knowledge
MR1. Enable automatic task elicitation within natural language
documents
MR2. Allow manual adaptions of automatically elicited tasks
MR3. Require minimal efforts to
build up task knowledge
MR4. Support simple supplementation of
domain-specific knowledge
DF1. One-click Task Element Highlighting
DF2. Natural Language
Processing Capabilities
DF3. Knowledge Upload Capability
DF4. Knowledge Retrieval and Re-Use
Evaluation Methodology 19
Controlled within-subject experiment to rigorously test the effect of two design principles (DP1, DP2) on task elicitation productivity.
Experimental task: task elicitation with interview transcripts Task domain: Travel Management Similar length, readability, and the distribution of task elements
Sample size calculation: Calculated with G*Power 3 (Faul et al., 2007), at least 35 participants
are needed (f =0,25, 0.05 significance level)
Participants:
(Meth et al. 2012b)
Student sample (Lab)(N= 40)
Practitioner sample (Field) (N=5)
Gender Female 8 2
Male 32 3Avg. age 25.4 (SD=2.07) 34.8 (SD=3.56)
Evaluation Model 20
(Meth et al. 2012b)
Recall
Precision
H1,H2
Task Elicitation Productivity
(in a fixed time period)
H3
Task Elicitation System (TES) Configuration
(1,2,3)
H1: In a fixed time period, TES configuration (2) results in higher recall than TES configuration (1)
H2: In a fixed time period, TES configuration (3) results in higher recall than TES configuration (2)
H3: In a fixed time period, TES configuration (1), (2) and (3) does NOT result in significantly different precision
Experimental Procedure 21
Introduction
Pre-task questionnaire
Training & Practice
Experimental task
Post-task questionnaire
Demographic information, task elicitation experience
Use transcripts about “car sharing application”; 3 TES configurations,
counterbalanced
Task elicitation knowledge, motivation
Use transcripts about “train reservation application”
3 times
Overall: 70 minutes
Data Analysis: Descriptive Results
Data analysis method Internal reliability, normality and homogeneity of variance checked RMANCOVA: “Task elicitation knowledge” and “motivation” are not
covariates Univariate RMANOVA for hypotheses testing
22
Recall and Precision for Different TES Configurations
(1) Manual (2) Semi-automatic with
imported knowledge
(3) Semi-automatic with imported and retrieved
knowledge
Mean SD Mean SD Mean SD
Lab experiment (student participants, N=40)
Recall 50.7% 12.0% 69.8% 9.8% 79.5% 8.0%
Precision 71.0% 8.5% 72.0% 6.7% 73.2% 6.5%
Field experiment (practitioner participants, N=5)
Recall 37.6% 12.9% 68.6% 6.0% 77.8% 3.9%
Precision 70.1% 14.5% 72.7% 3.5% 68.5% 5.3%
Data Analysis: Hypotheses Testing Results
External validity evaluation: the practitioner sample doesn’t demonstrate a different behavioral pattern on recall and precision.
23
Results of RMANOVA for Recall and PrecisionDV Source DF MS F p η2 Cohen’s f
RecallTES Config. 2 0.861 129.76 < .001 .77 1.82
Error 78 0.007
PrecisionTES Config. 2 0.005 1.36 .263 .03 0.19
Error 78 0.004
Results of Pairwise Comparisons for Recall
Pair comparison Mean difference p*
95% CI*
Lower Upper
TES config. (2) TES config. (1) 19.2% < .001 14.4% 23.9%
TES config. (3) TES configur. (2) 9.7% < .001 5.8% 13.6%
* Bonferroni corrections are applied for multiple comparisons
H3: supported
H1: supportedH2: supported
Huberty & Morris (1989)
24Agenda
Agenda
1 Introduction and Motivation
2 Related Work
3 Methodology
4 Exploring and Evaluating Design Principles
5 Discussion, Future Work & Summary
Discussion 25
Design principles DP1 and DP2 impact recall: Suggestion mechanism based on imported knowledge leads to 20%
recall increase: Trust recommendations and increase recall through further manual elicitation of additional tasks in remaining time.
Dynamically retrieved knowledge leads to additional 10% recall increase: Continuous contribution of additional knowledge through ongoing manual elicitation.
Limitations Limited complexity of task domain and time-constraint evaluation
approach. Laboratory sessions were conducted with master IS students, only
small-scale experiment was carried out with experts.
Future Research 26
Presented work contributes to the design theory body of knowledge for task elicitation in the analysi phase.
Interdisciplinary perspective is promising, research on task elicitation needs to be embedded:
Process Models &Management
Concepts
End-to-EndDevelopment
Tools Analysis Phase
Engineering Phase
Analysis Phase
Analysis Phase
http://www.usability-in-germany.de/
Example: From Task Elicitation to Interaction Flows 27
(Meth et al. 2012a)
Summary 28
• Design principles of an advanced task elicitation system following a design science research approach have been presented.1
• Rigorous experimental evaluation has shown that semi-automatic and knowledge-based elicitation has positive impact on elicitation productivity; 2
• Contribution: The design theory body of knowledge for task elicitation systems has been expanded. Software vendors can leverage results to provide advanced tool-based elicitation support
3
Thank you for your attention! 29
Prof. Dr. Alexander Mädche+49 621 181 [email protected]
Chair of Information Systems IV, Business School and Institute for Enterprise Systems, University of Mannheimhttp://eris.bwl.uni-mannheim.dehttp://ines.uni-mannheim.de
Q & A
References
Neill, C. J., and Laplante, P. A. 2003. “Requirements Engineering: The State of the Practice,” IEEE Software (20:6), pp. 40-45.
Mich, L., Franch, M., and Novi Inverardi, P. L. 2004. “Market research for requirements analysis using linguistic tools,” Requirements Engineering (9:1), pp. 40-56.
Meth, H., Maedche, A., and Einoeder, M. 2012a. “Exploring design principles of task elicitation systems for unrestricted natural language documents,” Proceedings of the 4th ACM SIGCHI symposium on Engineering interactive computing systems - EICS ’12. New York, New York, USA: ACM Press, pp. 205 - 210.
Meth, H., Li, Y., Maedche, A., and Mueller, B. 2012b. “Advancing Task Elicitation Systems - An Experimental Evaluation of Design Principles,” In ICIS 2012 Proceedings.
Jones, C. 2008. Applied Software Measurement. McGraw Hill. Taylor, A. 2000. “IT projects: sink or swim.” The Computer Bulletin, 42 (1): 24-26. Standish Group Report 2009, http://
luuduong.com/blog/archive/2009/03/04/applying-the-quot8020-rulequot-with-the-standish-groups-software-usage.aspx
Bonaccio, S. and Dalal, R.S. (2006) “Advice taking and decision-making: An integrative literature review, and implications for the organizational sciences,” Organizational Behavior and Human Decision Processes (101: 2), pp. 127-151.
Hevner, A. R., March, S. T., Park, J., and Ram, S. (2004) “Design Science in Information Systems Research,” MIS Quarterly (28:1), pp. 75-105.
30
References (cont’d)
March, S. T., and Smith, G. F. 1995. “Design and natural science research on information technology,” Decision Support Systems (15:4), pp. 251–266.
Lemaigre, C., García, J. G., and Vanderdonckt, J. (2008) “Interface Model Elicitation from Textual Scenarios,” in Proceedings of the Human-Computer Interaction Symposium, 272, pp. 53-66.
Mich, L., Franch, M., and Novi Inverardi, P. L. (2004) “Market research for requirements analysis using linguistic tools,” Requirements Engineering (9:1), pp. 40-56.
Kuechler, B., and Vaishnavi, V. (2008) “On theory development in design science research: anatomy of a research project,” European Journal of Information Systems (17:5), pp. 489–504.
Ratchev, S. M., Urwin, E., Muller, D., Pawar, K. S., and Moulek, I. (2003) “Knowledge based requirement engineering for one-of-a-kind complex systems,” Knowledge Based Systems (16:1), pp. 1-5.
Paterno, F. (2002) “Task Models in Interactive Software Systems,” in Handbook of Software Engineering and Knowledge Engineering Vol 1 Fundamentals, S. K. Chang (ed.), World Scientific, pp. 1-19.
Tam, R. C.-man, Maulsby, D., and Puerta, A. R. (1998) “U-TEL: A Tool for Eliciting User Task Models from Domain Experts,” in Proceedings of the 3rd international conference on Intelligent user interfaces, pp. 77-80.
Faul, F., Erdfelder, E., Lang, A.-G. and Buchner, A. (2007) “G*Power 3: a flexible statistical power analysis program for the social, behavioral, and biomedical sciences.,” Behavior research methods 39(2), pp. 175-91.
31
References (cont’d)
Gacitua, R., Sawyer, P., and Gervasi, V. (2011) “Relevance-based abstraction identification: technique and evaluation,” Requirements Engineering (16:3), pp. 251-265.
Goldin, L., and Berry, D. M. (1997) “AbstFinder, A Prototype Natural Language Text Abstraction Finder for Use in Requirements Elicitation,” Automated Software Engineering (4:4), pp. 375-412.
Kof, L. (2004) “Natural Language Processing for Requirements Engineering: Applicability to Large Requirements Documents,” in Proceedings of the 19th International Conference on Automated Software Engineering.
Rayson, P., Garside, R., and Sawyer, P. (2000) “Assisting requirements engineering with semantic document analysis,” in Proceedings of the RIAO, pp. 1363-1371.
Cleland-Huang, J., Settimi, R., Zou, X., and Solc, P. (2007) “Automated classification of non-functional requirements,” Requirements Engineering (12:2), pp. 103-120.
Casamayor, A., Godoy, D., and Campo, M. (2010) “Identification of non-functional requirements in textual specifications: A semi-supervised learning approach,” Information and Software Technology (52:4), pp. 436-445.
Kiyavitskaya, N., and Zannone, N. (2008) “Requirements model generation to support requirements elicitation: the Secure Tropos experience,” Automated Software Engineering (15:2), pp. 149-173.
Ambriola, V., and Gervasi, V. (2006) “On the Systematic Analysis of Natural Language Requirements with CIRCE,” Automated Software Engineering (13:1), pp. 107-167.
Huberty, C. J. and Morris, J. D. (1989) “Multivariate analysis versus multiple univariate analyses.,” Psychological Bulletin 105(2), pp. 302-308.
32