Research Methods - An Introductionanu.brighid.idc.ul.ie/CS6022-2018/ResearchMethods/...Research...
Transcript of Research Methods - An Introductionanu.brighid.idc.ul.ie/CS6022-2018/ResearchMethods/...Research...
![Page 1: Research Methods - An Introductionanu.brighid.idc.ul.ie/CS6022-2018/ResearchMethods/...Research methods in Software Engineering p Controlled experiment Wohlin et al. 2000 p Survey](https://reader033.fdocuments.net/reader033/viewer/2022042322/5f0c00e67e708231d43347d7/html5/thumbnails/1.jpg)
Research Methods - An Introduction
Gabriela Avram
![Page 2: Research Methods - An Introductionanu.brighid.idc.ul.ie/CS6022-2018/ResearchMethods/...Research methods in Software Engineering p Controlled experiment Wohlin et al. 2000 p Survey](https://reader033.fdocuments.net/reader033/viewer/2022042322/5f0c00e67e708231d43347d7/html5/thumbnails/2.jpg)
What’s the Difference Between “Method” and “Methodology”?
Method: p Techniques for
gathering evidence p The various ways of
proceeding in gathering information
Methodology: p The underlying
theory and analysis of how research does or should proceed, often influenced by discipline
(Sandra Harding)
![Page 3: Research Methods - An Introductionanu.brighid.idc.ul.ie/CS6022-2018/ResearchMethods/...Research methods in Software Engineering p Controlled experiment Wohlin et al. 2000 p Survey](https://reader033.fdocuments.net/reader033/viewer/2022042322/5f0c00e67e708231d43347d7/html5/thumbnails/3.jpg)
Epistemology, Methodology, and Method
"a research method is a technique for (or way of proceeding in) gathering evidence" while "methodology is a theory and analysis of how research does or should proceed" and "an epistemology is a theory of knowledge"
Sandra Harding, “Is There a Feminist Method?”
![Page 4: Research Methods - An Introductionanu.brighid.idc.ul.ie/CS6022-2018/ResearchMethods/...Research methods in Software Engineering p Controlled experiment Wohlin et al. 2000 p Survey](https://reader033.fdocuments.net/reader033/viewer/2022042322/5f0c00e67e708231d43347d7/html5/thumbnails/4.jpg)
Epistemologyp Epistemology (theory of knowledge) is the branch of
philosophy concerned with the nature and scope (limitations) of knowledge; addresses the questions: ■ "What is knowledge?" ■ "How is knowledge acquired?" ■ "What do people know?" ■ "How do we know what we know?"
p Internalism vs externalism p A priori/a posteriori p Empiricism; Rationalism; Constructivism
![Page 5: Research Methods - An Introductionanu.brighid.idc.ul.ie/CS6022-2018/ResearchMethods/...Research methods in Software Engineering p Controlled experiment Wohlin et al. 2000 p Survey](https://reader033.fdocuments.net/reader033/viewer/2022042322/5f0c00e67e708231d43347d7/html5/thumbnails/5.jpg)
An Overview of Empirical Research Methods
Descriptive (Qualitative) p Ethnography p Case Study p Survey/Sampling p Focus Groups p Discourse/Text Analysis p Quantitative Description p Prediction/Classification
Experimental (Quantitative) p True Experiment p Quasi-Experiment p Meta-Analysis
From Lauer and Asher, Composition Research: Empirical Designs and MacNealy, Empirical Research in Writing
![Page 6: Research Methods - An Introductionanu.brighid.idc.ul.ie/CS6022-2018/ResearchMethods/...Research methods in Software Engineering p Controlled experiment Wohlin et al. 2000 p Survey](https://reader033.fdocuments.net/reader033/viewer/2022042322/5f0c00e67e708231d43347d7/html5/thumbnails/6.jpg)
Research methods in Software Engineering
p Controlled experiment ■ Wohlin et al. 2000
p Survey ■ Pfleger and Kitchenham 2001
p Case study ■ Yin 1994
p Ethnography p Data collection: Interviews, observation, documentation, ... p Design science / constructive research
■ Hevner et al. 2004, Kasanen et al. 1993 p Action / participatory research
■ Avison et al. 1999, Stringer 1999
![Page 7: Research Methods - An Introductionanu.brighid.idc.ul.ie/CS6022-2018/ResearchMethods/...Research methods in Software Engineering p Controlled experiment Wohlin et al. 2000 p Survey](https://reader033.fdocuments.net/reader033/viewer/2022042322/5f0c00e67e708231d43347d7/html5/thumbnails/7.jpg)
A different view –Research methods in IxD
Three ways of studying design (Jacob Buur, Mads Clausen Institute) ■ Ethnographic research - Observing actual practice ■ Action research - Experiments in practice - ■ Research through design - Experiments with artefacts
■ Critical design/ Speculative design
![Page 8: Research Methods - An Introductionanu.brighid.idc.ul.ie/CS6022-2018/ResearchMethods/...Research methods in Software Engineering p Controlled experiment Wohlin et al. 2000 p Survey](https://reader033.fdocuments.net/reader033/viewer/2022042322/5f0c00e67e708231d43347d7/html5/thumbnails/8.jpg)
Assessing Methodsp Research Question(s) is/are key p Methods must answer the research question(s) p Methodology guides application p Epistemology guides analysis p All must include “rigor”
![Page 9: Research Methods - An Introductionanu.brighid.idc.ul.ie/CS6022-2018/ResearchMethods/...Research methods in Software Engineering p Controlled experiment Wohlin et al. 2000 p Survey](https://reader033.fdocuments.net/reader033/viewer/2022042322/5f0c00e67e708231d43347d7/html5/thumbnails/9.jpg)
Formulating a research questionp The research question serves two purposes:
■ It determines where and what kind of research the writer will be looking for.
■ It identifies the specific objectives the study or paper will address.
p Qualitative or quantitative?
![Page 10: Research Methods - An Introductionanu.brighid.idc.ul.ie/CS6022-2018/ResearchMethods/...Research methods in Software Engineering p Controlled experiment Wohlin et al. 2000 p Survey](https://reader033.fdocuments.net/reader033/viewer/2022042322/5f0c00e67e708231d43347d7/html5/thumbnails/10.jpg)
Example of Research Question
p “What are the practices of cultural heritage professionals in small museums in preparing, creating and adapting museum activities, and how do digital technologies mediate those practices? “ (Laura Maye, PhD thesis, 2016)
![Page 11: Research Methods - An Introductionanu.brighid.idc.ul.ie/CS6022-2018/ResearchMethods/...Research methods in Software Engineering p Controlled experiment Wohlin et al. 2000 p Survey](https://reader033.fdocuments.net/reader033/viewer/2022042322/5f0c00e67e708231d43347d7/html5/thumbnails/11.jpg)
Experimental Research: True Experiment
+ Random sampling, or selection, of subjects (which are also stratified)
+ Introduction of a treatment + Use of a control group for comparing subjects who
don’t receive treatment with those who do - Adherence to scientific method (seen as positive,
too) - Must have both internal and external validity - Treatment and control might seem artificial
![Page 12: Research Methods - An Introductionanu.brighid.idc.ul.ie/CS6022-2018/ResearchMethods/...Research methods in Software Engineering p Controlled experiment Wohlin et al. 2000 p Survey](https://reader033.fdocuments.net/reader033/viewer/2022042322/5f0c00e67e708231d43347d7/html5/thumbnails/12.jpg)
Experimental Research: Quasi-Experiment
+ Similar to Experiment, except that the subjects are not randomized. Intact groups are often used (for example, students in a classroom).
+ To draw more fully on the power of the experimental method, a pretest may be employed.
+ Employ treatment, control, and scientific method - Act of control and treatment makes situation artificial - Small subject pools
![Page 13: Research Methods - An Introductionanu.brighid.idc.ul.ie/CS6022-2018/ResearchMethods/...Research methods in Software Engineering p Controlled experiment Wohlin et al. 2000 p Survey](https://reader033.fdocuments.net/reader033/viewer/2022042322/5f0c00e67e708231d43347d7/html5/thumbnails/13.jpg)
Positive Aspects of Experimental Research
p Tests the validity of generalizations p Seen as rigorous p Identifies a cause-and-effect relationship p Seen as more objective, less subjective p Can be predictive
![Page 14: Research Methods - An Introductionanu.brighid.idc.ul.ie/CS6022-2018/ResearchMethods/...Research methods in Software Engineering p Controlled experiment Wohlin et al. 2000 p Survey](https://reader033.fdocuments.net/reader033/viewer/2022042322/5f0c00e67e708231d43347d7/html5/thumbnails/14.jpg)
Problems with Experimental Researchp Generalizations need to be qualified according to
limitation of research methods employed p Controlled settings don’t mirror actual conditions;
unnatural p Difficult to isolate a single variable p Doesn’t allow for self-reflection (built-in)
![Page 15: Research Methods - An Introductionanu.brighid.idc.ul.ie/CS6022-2018/ResearchMethods/...Research methods in Software Engineering p Controlled experiment Wohlin et al. 2000 p Survey](https://reader033.fdocuments.net/reader033/viewer/2022042322/5f0c00e67e708231d43347d7/html5/thumbnails/15.jpg)
Survey Research+ An efficient means of gathering large amounts of
data + Can be anonymous and inexpensive - Feedback often incomplete - Wording of instrument can bias feedback - Details often left out
![Page 16: Research Methods - An Introductionanu.brighid.idc.ul.ie/CS6022-2018/ResearchMethods/...Research methods in Software Engineering p Controlled experiment Wohlin et al. 2000 p Survey](https://reader033.fdocuments.net/reader033/viewer/2022042322/5f0c00e67e708231d43347d7/html5/thumbnails/16.jpg)
Survey Research+ Important issues: + Data requirements + Data generation method + Sampling frame + Sampling technique + Response rate and non-responses + Sample size + Accuracy range and confidence level
![Page 17: Research Methods - An Introductionanu.brighid.idc.ul.ie/CS6022-2018/ResearchMethods/...Research methods in Software Engineering p Controlled experiment Wohlin et al. 2000 p Survey](https://reader033.fdocuments.net/reader033/viewer/2022042322/5f0c00e67e708231d43347d7/html5/thumbnails/17.jpg)
Case Studies+ Focus is on individual or small group + Able to conduct a comprehensive analysis from a
comparison of cases + Allows for identification of variables or phenomenon
to be studied - Multiple sources and methods - Time consuming - Depth rather than breadth - Not necessarily representative or generalizable
![Page 18: Research Methods - An Introductionanu.brighid.idc.ul.ie/CS6022-2018/ResearchMethods/...Research methods in Software Engineering p Controlled experiment Wohlin et al. 2000 p Survey](https://reader033.fdocuments.net/reader033/viewer/2022042322/5f0c00e67e708231d43347d7/html5/thumbnails/18.jpg)
Types of case studiesp Exploratory p Descriptive p Explanatory
Approach to time: p Historical p Short-term p Longitudinal
![Page 19: Research Methods - An Introductionanu.brighid.idc.ul.ie/CS6022-2018/ResearchMethods/...Research methods in Software Engineering p Controlled experiment Wohlin et al. 2000 p Survey](https://reader033.fdocuments.net/reader033/viewer/2022042322/5f0c00e67e708231d43347d7/html5/thumbnails/19.jpg)
Ethnographies+ Observational field work done in the actual
context being studied + Focus on how individuals interrelate in their own
environment (and the influence of this environment)
- Difficult to interpret/analyze - Time consuming/expensive - Can influence subject behavior
![Page 20: Research Methods - An Introductionanu.brighid.idc.ul.ie/CS6022-2018/ResearchMethods/...Research methods in Software Engineering p Controlled experiment Wohlin et al. 2000 p Survey](https://reader033.fdocuments.net/reader033/viewer/2022042322/5f0c00e67e708231d43347d7/html5/thumbnails/20.jpg)
Focus Groups+ Aid in understanding audience, group, users + Small group interaction more than individual
response + Helps identify and fill gaps in current knowledge re:
perceptions, attitudes, feelings, etc. - Does not give statistics - Marketing tools seen as “suspect” - Analysis subjective
![Page 21: Research Methods - An Introductionanu.brighid.idc.ul.ie/CS6022-2018/ResearchMethods/...Research methods in Software Engineering p Controlled experiment Wohlin et al. 2000 p Survey](https://reader033.fdocuments.net/reader033/viewer/2022042322/5f0c00e67e708231d43347d7/html5/thumbnails/21.jpg)
Discourse/Text Analysis+ Examines actual discourse produced for a
particular purpose (job, school) + Helps in understanding of context, production,
audience, and text + Schedule for analysis not demanding - Labor intensive - Categories often fluid, making analysis difficult
![Page 22: Research Methods - An Introductionanu.brighid.idc.ul.ie/CS6022-2018/ResearchMethods/...Research methods in Software Engineering p Controlled experiment Wohlin et al. 2000 p Survey](https://reader033.fdocuments.net/reader033/viewer/2022042322/5f0c00e67e708231d43347d7/html5/thumbnails/22.jpg)
Quantitative Descriptive Studies+ Isolate systematically the most important variables
(often from case studies) and to quantify and interrelate them (often via survey or questionnaire)
+ Possible to collect large amounts of data + Not as disruptive + Biases not as likely - Data restricted to information available
![Page 23: Research Methods - An Introductionanu.brighid.idc.ul.ie/CS6022-2018/ResearchMethods/...Research methods in Software Engineering p Controlled experiment Wohlin et al. 2000 p Survey](https://reader033.fdocuments.net/reader033/viewer/2022042322/5f0c00e67e708231d43347d7/html5/thumbnails/23.jpg)
Positive Aspects of Descriptive/Qualitative Research
p Naturalistic; allows for subjects to interact with environment
p Can use statistical analysis p Seeks to further develop theory (not to influence
action); Prescientific p Coding schemes often arise from interplay
between data and researcher’s knowledge of theory
![Page 24: Research Methods - An Introductionanu.brighid.idc.ul.ie/CS6022-2018/ResearchMethods/...Research methods in Software Engineering p Controlled experiment Wohlin et al. 2000 p Survey](https://reader033.fdocuments.net/reader033/viewer/2022042322/5f0c00e67e708231d43347d7/html5/thumbnails/24.jpg)
Problems with Descriptive/Qualitative Research
p Impossible to overlay structure p Impossible to impose control p Subject pool often limited, not representative p Seen as more “subjective,” less rigorous p Beneficial only in terms of initial investigation to
form hypothesis
![Page 25: Research Methods - An Introductionanu.brighid.idc.ul.ie/CS6022-2018/ResearchMethods/...Research methods in Software Engineering p Controlled experiment Wohlin et al. 2000 p Survey](https://reader033.fdocuments.net/reader033/viewer/2022042322/5f0c00e67e708231d43347d7/html5/thumbnails/25.jpg)
“Bovine design”
p Ann Light, Alison Powell, and Irina Shklovski. 2017. Design for Existential Crisis in the Anthropocene Age. In Proceedings of the 8th International Conference on Communities and Technologies (C&T '17). ACM, New York, NY, USA, 270-279. DOI: https://doi.org/10.1145/3083671.3083688