Post on 23-Dec-2015
Educational Research Methods
Dr. K. A. KorbUniversity of Jos
Introduction
• There is no point to analyzing data from a study that was not properly designed to answer the research question under investigation. In fact, there’s a real point in refusing to analyze such data less faulty results be responsible for implementing a program or policy contrary to what’s really needed.
• Two of the valuable things a researcher can possess are:
1. Knowledge of the principle of good study design 2. The courage to refuse to cut corners
(Dallal, 1998)
Dr. K. A. KorbUniversity of Jos
Scientific Method1: Ask
Question
2: Design Study
3: Collect Data
4: Analyze Results
5: Reach Conclusions
6: Share Findings
Dr. K. A. KorbUniversity of Jos
1. Ask a Question
• What is your general research question?• What is the independent variable in the
research question?• What is the dependent variable in the
research question?
Dr. K. A. KorbUniversity of Jos
2. Design the Study
• Research Design: Structure the research to show how the major parts of the research project work together to address the central research questions– Participants
• What is the population of people I am interested in?– Instruments
• How will I accurately measure my independent and dependent variables?
– Procedure• How will I collect data in an appropriate manner?
Dr. K. A. KorbUniversity of Jos
Outline
• Types of Research Methodologies– Descriptive– Causal Comparative– Correlation– Experimental– Quasi-Experiment
• Threats to valid Research Design
Dr. K. A. KorbUniversity of Jos
Descriptive
• Purpose: Carefully describe a naturally occurring educational phenomenon through systematic observation
• Key characteristic: Describes one (sometimes more) variable within a particular population
• Descriptive research designs are sometimes also called Survey Research Designs.
Dr. K. A. KorbUniversity of Jos
Descriptive
• Research hypotheses are impossible for descriptive designs. Only research questions should be posed.
• There are no independent or dependent variables, simply the key psychological variables that the researchers is interested in.
– Example questions:• What are teachers’ attitude towards a new moral educational
program?• What would counselors recommend be included in an HIV/AIDS
curriculum?• How many students engage in exam malpractice?
Dr. K. A. KorbUniversity of Jos
Descriptive
1. Select participants: Define the group that possess the variables you want to study
2. Data collection: Administer valid measures of the variables of interest
3. Data analysis: Compute descriptive statistics
Dr. K. A. KorbUniversity of Jos
Descriptive
Dr. K. A. KorbUniversity of Jos
Causal Comparative
• Purpose: Determine a cause/effect relationship where the independent variable cannot be manipulated
• Key characteristic: Compare two or more naturally-occurring groups on the dependent variable
• Causal Comparative designs can also be called ex post facto designs
Dr. K. A. KorbUniversity of Jos
Causal Comparative
• Null hypothesis: There is no significance difference between GROUPS on DEPENDENT VARIABLE.
• Independent variable: Group category• Dependent variable: What the groups are hypothesized to differ
on.– Examples:
• There is no significant differences between boys and girls on interest in mathematics.
• There is no significant differences between children from low and high socioeconomic status in the number of books read in a month.
• There is no significance differences between Nigeria and British children in academic self efficacy.
Dr. K. A. KorbUniversity of Jos
Causal Comparative
1. Select participants: Obtain a critical number of participants in each group
2. Data collection: Administered valid measures of your variables to all groups of participants
3. Data analysis: Calculate the mean score for each group on the dependent variable.– However, any difference in the mean score between
groups might be due to error.– Therefore, inferential statistics are necessary to
determine if the difference are significant.
Dr. K. A. KorbUniversity of Jos
Inferential Statistics– The inferential statistics to be depends on how many
groups are being compared• Two groups: t-test• Three or more groups: ANOVA
– If the value is greater than.05, retain the null hypothesis.• There is no significant difference between boys and girls on
interest in math.– If the p-value is less than .05, reject the null hypothesis.
• There is a significant difference between boys and girls on interest in math.– Look at the mean score to determine which group has the stronger
interest.
Dr. K. A. KorbUniversity of Jos
Causal Comparative
Dr. K. A. KorbUniversity of Jos
Correlational
• Purpose: Quantity the extent to which two variables are associated
• Key characteristic: The same group of participants are given measures of both key variables in order to calculate the correlation coefficient
• Because of the directionality and third variable problems, there are no independent or dependent variables in correlational designs
Dr. K. A. KorbUniversity of Jos
Correlational
• Null hypothesis: There is no significant relationship between VARIABLE 1 and VARIABLE 2.
• Examples– There is no significant relationship between number of
books a child reads at home and reading ability.– There is no significant relationship between WAEC scores
and university GPA.– There is no significant relationship between the amount of
time spent in lecture and frequency of exam malpractice.
Dr. K. A. KorbUniversity of Jos
Correlational
1. Select participants: A homogeneous sample2. Data collection: Administer valid measures
of the variables 3. Data Analysis: Calculate the correlation
coefficient and the significance test for correlation
Dr. K. A. KorbUniversity of Jos
Interpreting Correlations
• Nature– Positive: Two variables increase or decrease together– Negative: As one variable increases, the other decreases
• Strength– Closer to -1 or +1 is stronger relationship– 0 is no relationship
Negative PositiveNature:
Strength:
0-1 +1
Dr. K. A. KorbUniversity of Jos
Dr. K. A. KorbUniversity of Jos
Correlation = .78
-
10
20
30
40
50
60
70
80
90
100
0 2 4 6 8 10
Hours Studied
Exa
m S
core
.
Dr. K. A. KorbUniversity of Jos
Correlation = -.86
0.00
0.50
1.00
1.50
2.00
2.50
3.00
3.50
4.00
0 2 4 6 8 10
Hours per day watching TV
GP
A .
Dr. K. A. KorbUniversity of Jos
Correlation: .04
0.00
0.50
1.00
1.50
2.00
2.50
3.00
3.50
4.00
4.50
0.00 2.00 4.00 6.00 8.00 10.00 12.00 14.00
Number of Stamps on Passport
Gla
sses
of M
ilk
Consu
med
per
Day
.
Dr. K. A. KorbUniversity of Jos
Research Example
• The effect of regular leisure reading on reading achievement in primary school.
• Students in a primary school will be randomly assigned to either a treatment or a control group.– The treatment group will spend 10 minutes everyday reading
a book with an adult.– In an attempt to keep the treatment and control groups as
similar as possible, the control group will also spend 10 minutes with an adult everyday, but will instead do math problems.
Dr. K. A. KorbUniversity of Jos
Correlational
InterestAcademic
Achievement
Dr. K. A. KorbUniversity of Jos
Correlation does not prove causation.
InterestAcademic
Achievement
Directionality Problem
InterestAcademic
Achievement
Amount of time spent
studying
Third Variable Problem
Experimental
• Purpose: Establish cause and effect• Key characteristic: Statement about how one
variables affects (causes) another• Three essential components of an experiment– Treatment– Control– Random assignment to treatment and control groups
Dr. K. A. KorbUniversity of Jos
Experimental• Null hypothesis: There is no significant effect of
TREATMENT on DEPENDENT VARIABLE.– Independent variable: Treatment – Dependent variable: What the treatment should change
• Examples– There is no significant effect of a counseling intervention on
homework completion.• Treatment: Counseling
– There is no significance effect of reading a book a day on reading achievement.• Treatment: Reading a book a day
– There is no significant effective of an anti-exam malpractice campaign on the frequency of exam malpractice.• Treatment: Anti-exam malpractice campaign
Dr. K. A. KorbUniversity of Jos
Experimental
• Goal: Keep the experiences of the treatment and control groups as identical as possible except for the treatment in order to conclude that the change in the dependent variable is the result of the treatment.
Read a book
every dayReading
Achievement
Experimental Group
Reading Class
ReadingAchievement
Reading Class
Control Group
Dr. K. A. KorbUniversity of Jos
Experimental
• Random Assignment: Each subject has an equal chance of being assigned to either the treatment or control group– Minimize individual
differences in extra variables that might influence the dependent variable
Read a book
every dayReading
Achievement
Experimental Group
Reading Class
ReadingAchievement
Reading Class
Control Group
Interest
Dr. K. A. KorbUniversity of Jos
Random Assignment
• Draw names out of a hat
• Random number table
S/No Random Number
Group
1 1 Treatment
2 1 Treatment
3 0 Control
4 1 Treatment
5 1 Treatment
6 0 Control
7 1 Treatment
8 0 Control
9 0 Control
10 0 Control
Dr. K. A. KorbUniversity of Jos
Experiment• Pretest, Post-test Procedure
1. Random assignment of participants to experimental and control groups
2. Administered valid measure of DV as pretest to all groups
3. Administer treatment to experimental groups(s)4. Administer identical (or conceptually identical) measure
as posttest5. Statistical Analysis
– Descriptive statistics of pretest and post-test score for experiment and control groups separately
– Significance tests to determine significance between differences (Analysis of Covariance: ANCOVA)
Dr. K. A. KorbUniversity of Jos
Experimental
Dr. K. A. KorbUniversity of Jos
Experimental• Post-test Only Procedure
1. Random assignment of participants to experimental and control groups
2. Administered treatment to experimental group(s) 3. Administer valid measure of DV post-test4. Statistical Analysis
– Descriptive statistics of post-test score for experimental and control groups separately
– Significance tests to determine significance between differences (Analysis of Variance – ANOVA or t-test)
– The post-test only procedure is typically recommended because of the possibility that the pretest may have an effect on the experiment treatment
Dr. K. A. KorbUniversity of Jos
Experimental
Dr. K. A. KorbUniversity of Jos
Quasi-Experiment
• Purpose: Conduct an experiment when random assignment is not possible
• Key characteristic: Treatment and control groups, but no random assignment
• The null hypotheses are exactly the same as the Experimental method
Dr. K. A. KorbUniversity of Jos
Quasi-Experiment
• Pretest-Posttest Procedure1. Administer valid measure of DV as pretest2. Administer treatment to experimental group(s)3. Administer identical (or conceptually identical) measure
as posttest4. Statistical Analysis
• Descriptive statistical of pretest and posttest score for experiment and control groups separately
• Significance tests to determine significance between difference (ANCOVA)
Dr. K. A. KorbUniversity of Jos
Review: Research Designs• Descriptive: Carefully describe a naturally occurring
educational phenomenon through systematic observation
• Causal Comparative: Compare two or more groups on a dependent variable
• Correlational: Quantify the extent to which two variables are related
• Experiment: Establish cause and effect– Three requirement:
1. Treatment Groups(s)2. Control Group3. Random Assignment
• Quasi-Experiment: Experiment where random assignment is not possible
Dr. K. A. KorbUniversity of Jos
Research Designs Conclusion
• Testing the effectiveness of a counseling or teaching intervention MUST either use an experimental or quasi-experimental design
• Simply administering surveys CANNOT establish the quality of a counseling or teaching intervention
Dr. K. A. KorbUniversity of Jos
Research Example
• The effect of regular leisure reading on reading achievement in primary school
• Students in a primary school will be randomly assigned to either a treatment or a control group.– The treatment group will spend 10 minutes everyday reading a
book with an adult.– In an attempt to keep the treatment and control groups as
similar as possible, the control group will also spend 10 minutes with an adult everyday, but will instead do math problems.
• After one month of the treatment, both groups will be tested on a reading achievement test.
Dr. K. A. KorbUniversity of Jos
Research Example
ReadingAchievement
ReadingAchievement
Control Group
Experimental Group
Read BookEvery Daywith Adult
IV: RegularLeisureReading
DV:Reading
Achievement
ReadingAchievement
ReadingAchievement
Do MathsEvery Daywith Adult Dr. K. A. Korb
University of Jos
Construct Validity• Construct Validity: Establishing valid operational
measures for the concepts being studied– This relates to:• Independent Variable• Dependent Variable
• Definitions of Variable:– Construct Definition: General explanation of the
construct– Operational Definition: Statement of specifically how
the construct will be measured or implemented in the study
Dr. K. A. KorbUniversity of Jos
Construct vs. Operational Definitions
AcademicAcademicAchievementAchievement
State ExamsState Exams
NECO ScoresNECO Scores
WAEC ScoresWAEC Scores
Final Course ExamFinal Course Exam
Self-Report of Parents’Self-Report of Parents’Yearly IncomeYearly Income
Public/Private SchoolPublic/Private School
EconomicEconomicStatusStatus
ConstructConstruct Operational DefinitionOperational Definition
Dr. K. A. KorbUniversity of Jos
Research ExampleConstruct validity focuses on:
ReadingAchievement
ReadingAchievement
Control Group
Experimental Group
Read a BookEvery Day
IV: RegularReading
DV:Reading
Achievement
ReadingAchievement
ReadingAchievement
Dr. K. A. KorbUniversity of Jos
Construct Validity
• Research studies begin with hypotheses about psychological construct.– There is no significant effect of regular leisure reading on
reading achievement scores.• Independent variable: Regular leisure reading• Dependent variable: Reading achievement score
– The psychological constructs are then operationalized into manipulatable, measurable terms.• Regular leisure reading: Reading a book with an adult everyday
for 10 minutes• Reading achievement score: Classroom exam scores
Dr. K. A. KorbUniversity of Jos
Construct Validity
• When finished analyzing the data, the result will be translated back into psychological constructs. – The conclusion should not be limited to Reading a book for
10 minutes everyday with an adult leads to higher score on a classroom reading exams.
– The conclusion should be Regular leisure reading will lead to higher reading achievement scores in general.
• However, in order to translate the results back to psychological construct, the study must have construct valid measures of the independent and dependent variables.
Dr. K. A. KorbUniversity of Jos
Construct Validity• When you develop your research project, you start with a theory.
– For example, regular leisure reading will lead to increased reading achievement scores.
• Then you operationalize your theory into your research study.– You translate “regular leisure reading” into how it will be implemented in your
study – reading a book with an adult everyday for 10 minutes.– You translate “reading achievement scores” into how it will be measured in
your study – classroom exam scores.• When you finish collecting data, you hope to be able to move back to
theory.– You want to be able to say that not only did reading a book for 10 minutes
everyday with an adult lead to higher classroom exam scores on reading achievement, but also that regular leisure reading will lead to higher reading achievement scores in general.
– However, in order to do that, you must have construct valid measures of your independent and dependent variables.
Dr. K. A. KorbUniversity of Jos
Research Example
IV: RegularReading
DV:Reading
Achievement
Read bookevery day with
an adult for10 min.
Scores onreading sectionof classroom
exams
Constructs:
Operationalized:
IV: RegularReading
DV:Reading
AchievementConstructs:
Dr. K. A. KorbUniversity of Jos
Construct Validity
• Construct Validity of Research Designs: Allows generalization from the operationalized treatment and measures to the general psychological constructs
• Threats to Construct Validity– Poor construct definitions in the paper– Flawed matching of operationalization to constructs
• The measures do not align with the constructs they were designed to measure.
Dr. K. A. KorbUniversity of Jos
Construct Validity
• Research studies begin with hypotheses about psychological construct.– There is no significant effect of regular leisure reading on
reading achievement scores.• Independent variable: Regular leisure reading• Dependent variable: Reading achievement score
– The psychological constructs are then operationalized into manipulatable, measurable terms.• Regular leisure reading: Reading a book with an adult everyday
for 10 minutes• Reading achievement score: Classroom exam scores
Dr. K. A. KorbUniversity of Jos
Construct Validity
• When finished analyzing the data, the result will be translated back into psychological constructs. – The conclusion should not be limited to Reading a book for
10 minutes everyday with an adult leads to higher score on a classroom reading exams.
– The conclusion should be Regular leisure reading will lead to higher reading achievement scores in general.
• However, in order to translate the results back to psychological construct, the study must have construct valid measures of the independent and dependent variables.
Dr. K. A. KorbUniversity of Jos
Construct Validity• When you develop your research project, you start with a theory.
– For example, regular leisure reading will lead to increased reading achievement scores.
• Then you operationalize your theory into your research study.– You translate “regular leisure reading” into how it will be implemented in your
study – reading a book with an adult everyday for 10 minutes.– You translate “reading achievement scores” into how it will be measured in
your study – classroom exam scores.• When you finish collecting data, you hope to be able to move back to
theory.– You want to be able to say that not only did reading a book for 10 minutes
everyday with an adult lead to higher classroom exam scores on reading achievement, but also that regular leisure reading will lead to higher reading achievement scores in general.
– However, in order to do that, you must have construct valid measures of your independent and dependent variables.
Dr. K. A. KorbUniversity of Jos
Internal Validity• Internal validity: Extent to which variables other than the
treatment provide plausible explanations to the experimental results– Only relevant for experimental designs
• Primary consideration for interventions– Extraneous variable: Any variable other than treatment variable that,
if not controlled, can affect the experimental outcome– Hold constant or eliminate all extraneous variables that might affect
the posttest– Goal of Research Design: Create set of conditions so any observed
changes in your dependent variable can be attributed to experimental treatment instead of extraneous variables
• Do variables other than the treatment provide plausible explanations to the experimental results?
Dr. K. A. KorbUniversity of Jos
Research Example
ReadingAchievement
ReadingAchievement
Control Group
Read a BookEvery Day
IV: RegularReading
DV:Reading
Achievement
ReadingAchievement
ReadingAchievement
Experimental Group
Adult Attention
ReadingAbility
Interest
Maturation
Dr. K. A. KorbUniversity of Jos
Threats to Internal Validity• History: If treatment lasts over time, other events
may influence dependent variable– To prevent: Keep the control group equivalent in all
aspects but treatment• Maturation: Physical or psychological changes may
influence the dependent variable– To prevent: Use a control group
• Selection Bias: Treatment and control groups are different on an important extraneous variable– To prevent: Random assignment OR Use a pre-test
Dr. K. A. KorbUniversity of Jos
Threats to Internal Validity• Testing: Giving a pretest may increase performance
on the post-test– To prevent: Don’t use a pre-test OR Use different pre- and
post-tests• Instrumentation: Nature of measuring instrument
has changed between pre- and post-test– To prevent: Do not use a pre-test OR Use the same for
pre- and post-test• Many of these solutions to prevent threats to
internal validity contradict. – Consider the threats that are most relevant to a particular
study and then develop a strategy for overcoming the most important threats.
Dr. K. A. KorbUniversity of Jos
External Validity
• External Validity: Establishing the group of people to which the research findings can be generalized
Dr. K. A. KorbUniversity of Jos
Research Example
ReadingAchievement
ReadingAchievement
Control Group
Read a BookEvery Day
ReadingAchievement
ReadingAchievement
Experimental Group
PopulationPopulation
SampleSampleDr. K. A. KorbUniversity of Jos
Types of External Validity• Population Validity: Generalizing the results from an
experimental sample to a defined population– To establish: Give a thorough explanation of the constitution of
the constitution of the sample and how the sample was selected.• Ecological Validity: Generalizing the results of a study from
the artificial conditions created by researcher to real-life conditions– To establish:
• Give an adequate description of treatment• Give an adequate description of the measure for the dependent
variable– Threats to Ecological Validity:
• Novelty: The treatment was different than typical treatment• Experimenter Effect: The particular person delivering the treatment
affects results
Dr. K. A. KorbUniversity of Jos
Reliability
• Reliability: Demonstrating that the operations of the study can be repeated with the same results– Conduct research as if someone were always
looking over your shoulder– Report the procedure of your study in explicit
detail
Dr. K. A. KorbUniversity of Jos
Other Issues in Valid Research Design
• Experimenter Bias: Researcher’s expectations about the outcome of experiment influences participants’ response– To prevent: Use research assistants who are blind to the
study (aka do not know the purpose of the study)• Treatment Fidelity: Extent to which treatment
conditions are implemented according to the researcher’s specifications– To support: • Carefully train research assistants• Periodically check up on research assistants
unexpectedly to determine if they are following guidelines
Dr. K. A. KorbUniversity of Jos
Other Issues in Valid Research Design
• Strong Experimental Treatments: Developing a treatment that will make a robust effect on the dependent variable– To support:• Develop a thorough understand of the dependent
variable and how it can be influenced.• Spend much thoughtful time developing your
treatment.
Dr. K. A. KorbUniversity of Jos
Revision• Describe the five types of educational research
designs.• What is construct validity as related to research
designs? How can it be supported?• What is internal validity? What are threats to
internal validity?• What is external validity? What are types of
external validity?• What are some other issues to consider when
conducting educational research?
Dr. K. A. KorbUniversity of Jos