Designing and implementing livestock value chain studies ...
Class 5 designing so tl studies
Transcript of Class 5 designing so tl studies
Designing SoTL Studies – Part IValidity
Class Session 5 – Henry Clark, PhD
Independent variable = the
cause
Dependent variable = the
effect
The researcher controls or
manipulates the independent variable (the treatment)
The dependent variable is what is measured, often
called the assessment
(knowledge, skills or attitudes).
Testing Hypotheses
A Simple Hypothesis : The treatment (independent variable) improves
students on the assessment (dependent variable).
Three possible major problems related to causality:
1. The assessment was not measured well
2. The treatment was not manipulated well
3. Something other than the treatment caused change in the assessment (internal validity).
Construct and Internal Validity
0Construct Validity:
Am I measuring what I think I am measuring?
Am I implementing what I think I am implementing?
0Internal Validity: Did the treatment cause the outcome?
A Simple Hypothesis : The treatment (independent variable) improves
students on the assessment (dependent variable).
Three possible major problems related to causality:
1. The assessment was not measured well (reliability and construct validity)
2. The treatment was not manipulated well (construct validity)
3. Something other than the treatment caused change in the assessment (internal validity).
A study does not have absolute validity or absolutely no validity
The level of validity relates to the confidence in the conclusions
Construct and internal validity are measured on a continuum
Construct validity does not imply internal validity (and vice versa)
When a hypothesis is supported, it does not necessarily mean that the
study has either construct or internal validity
Some notes on
evaluating construct
and internal validity
What is meant by “construct”?0A concept, model, or schematic idea
0A construct is the global notion of the measure, such as:
0 Student motivation
0 Intelligence
0 Student learning
0 Student anxiety
0The specific method of measuring a construct is called the operational definition
0For any construct, researchers can choose many possible operational definitions
To Improve Construct Validity of Measures
0 Measure learning directly (clear operational definitions; learning is not the same as enjoyment or perceived learning)
0 Measure student learning through student learning objectives (ensure these are aligned with assessments)
0 Use Established Scales to Measure Student Attitudes and Personality (Don’t reinvent the wheel; Tests in Print)
Good Measurement is Important To Improve Construct Validity
0Know How To Score the Measure (make sure you’ve established this before data collection; know what is reasonable; IOTT; rubrics; training; IRR)
0Determine Whether to Use Graded or Ungraded Measures (pros and cons of both)
0Minimize Participant and Researcher Expectancies
To Improve Construct Validity
0Determine Whether to Use Multiple Operational Definitions (can use multiple measures)
0 Use a Retention Measure to Investigate Long-term Effects (but treat long term results with caution about other influences)
Good Differences between Conditions Improve Construct Validity
The treatment (intervention) needs to be manipulated well to ensure
construct validity
The only difference between conditions should be the treatment
Other variables that are different between conditions are confounds
To determine construct validity, treatments need specific operational definitions
Anything that can affect the results and cause a difference between students in treatment and control conditions needs to be documented
Potential problems in using different sections of a class
Construct validity of the treatment is questionable in any design that
compares one section of a class with another
Classes are a social space, and the students and instructors are interdependent
Students can ask different questions
The class may have a different “tone”
Splitting a class into two groups can minimize this concern; if students in a split class can be randomly assigned to a condition, internal validity will increase
Different Types of Comparison in Research Design Between
ParticipantsWithin Participants: Multiple Treatments
Within Participants: Multiple Measures
How comparison works
Students in one condition compared to students in another condition (control – Treatment; multiple T’s)
All students in both control and treatment conditions
Students receive both pre-test (control) and post-test (treatment)
Strengths No carryover effects from multiple treatments; no instrumentation or testing effects from multiple assessments
No selection bias; greater statistical power
No selection bias; greater statistical power
Weaknesses Selection bias without random assignment; many differences if groups are separate (e.g., two separate classes); lower statistical power
Instrumentation and testing effects; carryover effects
Instrumentation and testing effects; other confounds that occur between assessmens
Improve Internal Validity by:
Random assignment ; adding covariates
Counterbalancing Increase number of assessments; add no treatment separate control condition; use alternative measures for assessment
External Validity
Can the sample used in the study
generalize to other groups or
populations?
Generally, it is impossible in
classroom studies to get a sample
that will generalize to all
students.
The researcher should report demographic chacteristics
How realistic is the situation? In
a classroom, if the treatment works, external validity
is higher
Designing SoTL Studies – Part IIPracticality
Common Practical Problems in SoTL Research
Researchers who think they need to measure everything
Researchers who do not have many students: low statistical power
Researchers who only have a single class; limits to type of design
Difficulties in random assignment
Difficulties in determining whether the treatment is potent enough to have an effect (see power above)
Concerns about conducting an ethical study in a classroom or training situation
Don’t Use Want to make statement
about causality Have low number of
students
Use Have single group of
students that cannot be divided
Have only one session in which to collect data
Additional Options:Correlate many variables at the same time
Simple Correlation
One-Group, Post Test Only
Don’t Use Want to make statement
about causality
Want to make comparison to another group
Use Desired focus is on
describing treatment and not assessment
Cannot have pre-test or control group
Want single group of students that cannot be divided
Two-Group, Post-Test Only Don’t Use
Have low number of students
Groups are very different Have different assessments
for each condition
Use Concerned about carryover
effects Concerned about testing and
instrumentation effects Have multiple groups Have only one session to
collect data
Additional Options:• Use random assignment to improve internal validity• Add post-test to assess long-term change• Add additional conditions• Use covariates to improve internal validity and power
One Group, Pre-test, Post-test
Don’t Use Items other than treatment
occur between assessments First assessment affects
second Students likely to change
between assessments with no treatment
Use Have low number of
students Have single group that
cannot e divided Cannot have control
condition
Additional Options:• Add post-test to assess long-term change• Use alternative measures to minimize testing and
instrumentation effects
Two-Group, Pre-test/Post-test
Don’t Use Have single group of
students that cannot be divided
Use Have multiple groups
Additional Options:• Use random assignment to improve internal validity• Add post-test to assess long-term change• Use alternative measures to minimize testing and
instrumentation effects• Add additional conditions• Use covariates to improve internal validity and power
Within Participants Design
Don’t Use Early treatments affect
later treatments Early assessments affect
later assessments
Use Have low number of
students Have single group that
cannot be divided
Additional Options:• Add additional treatments• Counterbalance conditions to improve internal validity• Include pre-test to assess students before any treatment
Crossover DesignDon’t Use
First assessment, by itself, affects second
Have single group of students that cannot be divided
Use Have low number of
students Have multiple groups
Additional Options:• Include pre-test to assess before treatment• Add post-test to examine long-term change• Use random assignment to improve internal validity• Use alternative measures to minimize testing and
instrumentation effects
Interrupted Time-Series Design
Don’t Use Have only one session to
collect data Early assessments affect
later assessments
Use Have low number of
students Have single group that
cannot be divided Want to determine long-
term effects
Additional Options:• Add control condition to improve internal validity• Add additional treatment condition, with treatment at
different time to improve internal validity
More Complex Designs
0Use Multiple Treatments to Investigate Interactions (Interactions)
0Use Moderators to Determine When Treatment Has Effect (Concept of ATI)
0Use Mediators to Investigate How Treatment Has Effect (Mixed Method?)
Remember!0Each design has advantages and
disadvantages
0Often, there is no clear right way, although some designs will be better than others
0There is no single ideal study that eliminates all potential problems and all alternative hypotheses
0No one study can answer all of your questions!