Educational Educational Action ResearchAction Research

Todd TwymanTodd Twyman

Summer 2011Summer 2011

Week 1Week 1

AgendaAgenda

Discuss Internal and External Discuss Internal and External ValidityValidity

Review Threats to Internal ValidityReview Threats to Internal Validity Consider Precautions for Minimizing Consider Precautions for Minimizing

Threats to Internal ValidityThreats to Internal Validity

Why are you here?Why are you here?

How can you use the results from How can you use the results from your action research project to your action research project to inform your practices?inform your practices?

Why is action research important for Why is action research important for improving schools?improving schools?

The internal validity of a study relies on the researcher’s ability to rule out factors other than the intervention as possible causes of changes in the dependent variable.

Internal Validity

Improving Internal Improving Internal ValidityValidity

Maximize systematic varianceMaximize systematic variance Make the groups as different as Make the groups as different as

possible in regard to the possible in regard to the interventionintervention Minimize error varianceMinimize error variance

Reduce the differences between the Reduce the differences between the participantsparticipants

Control extraneous varianceControl extraneous variance Manipulate the environment to Manipulate the environment to

maximize the controlsmaximize the controls

Improving Internal ValidityImproving Internal Validity

Focus is on trying to “control” Focus is on trying to “control” extraneous factors (or confounds). extraneous factors (or confounds). Identify threats and then either Identify threats and then either delete or reduce threats to validity delete or reduce threats to validity through your research design.through your research design.

Thinking about Internal Thinking about Internal Validity…Validity…

What could have caused these findings?What could have caused these findings? What could have influenced the results What could have influenced the results

besides the intervention?besides the intervention? What are alternative hypotheses for the What are alternative hypotheses for the

results?results? What decisions would you make based What decisions would you make based

on these results?on these results? What are the implications of the What are the implications of the

findings?findings?

Threats to Internal Threats to Internal ValidityValidity

HistoryHistory MaturationMaturation Testing Testing InstrumentationInstrumentation Statistical Statistical

regression regression Differential selectionDifferential selection Experimental Experimental

mortalitymortality Selection-maturation Selection-maturation

interactioninteraction

Experimental Experimental treatment diffusiontreatment diffusion

Compensatory Compensatory rivalry by the rivalry by the control groupcontrol group

Compensatory Compensatory equalization of equalization of treatmentstreatments

Resentful Resentful demoralization of demoralization of the control groupthe control group

Threats to Internal Threats to Internal ValidityValidity

History:History: Events may occur that influence Events may occur that influence the observed effects.the observed effects.

Maturation:Maturation: Physical/psychological Physical/psychological changes that influence the observed changes that influence the observed effects.effects.

Testing:Testing: Pre-test influences the post-test. Pre-test influences the post-test. Instrumentation:Instrumentation: Changes in the Changes in the

measurement tool influence the observed measurement tool influence the observed effects.effects.

Identifying ThreatsIdentifying Threats

A tenth-grade biology teacher A tenth-grade biology teacher institutes a new approach to the institutes a new approach to the teaching of one-celled animal life teaching of one-celled animal life based on a series of seven-minute based on a series of seven-minute single-concept films. Prior to starting single-concept films. Prior to starting the new unit, she develops a test the new unit, she develops a test covering the material and administers covering the material and administers it to her class before and after the unit, it to her class before and after the unit, noting with satisfaction a dramatic noting with satisfaction a dramatic improvement by her pupils on the test.improvement by her pupils on the test.

Threats to Internal Threats to Internal ValidityValidity

Statistical regressionStatistical regression: Test scores : Test scores move toward the mean.move toward the mean.

Differential selectionDifferential selection: Participants : Participants are differentially selected into groups.are differentially selected into groups.

AttritionAttrition: Participants differentially : Participants differentially drop out of the study.drop out of the study.

Selection-maturation interactionSelection-maturation interaction: : Participants differ based on Participants differ based on maturation.maturation.

Identifying ThreatsIdentifying Threats A school principal has received a federal A school principal has received a federal

grant to institute a new mathematics grant to institute a new mathematics enrichment program. Using school records, enrichment program. Using school records, he computes the average mathematics he computes the average mathematics achievement score on a nationally achievement score on a nationally standardized examination for his fifth-, standardized examination for his fifth-, ninth-, and twelfth-grade pupils during each ninth-, and twelfth-grade pupils during each of the preceding three years. He plans to of the preceding three years. He plans to have the same tests administered for the have the same tests administered for the two years following the institution of the two years following the institution of the new program. He wishes to compare the new program. He wishes to compare the relative performance of the three classes relative performance of the three classes before and after the implementation of the before and after the implementation of the new program.new program.

Threats to Internal Threats to Internal ValidityValidity

Treatment diffusionTreatment diffusion: Control group seeks : Control group seeks access to treatment.access to treatment.

Compensatory rivalryCompensatory rivalry: Control group : Control group works harder to make up for group works harder to make up for group placement.placement.

Compensatory equalizationCompensatory equalization: Outsiders : Outsiders provide other treatment to compensate.provide other treatment to compensate.

Demoralization of control groupDemoralization of control group:: Control group performs worse due to Control group performs worse due to discouragement.discouragement.

Identifying ThreatsIdentifying Threats

A researcher uses a table of random A researcher uses a table of random numbers to subdivide 15 junior high numbers to subdivide 15 junior high school classes into three groups of five school classes into three groups of five classes each (30 pupils per class) as classes each (30 pupils per class) as follows: “Treatment A,” “Treatment B,” follows: “Treatment A,” “Treatment B,” and “Control.” The treatments are and “Control.” The treatments are administered during the first four months administered during the first four months of the school year, and at mid-semester a of the school year, and at mid-semester a posttest is given to all of the 450 pupils posttest is given to all of the 450 pupils involved.involved.

3 Steps to Strengthening 3 Steps to Strengthening Treatment FidelityTreatment Fidelity

Write precise specifications for Write precise specifications for dependent and independent dependent and independent variables. variables.

TRAIN extensively to ensure TRAIN extensively to ensure treatment is delivered as intended.treatment is delivered as intended.

Collect data on fidelity of treatment Collect data on fidelity of treatment implementation.implementation.

Internal Validity ReviewInternal Validity Review

Threats to Internal Threats to Internal ValidityValidity

1. History (H)1. History (H)2. Maturation (M)2. Maturation (M)3. Testing (T) 3. Testing (T) 4. Instrumentation (I)4. Instrumentation (I)5. Statistical regression 5. Statistical regression

(SR) (SR) 6. Differential selection 6. Differential selection

(DS)(DS)7. Experimental 7. Experimental

mortality (EM)mortality (EM)8. Selection-maturation 8. Selection-maturation

interaction (SMI)interaction (SMI)

9. Experimental 9. Experimental treatment diffusion treatment diffusion (ETD)(ETD)

10. Compensatory 10. Compensatory rivalry by the control rivalry by the control group (CR-control)group (CR-control)

11. Compensatory 11. Compensatory equalization of equalization of treatments (CE)treatments (CE)

12. Resentful 12. Resentful demoralization of the demoralization of the control group (Rdem)control group (Rdem)

Control group works harder to make Control group works harder to make up for group placementup for group placement

Pre-test influences the post-testPre-test influences the post-test Control group performs worse due Control group performs worse due

to discouragementto discouragement Test scores move toward the meanTest scores move toward the mean Events may occur that influence the Events may occur that influence the

observed effectsobserved effects

Control group seeks access to Control group seeks access to treatmenttreatment

Participants are differentially Participants are differentially selected into groups selected into groups

Physical/psychological changes that Physical/psychological changes that influence the observed effectsinfluence the observed effects

Participants differentially drop out of Participants differentially drop out of the studythe study

Participants differ based on Participants differ based on maturationmaturation

Changes in the measurement tool Changes in the measurement tool influence the observed effectsinfluence the observed effects

Outsiders provide other treatment to Outsiders provide other treatment to compensatecompensate

QuizQuiz

Steps to Reducing Steps to Reducing Threats to ValidityThreats to Validity

Debrief with peers: identify strengths/ Debrief with peers: identify strengths/ weaknesses/ biases.weaknesses/ biases.

Align data sources with research questions.Align data sources with research questions. Triangulate data sources.Triangulate data sources. Record data accurately and without bias.Record data accurately and without bias. Embed fidelity checks.Embed fidelity checks. Thoroughly describe the setting and study.Thoroughly describe the setting and study. Engage in continuous and ongoing reflection.Engage in continuous and ongoing reflection.

Read with an Informed Read with an Informed EyeEye

Assignment 3 will ask you to critique an Assignment 3 will ask you to critique an Action Research article of your choice.Action Research article of your choice.

To prepare:To prepare: Look carefully at the methodology of Look carefully at the methodology of

research articles.research articles. Evaluate the degree to which the Evaluate the degree to which the

authors have controlled threats to authors have controlled threats to internal validity.internal validity.

Question everything!Question everything!

Critiquing ResearchCritiquing Research

Are the problems and their educational Are the problems and their educational importance explained clearly?importance explained clearly?

Does the literature review sufficiently Does the literature review sufficiently set up a rationale for the need for this set up a rationale for the need for this study?study?

Are the research variables easily Are the research variables easily identified from the hypothesis or identified from the hypothesis or purpose statement?purpose statement?

Critiquing ResearchCritiquing Research What instruments are used in the study? What instruments are used in the study?

Are they appropriate for the research Are they appropriate for the research question and for the participants?question and for the participants?

Is the research design appropriate to Is the research design appropriate to the research question?the research question?

How does the study control for threats How does the study control for threats to internal or external validity?to internal or external validity?

Are the conclusions logically related to Are the conclusions logically related to the questions posed in the research the questions posed in the research question?question?

Your own studyYour own study

Identify the goals for conducting your Identify the goals for conducting your studystudy

Identify the types of validity that are Identify the types of validity that are applicable to your study (consider the applicable to your study (consider the purpose of the study, activities, purpose of the study, activities, audience, data sources)audience, data sources)

Create a plan for reducing the threats Create a plan for reducing the threats to internal validity that is aligned to internal validity that is aligned with the types of validity addressed with the types of validity addressed above.above.