1 Reliability and Validity. 2 Not Reliable, Not Valid.
-
Upload
fay-reynolds -
Category
Documents
-
view
317 -
download
1
Transcript of 1 Reliability and Validity. 2 Not Reliable, Not Valid.
1
Reliability and ValidityReliability and Validity
2Not Reliable, Not ValidNot Reliable, Not Valid
3Reliable, Not ValidReliable, Not Valid
4Reliable and ValidReliable and Valid
5
Relationship of Reliability and ValidityRelationship of Reliability and Validity
A measure can be reliable without A measure can be reliable without being valid.being valid.
A measure must be reliable to be valid.A measure must be reliable to be valid.
6 •ValidityValidity
• Types of ValidityTypes of Validity
• ContentContent
• ConstructConstruct
• Criterion-RelatedCriterion-Related
7 •ValidityValidity
• Content ValidityContent Validity
• To what extent do the items on the test To what extent do the items on the test adequately sample from the intended adequately sample from the intended universe of content?universe of content?
• Judgmental EvidenceJudgmental Evidence• Literature ReviewLiterature Review
• Expert ReviewExpert Review
8 •ValidityValidity
• Construct ValidityConstruct Validity• To what extent do certain explanatory concepts To what extent do certain explanatory concepts
explain:explain:• covariation in the responses to the items, orcovariation in the responses to the items, or• relationships with other know indicators?relationships with other know indicators?
• Empirical evidenceEmpirical evidence• CorrelationsCorrelations• Factor analysisFactor analysis• Structural equation modelingStructural equation modeling• Item response theoryItem response theory• Known groupsKnown groups
9 •ValidityValidity
• Criterion-Related ValidityCriterion-Related Validity
• What is the relationship between scores on the What is the relationship between scores on the instrument and some external criterion that instrument and some external criterion that provides a more direct measure of the targeted provides a more direct measure of the targeted characteristic?characteristic?
• TypesTypes• ConcurrentConcurrent
• PredictivePredictive
10 •ReliabilityReliability
•““Does the instrument Does the instrument provide us with accurate provide us with accurate
assessments?”assessments?”
11 •ReliabilityReliability
•TheoryTheory
•rr1122
TrueTrue = = ReliabilityReliability
• “ “Proportion of variance in Proportion of variance in scores that can be considered scores that can be considered
‘true’ variance”‘true’ variance”
12 •ReliabilityReliability
• Sources of ErrorSources of Error
• Inadequate sampling of itemsInadequate sampling of items• Cronbach’s alpha internal Cronbach’s alpha internal
consistency reliabilityconsistency reliability
• Different occasionsDifferent occasions• Stability reliabilityStability reliability
13 •ReliabilityReliability
• Types of ReliabilityTypes of Reliability
• Internal consistencyInternal consistency
• StabilityStability
14 •ReliabilityReliability
•Generalizability TheoryGeneralizability Theory
•““Simultaneously examines Simultaneously examines homogeneity of the items and homogeneity of the items and
stability of responses over time”stability of responses over time”
15 •Qualitative ResearchQualitative Research
•Procedures which produce Procedures which produce descriptive data -- peoples’ descriptive data -- peoples’ written or spoken word, or written or spoken word, or observations of behaviorobservations of behavior
16 •Qualitative ResearchQualitative Research
• Ethnography - Naturalistic InquiryEthnography - Naturalistic Inquiry
• Study behavior in a natural settingStudy behavior in a natural setting
• Involvement with subjectsInvolvement with subjects
• See the world through their eyesSee the world through their eyes
17 •Research MethodsResearch Methods
• Selecting a Research Selecting a Research StrategyStrategy
• QualitativeQualitative
• QuantitativeQuantitative
18 •Research MethodsResearch Methods
•Philosophical BasePhilosophical Base Logical PositivismLogical Positivism PhenomenologyPhenomenology
•PurposePurpose VerificationVerification DiscoveryDiscovery
•Framework/DesignFramework/Design Preordinate/FixedPreordinate/Fixed Emergent/FlexibleEmergent/Flexible
•ConditionsConditions ControlledControlled Invited InterferenceInvited Interference
•TreatmentTreatment StableStable VariableVariable
Quantitative Quantitative QualitativeQualitative
19 •Research TargetResearch Target
• ProcessProcess ProductProduct
• ImplementationImplementationOutcomeOutcome
• ““Implementation as an Implementation as an independent variable”independent variable”
20 •Qualitative ResearchQualitative Research
• Front StageFront Stage• Packaged RealityPackaged Reality
• Back StageBack Stage• RealityReality
• GatekeepersGatekeepers
““Schools are not researchers’ sandbox”Schools are not researchers’ sandbox”
Access and EntryAccess and Entry
21 •Qualitative ResearchQualitative Research
• InterviewInterview• What is What is saidsaid
• Document AnalysisDocument Analysis• What is What is writtenwritten
• Participant ObservationParticipant Observation• What is What is donedone
Data CollectionData Collection
22 •Qualitative ResearchQualitative Research
• Combination of Methodologies Combination of Methodologies • Multiple Reference PointsMultiple Reference Points
• Increase AccuracyIncrease Accuracy• Increase ConfidenceIncrease Confidence
• Convergence of EvidenceConvergence of Evidence• Among MethodsAmong Methods• Within MethodsWithin Methods
TriangulationTriangulation
23 •Qualitative ResearchQualitative Research
• OngoingOngoing At conclusionAt conclusion
• InductiveInductive DeductiveDeductive• Generating TheoryGenerating Theory Testing TheoryTesting Theory
QualitativeQualitative QuantitativeQuantitative
--
Data AnalysisData Analysis
24 •Qualitative ResearchQualitative Research
• Reporting ResultsReporting Results• Natural HistoryNatural History
• Thematic AnalysisThematic Analysis
• Social NetworksSocial Networks
• Life HistoryLife History
• Allocation and Distribution of ResourcesAllocation and Distribution of Resources
• Organizational AnalysisOrganizational Analysis
• Critical IncidenceCritical Incidence
25 •Research DesignResearch Design
• Development of EducationDevelopment of EducationResearch MethodsResearch Methods
• Genetics Genetics • Correlation (Galton)Correlation (Galton)
• AgricultureAgriculture• Controlled experiments (Fisher - ANOVA)Controlled experiments (Fisher - ANOVA)
26 •Research DesignResearch Design
• Complexity of Educational Complexity of Educational ProcessesProcesses
• Experimental research: failure of the Experimental research: failure of the
promisepromise
• Trend toward qualitative methodsTrend toward qualitative methods
27 •Research DesignResearch Design
• Experimental and Quasi-Experimental Designs for Experimental and Quasi-Experimental Designs for Research Research (1963)(1963)
• Donald CampbellDonald Campbell• Julian StanleyJulian Stanley
• Shift focus from statistics to problems of acquiring Shift focus from statistics to problems of acquiring data in educationdata in education
• Develop benchmarks for research designsDevelop benchmarks for research designs
• Compare research designs with respect to benchmarksCompare research designs with respect to benchmarks
28 •Research DesignResearch Design
• Research Design FocusResearch Design Focus
• 1960’s - 1970’s: Experimental1960’s - 1970’s: Experimental
• The External Validity of ExperimentsThe External Validity of ExperimentsBracht & Glass, 1968Bracht & Glass, 1968
• 1980 +: Quasi-Experimental1980 +: Quasi-Experimental
• Quasi-Experimentation: Design and Analysis Issues for Quasi-Experimentation: Design and Analysis Issues for Field SettingsField SettingsCook & Campbell, 1979Cook & Campbell, 1979
29 •Research DesignResearch Design
•Internal ValidityInternal Validity
•To what extent have extraneous variables been To what extent have extraneous variables been controlled so that the observed effect (dependent) controlled so that the observed effect (dependent) can be attributed to the treatment variable can be attributed to the treatment variable (independent)?(independent)?
•Is it reasonably plausible that the outcome Is it reasonably plausible that the outcome (dependent) is created by the treatment (dependent) is created by the treatment (independent)?(independent)?
30 •Research DesignResearch Design
• Threats to External ValidityThreats to External Validity
• Results generalizable to other:Results generalizable to other:
• People?People?• Settings?Settings?• Independent variables?Independent variables?• Dependent variables?Dependent variables?
31 •Research DesignResearch Design
• Internal Validity ThreatsInternal Validity Threats
• TestingTesting
• InstrumentationInstrumentation
• Statistical regressionStatistical regression
• Differential selectionDifferential selection
• Experimental treatment diffusionExperimental treatment diffusion
32 •Research DesignResearch Design
• Internal Validity ThreatsInternal Validity Threats
• Compensatory rivalry by the control Compensatory rivalry by the control groupgroup
• Compensatory equalization of Compensatory equalization of treatmentstreatments
• Resentful demoralization of the control Resentful demoralization of the control groupgroup
33
One Group Pre-PostOne Group Pre-Post
X
•Research DesignResearch Design
34
Pre-Post Comparison GroupPre-Post Comparison Group
X
•Research DesignResearch Design
35
Pre-Post Comparison GroupPre-Post Comparison Groupwith Random Assignmentwith Random Assignment
XRR
Research DesignResearch Design
36
Post-Only Comparison GroupPost-Only Comparison Groupwith Random Assignmentwith Random Assignment
XRR
Research DesignResearch Design
37 •External Validity of ResearchExternal Validity of Research
• ““Generalizability”Generalizability”
• Population ValidityPopulation Validity• Generalize from sample to population?Generalize from sample to population?
• Personalogical variables interact with Personalogical variables interact with treatment?treatment?
38 •External Validity of ResearchExternal Validity of Research
• ““Generalizability”Generalizability”
• Ecological ValidityEcological Validity• Explicit description of treatmentExplicit description of treatment• Multiple treatment interferenceMultiple treatment interference• Hawthorne effectHawthorne effect• Novelty and disruption effectsNovelty and disruption effects• Experimenter effectExperimenter effect• Pretest sensitizationPretest sensitization• Measurement of dependent variableMeasurement of dependent variable
39 •Statistical AnalysisStatistical Analysis
Experimental
Xchange
ComparisonChange Scores
Post Pre = Change Post Pre = Change
Xchange
40 •Statistical AnalysisStatistical Analysis
Repeated Measures ANOVARepeated Measures ANOVARMANOVARMANOVA
ExperimentalExperimental
ComparisonComparison
PrePre PostPost
41 •Statistical AnalysisStatistical Analysis
RMANOVARMANOVASources of VarianceSources of Variance
BetweenBetweenGroupGroup FFgroupgroup
Error (b)Error (b)
WithinWithinTimeTime FFtimetime
Time x GroupTime x Group FFtime x grouptime x group
Error (w)Error (w)
TotalTotal
42 •Statistical AnalysisStatistical Analysis
Research Question
Will there be a difference between groups Will there be a difference between groups with respect to posttest scores after with respect to posttest scores after controlling for initial differences in controlling for initial differences in pretest scores?pretest scores?
Analysis of CovarianceAnalysis of CovarianceANCOVAANCOVA
43
Pre CovariatePre Covariate
PostPostDependentDependent
ExpExp
ControlControl
43
44
Pre CovariatePre CovariatePÕST = b(PRE) + aPÕST = b(PRE) + a
PostPostDependentDependent
ExpExp
ControlControl
44
45
Pre CovariatePre Covariate
PostPostDependentDependent
ExpExp
ControlControl
XXContCont XXExpExp
YYCC
YYEE
MarginalMarginalDistributionDistribution
45
46
Pre CovariatePre Covariate
PostPostDependentDependent
ExpExp
ControlControl
XXContCont XXExpExp
YYCC
YYEE
MarginalMarginalDistributionDistribution
XX
46
47 •Effect SizeEffect Size
““The difference between two means expressed The difference between two means expressed in standard deviation units”in standard deviation units”
ES =ES = Mean 1 - Mean 2Mean 1 - Mean 2Standard DeviationStandard Deviation
ESES Qualitative DifferenceQualitative Difference
.20.20 SmallSmall
.50.50 MediumMedium
.80.80 LargeLarge
48 •Statistical Significance: Statistical Significance: Sample Size and Effect SizeSample Size and Effect Size
““I have a meaningful say in designing my job/work.”*I have a meaningful say in designing my job/work.”*
StandardStandard Sign. Sign. EffectEffect Group Group NN Mean Mean DeviationDeviation LevelLevel SizeSize
.000.000 .22.22MalesMales 10,74710,747 3.003.00 1.201.20
FemalesFemales 8,0888,088 3.263.26 1.191.19
MalesMales 111111 3.033.03 1.171.17
FemalesFemales 8181 3.283.28 1.221.22.157.157 .21.21
* Responded to on a 5-point Likert agreement scale.* Responded to on a 5-point Likert agreement scale.