Reliability & Validity Reliability & Validity
ReliabilityReliability ““dependability”dependability” is the indicator consistent?is the indicator consistent? same result every time?same result every time? Does not necessary measure what you think it measures Does not necessary measure what you think it measures
(ex. May consistently measure something other than (ex. May consistently measure something other than the concept)the concept)
ValidityValidity measurement validity - how well the conceptual and measurement validity - how well the conceptual and
operational definitions mesh with each otheroperational definitions mesh with each other does measurement tool measure what we think ?does measurement tool measure what we think ?
Relationship between Measurement Reliability and ValidityRelationship between Measurement Reliability and Validity
reliability necessary for validity but does not reliability necessary for validity but does not
guarantee it (guarantee it (“necessary but not sufficient”)“necessary but not sufficient”) measure can be reliable but invalidmeasure can be reliable but invalid
Examples of Types of ReliabilityExamples of Types of Reliability
stability stability over timeover time
representative representative across different subgroups of a populationacross different subgroups of a population
(ex. of problems: young people may exaggerate (ex. of problems: young people may exaggerate their ages, older people may reduce theirs)their ages, older people may reduce theirs)
equivalence equivalence Comparable results from multiple indicatorsComparable results from multiple indicators intercoder reliabilityintercoder reliability
Improving ReliabilityImproving Reliability
clearly conceptualize constructsclearly conceptualize constructs increase level of measurement increase level of measurement use pretests, pilot studiesuse pretests, pilot studies e.g. use multiple indicators :e.g. use multiple indicators :
DependentVariable Measure
IndependentVariable Measure
Empirical
Association?
a2 a3a1 b1 b2
A B
Specific IndicatorsSpecific Indicators
Measurement Using Multiple Indicators
Neuman (2000: 167)
Selected Types of Validity: Face ValiditySelected Types of Validity: Face Validity
judgement by the scientific community that indicator judgement by the scientific community that indicator measures the conceptmeasures the concept
Construct/concept
MeasureScientific Community
Content ValidityContent Validity measure represents all the aspects of conceptual definition of measure represents all the aspects of conceptual definition of
construct.construct. judgement concerning how adequately a test samples behavior judgement concerning how adequately a test samples behavior
representative of the universe of behavior the test was representative of the universe of behavior the test was
designed to sampledesigned to sample..
Love
Beliefs& Values
CriterionCriterion The validity of indicator is verified by comparing it with another The validity of indicator is verified by comparing it with another
measure of the same constructmeasure of the same construct
PredictivePredictive Relies on occurrence of future event or behavior for Relies on occurrence of future event or behavior for
verification of indicatorverification of indicator ConcurrentConcurrent
relies on pre-existing accepted measure to verify indicatorrelies on pre-existing accepted measure to verify indicator Construct ValidityConstruct Validity
A type of measurement validity that uses multiple indicatorsA type of measurement validity that uses multiple indicators convergent and discriminateconvergent and discriminate
Some types of Measurement Validity Some types of Measurement Validity
Other Dimensions of ValidityOther Dimensions of Validity
Internal ValidityInternal Validity no error of logic internal to research designno error of logic internal to research design
External ValidityExternal Validity results can be generalizedresults can be generalized
Statistical validityStatistical validity correct statistical methodology chosen ?correct statistical methodology chosen ? assumptions fully metassumptions fully met
More terms & concepts for assessing validity (usually in experimental research)More terms & concepts for assessing validity (usually in experimental research)
Threats to Internal ValidityThreats to Internal Validity Selection biasSelection bias History effects (something in context changes)History effects (something in context changes) Maturation Maturation Testing effectTesting effect InstrumentationInstrumentation MortalityMortality Statistical regressionStatistical regression ContaminationContamination Compensatory behaviourCompensatory behaviour Experimenter expectancExperimenter expectanc
Threats to External ValidityThreats to External Validity RealismRealism ReactivityReactivity
Another Aspect of Measurement Design: Levels of MeasurementAnother Aspect of Measurement Design: Levels of Measurement
1-Nominal (ex. Mother tongue)1-Nominal (ex. Mother tongue) different categories (names, labels, images) different categories (names, labels, images) not rankednot ranked
2-Ordinal (county fair prizewinners ranked by first, second & third prize)2-Ordinal (county fair prizewinners ranked by first, second & third prize) different categoriesdifferent categories rank-orderedrank-ordered attributes indicate relatively more or less of that variableattributes indicate relatively more or less of that variable distance between the attributes of a variable is imprecisedistance between the attributes of a variable is imprecise
Levels of Measurement (cont’d)Levels of Measurement (cont’d) 3- Interval Measures (age measured by 5 year age groups)3- Interval Measures (age measured by 5 year age groups)
different categoriesdifferent categories ranked in orderranked in order Can tell amount of difference between categoriesCan tell amount of difference between categories Usually no true zeroUsually no true zero
4- Ratio Measures (age measured by date of birth)4- Ratio Measures (age measured by date of birth) different categoriesdifferent categories ranked in order ranked in order amount of difference between categories amount of difference between categories also possible to state proportion (have a true zero)also possible to state proportion (have a true zero)
Relations between levels --Relations between levels --can collapse from higher can collapse from higher into lower, not vice versainto lower, not vice versa
Babbie (1995: 101)
The Research ProcessThe Research Process