IOP 301-T Test Validity. IOP 301-T What is validity? In simple English, the validity of a test...
-
Upload
lisa-barks -
Category
Documents
-
view
248 -
download
7
Transcript of IOP 301-T Test Validity. IOP 301-T What is validity? In simple English, the validity of a test...
IOP 301-TIOP 301-T
Test ValidityTest Validity
IOP 301-TIOP 301-T
What is validity?What is validity?
In simple English, the validity of a test concerns In simple English, the validity of a test concerns
what the test measures andwhat the test measures and how well it measureshow well it measures
It is the accuracy of the measure in reflecting It is the accuracy of the measure in reflecting the concept it is supposed to measure.the concept it is supposed to measure.
IOP 301-TIOP 301-T
Types of validityTypes of validity
Content-descriptionContent-description Criterion-descriptionCriterion-description Construct-identificationConstruct-identification
IOP 301-TIOP 301-T
Faceva lid ity
Contentva lid ity
Concurren tva lid ity
P red ictiveva lid ity
C rite rion-re la tedva lid ity
Corre la tion
Factoria lva lid ity
Convergentva lid ity
D iscrim inantva lid ity
Constructva lid ity
VALIDITY
IOP 301-TIOP 301-T
Content validityContent validity
non-statistical in naturenon-statistical in nature involves determining whether the involves determining whether the
sample used for the measure is sample used for the measure is representative of the aspect to representative of the aspect to be measuredbe measured
IOP 301-TIOP 301-T
Content validityContent validity
It adopts a It adopts a subjectivesubjective approach approach whereby we have recourse, for e.g, whereby we have recourse, for e.g, to to expert opinionexpert opinion for the evaluation for the evaluation of items during the test of items during the test construction phase.construction phase.
IOP 301-TIOP 301-T
Content validityContent validity
Relevant for evaluatingRelevant for evaluating
AchievementAchievement EducationalEducational OccupationalOccupational
measures.measures.
IOP 301-TIOP 301-T
Content validityContent validity
Basic requirement forBasic requirement for Criterion-referencedCriterion-referenced Job sampleJob sample
measures, which are essential formeasures, which are essential for Employee selectionEmployee selection Employee classificationEmployee classification
IOP 301-TIOP 301-T
Content validityContent validity
Measures are interpreted in terms ofMeasures are interpreted in terms of
Mastery of knowledgeMastery of knowledge SkillsSkills
for a specific job.for a specific job.
IOP 301-TIOP 301-T
Content validityContent validity
Not appropriateNot appropriate for for
AptitudeAptitude PersonalityPersonality
measures since validation has measures since validation has to be made through criterion-to be made through criterion-prediction procedures.prediction procedures.
IOP 301-TIOP 301-T
Face validityFace validity
It is not validity in psychometric It is not validity in psychometric terms !terms !
It just refers to what the test It just refers to what the test appearsappears to measure and not to to measure and not to what it measures in fact.what it measures in fact.
IOP 301-TIOP 301-T
Face validityFace validity
It is not useless since the aim It is not useless since the aim may be achieved by using may be achieved by using appropriate phrasing appropriate phrasing only !only !
In a sense, it ensures relevance In a sense, it ensures relevance to the context by employing to the context by employing correct expressions.correct expressions.
IOP 301-TIOP 301-T
Criterion-related validityCriterion-related validity
DefinitionDefinition
A A criterion variablecriterion variable is one with is one with (or against) which psychological (or against) which psychological measures are compared or measures are compared or evaluated.evaluated.
A criterion must be A criterion must be reliablereliable ! !
IOP 301-TIOP 301-T
Criterion-related validityCriterion-related validity
It is a It is a quantitative quantitative procedure which procedure which involves calculating the involves calculating the correlation correlation coefficientcoefficient between one or more between one or more predictor variables and a criterion predictor variables and a criterion variable.variable.
IOP 301-TIOP 301-T
Criterion-related validityCriterion-related validity
The validity of the measure is also The validity of the measure is also determined by its determined by its ability to predictability to predict performance on the criterion.performance on the criterion.
IOP 301-TIOP 301-T
Criterion-related validityCriterion-related validity
Concurrent validityConcurrent validity Accuracy to identify the current Accuracy to identify the current
status regarding skills and status regarding skills and characteristicscharacteristics
Predictive validityPredictive validity Accuracy to forecast future Accuracy to forecast future
behaviour. It implicitly contains the behaviour. It implicitly contains the concept of decision-making concept of decision-making
IOP 301-TIOP 301-T
Criterion-related validityCriterion-related validity
Warning !Warning !
Sometimes a factor may affect Sometimes a factor may affect the criterion such that it is no the criterion such that it is no longer a valid measure.longer a valid measure.
This is known as This is known as criterion criterion contaminationcontamination..
IOP 301-TIOP 301-T
Criterion-related validityCriterion-related validity
Warning !Warning ! For example, the rater mightFor example, the rater might Be too lenientBe too lenient Commit Commit halo errorhalo error, i. e, rely on , i. e, rely on
impressionsimpressions
IOP 301-TIOP 301-T
Criterion-related validityCriterion-related validity
Warning !Warning ! Therefore, we must make sure Therefore, we must make sure
that the criterion is free from that the criterion is free from biasbias (prejudice). (prejudice).
Bias definitely influences the Bias definitely influences the correlation coefficient.correlation coefficient.
IOP 301-TIOP 301-T
Common criterion measuresCommon criterion measures
Academic achievementAcademic achievement Performance in specialised trainingPerformance in specialised training Job performanceJob performance Contrasted groupsContrasted groups Psychiatric diagnosesPsychiatric diagnoses
IOP 301-TIOP 301-T
Common criterion measuresCommon criterion measures
Academic achievementAcademic achievement Used for validation of intelligence, Used for validation of intelligence,
multiple aptitude and personality multiple aptitude and personality measures.measures.
Indices includeIndices include School, college or university gradesSchool, college or university grades Achievement test scoresAchievement test scores Special awardsSpecial awards
IOP 301-TIOP 301-T
Common criterion measuresCommon criterion measures
Performance in specialised trainingPerformance in specialised training
Used for specific aptitude measuresUsed for specific aptitude measures
Indices include training outcomes for Indices include training outcomes for Technical coursesTechnical courses Academic coursesAcademic courses
IOP 301-TIOP 301-T
Common criterion measuresCommon criterion measures
Job performanceJob performanceUsed for validating intelligence, special Used for validating intelligence, special
aptitude and personality measuresaptitude and personality measures
Indices include jobs in industry, business, Indices include jobs in industry, business, armed services, governmentarmed services, government
Tests describe duties performed and the Tests describe duties performed and the ways that they are measuredways that they are measured
IOP 301-TIOP 301-T
Common criterion measuresCommon criterion measures
Contrasted groupsContrasted groups
Sometimes used for validating personality Sometimes used for validating personality measuresmeasures
Relevant when distinguishing the nature Relevant when distinguishing the nature of occupations (e.g, social and non-of occupations (e.g, social and non-social – Public Relations Officer and social – Public Relations Officer and Clerk)Clerk)
IOP 301-TIOP 301-T
Common criterion measuresCommon criterion measures
Psychiatric diagnosesPsychiatric diagnoses
Used mainly for validating personality Used mainly for validating personality measuresmeasures
Based onBased on Prolonged observationProlonged observation Case historyCase history
IOP 301-TIOP 301-T
Common criterion measuresCommon criterion measures
Academic Academic achievementachievement
Performance Performance in specialised in specialised
trainingtraining
Job Job performanceperformance
Contrasted Contrasted groupsgroups
Psychiatric Psychiatric diagnosesdiagnoses
IntelligenceIntelligence
AptitudeAptitude
PersonalityPersonality
IOP 301-TIOP 301-T
Ratings Ratings
Suitable for almost any type of Suitable for almost any type of measuremeasure
Very subjective in natureVery subjective in nature Often the only source availableOften the only source available
IOP 301-TIOP 301-T
Ratings Ratings These are given by teachers, lecturers, These are given by teachers, lecturers,
instructors, supervisors, officers, etc,…instructors, supervisors, officers, etc,…
Raters may be trained to avoid common errors Raters may be trained to avoid common errors likelike
Halo errorHalo error AmbiguityAmbiguity Error of central tendencyError of central tendency LeniencyLeniency
IOP 301-TIOP 301-T
Criterion-related validityCriterion-related validity
We can always validate a new measure We can always validate a new measure by correlating it with another valid test by correlating it with another valid test (obviously, reliable as well !)(obviously, reliable as well !)
IOP 301-TIOP 301-T
Criterion-related validityCriterion-related validity
Some modern and popular Some modern and popular criterion-criterion-prediction proceduresprediction procedures which are now which are now widely used arewidely used are
Validity generalisationValidity generalisation Meta-analysisMeta-analysis Cross-validationCross-validation
IOP 301-TIOP 301-T
Criterion-related validityCriterion-related validity
Validity generalisationValidity generalisation
Schmidt, Hunter et al. showed that the Schmidt, Hunter et al. showed that the validity of tests measuring verbal, validity of tests measuring verbal, numeric and reasoning aptitudes can numeric and reasoning aptitudes can be generalised widely across be generalised widely across occupations (these require common occupations (these require common cognitive skills).cognitive skills).
IOP 301-TIOP 301-T
Criterion-related validityCriterion-related validity
Meta-analysisMeta-analysis Method of reviewing research Method of reviewing research
literatureliterature Statistical integration and analysis of Statistical integration and analysis of
previous and current findings on a previous and current findings on a topic. topic.
Validation by correlationValidation by correlation
IOP 301-TIOP 301-T
Criterion-related validityCriterion-related validity
Cross-validationCross-validation Refinement of initial measureRefinement of initial measure Application to another Application to another representative representative
normativenormative sample sample Recalculation of validity coefficientsRecalculation of validity coefficients Lowering of coefficient expected after Lowering of coefficient expected after
minimisation of minimisation of chance differenceschance differences and and sampling errorssampling errors (spuriousness) (spuriousness)
IOP 301-TIOP 301-T
Construct-identification validityConstruct-identification validity
Construct validity is the seConstruct validity is the sensitivity nsitivity of the instrument to pick up of the instrument to pick up minor variations in the concept minor variations in the concept being measured.being measured.
Can an instrument (questionnaire) to Can an instrument (questionnaire) to measure anxiety pick up different levels of measure anxiety pick up different levels of anxiety or just its presence or absence?anxiety or just its presence or absence?
IOP 301-TIOP 301-T
Construct-identification validityConstruct-identification validity Any data throwing light on the nature of the
trait and the conditions affecting its development and manifestations represent appropriate evidence for this validation.
Example
I have designed a program to lower girls’ Math phobia. The girls who complete my program should have lower scores on the Math Phobia Measure compared to their scores before the program and compared to the scores of girls who have not completed the program.
IOP 301-TIOP 301-T
Construct validity methodsConstruct validity methods
Correlational validityCorrelational validity Factor analysisFactor analysis Convergent and discriminant validityConvergent and discriminant validity
IOP 301-TIOP 301-T
Construct validity methodsConstruct validity methods
Correlational validityCorrelational validity
This involves correlating a new meThis involves correlating a new measure with similar previous measurasure with similar previous measures of the same name.es of the same name.
Warning !Warning !
High correlation may indicate dupliHigh correlation may indicate duplication of measures.cation of measures.
IOP 301-TIOP 301-T
Construct validity methodsConstruct validity methodsFactor analysis (FA)Factor analysis (FA) It is a multivariate statistical It is a multivariate statistical
technique which is used to group technique which is used to group multiple variables into a few multiple variables into a few factors.factors.
In doing FA you hope to find In doing FA you hope to find clusters of variables that can be clusters of variables that can be identified as new hypothetical identified as new hypothetical factors.factors.
IOP 301-TIOP 301-T
Construct validity methodsConstruct validity methods
Convergent and discriminant validityConvergent and discriminant validity
The idea is that a test should The idea is that a test should correlate highly with other similar correlate highly with other similar teststests and the test should and the test should correlate correlate poorly with tests that are very poorly with tests that are very dissimilardissimilar..
IOP 301-TIOP 301-T
Construct validity methodsConstruct validity methods
Convergent and discriminant validityConvergent and discriminant validity
ExampleExample
A newly developed test of motor A newly developed test of motor coordination should coordination should correlate highlycorrelate highly with other tests of motor coordination.with other tests of motor coordination.
It should also have It should also have low correlationlow correlation with tests that measure attitudes.with tests that measure attitudes.
IOP 301-TIOP 301-T
Indices and interpretation of validityIndices and interpretation of validity
Validity coefficientValidity coefficient
- Magnitude of coefficient- Magnitude of coefficient
- Factors affecting validity- Factors affecting validity Coefficient of determinationCoefficient of determination Standard error of estimationStandard error of estimation Regression analysis (prediction)Regression analysis (prediction)
IOP 301-TIOP 301-T
Indices and interpretation of validityIndices and interpretation of validity
Validity coefficientValidity coefficient
DefinitionDefinition
It is a correlation coefficient between It is a correlation coefficient between the criterion and the predictor(s) the criterion and the predictor(s) variables.variables.
IOP 301-TIOP 301-T
Indices and interpretation of validityIndices and interpretation of validity
Validity coefficientValidity coefficient
Differential validityDifferential validity refers to refers to differences in the magnitude of differences in the magnitude of the correlation coefficients for the correlation coefficients for different groups of test-takers.different groups of test-takers.
IOP 301-TIOP 301-T
Indices and interpretation of validityIndices and interpretation of validity
Magnitude of validity coefficientMagnitude of validity coefficient
Treated in the same way as the Treated in the same way as the Pearson correlation coefficient !Pearson correlation coefficient !
IOP 301-TIOP 301-T
Indices and interpretation of validityIndices and interpretation of validity
Factors affecting validityFactors affecting validity
Nature of the groupNature of the group Sample heterogeneitySample heterogeneity Criterion-predictor relationshipCriterion-predictor relationship Validity-reliability proportionalityValidity-reliability proportionality Criterion contaminationCriterion contamination Moderator variablesModerator variables
IOP 301-TIOP 301-T
Indices and interpretation of validityIndices and interpretation of validity
Factors affecting validityFactors affecting validity
Nature of the groupNature of the group Consistency of the validity Consistency of the validity
coefficient for subgroups which coefficient for subgroups which differ in any characteristic (e. differ in any characteristic (e. g. age, gender, educational g. age, gender, educational level, etc, …) level, etc, …)
IOP 301-TIOP 301-T
Indices and interpretation of validityIndices and interpretation of validity
Factors affecting validityFactors affecting validity
Sample heterogeneitySample heterogeneity
A wider range of scores results A wider range of scores results in a higher validity coefficient in a higher validity coefficient (range restriction phenomenon)(range restriction phenomenon)
IOP 301-TIOP 301-T
Indices and interpretation of validityIndices and interpretation of validity
Factors affecting validityFactors affecting validity
Criterion-predictor relationshipCriterion-predictor relationship There must be a There must be a linearlinear
relationship between predictor relationship between predictor and criterion. Otherwise, the and criterion. Otherwise, the Pearson correlation coefficient Pearson correlation coefficient would be of no use!would be of no use!
IOP 301-TIOP 301-T
Indices and interpretation of validityIndices and interpretation of validity
Factors affecting validityFactors affecting validity
Validity-reliability proportionalityValidity-reliability proportionality Reliability has a limiting Reliability has a limiting
influence on validity – we influence on validity – we simply cannot validate an simply cannot validate an unreliable measure!unreliable measure!
IOP 301-TIOP 301-T
Indices and interpretation of validityIndices and interpretation of validity
Factors affecting validityFactors affecting validity
Criterion contaminationCriterion contamination Get rid of bias by measuring Get rid of bias by measuring
contaminated influences. contaminated influences. Then correct this influence Then correct this influence statistically by use of statistically by use of partial partial correlationcorrelation..
IOP 301-TIOP 301-T
Indices and interpretation of validityIndices and interpretation of validity
Factors affecting validityFactors affecting validity
Moderator variablesModerator variables Variables like age, gender, Variables like age, gender,
personality characteristics may personality characteristics may help to predict performance for help to predict performance for particular variables only – keep particular variables only – keep them in mind!them in mind!
IOP 301-TIOP 301-T
Indices and interpretation of validityIndices and interpretation of validity
Coefficient of determinationCoefficient of determination Indicates the proportion of Indicates the proportion of
variance in the criterion variable variance in the criterion variable explained by the predictor.explained by the predictor.
E.g. If r = 0.9, rE.g. If r = 0.9, r2 2 = 0.81. 81% of = 0.81. 81% of the changes in the criterion is the changes in the criterion is accounted for by the predictor.accounted for by the predictor.
IOP 301-TIOP 301-T
Standard error of estimation (SE)Standard error of estimation (SE)
Treated just like the standard deviation.Treated just like the standard deviation. (True and predicted values for the (True and predicted values for the
criterion should differ by at most 1.96SE criterion should differ by at most 1.96SE at a 95% confidence level.)at a 95% confidence level.)
Indices and interpretation of validityIndices and interpretation of validity
21 xyyest rsSE
IOP 301-TIOP 301-T
Indices and interpretation of validityIndices and interpretation of validity
Regression analysisRegression analysis Mainly used to predict values of Mainly used to predict values of
the criterion variable. the criterion variable.
If If rr is high, prediction is more is high, prediction is more accurate.accurate.
Predicted values are obtained Predicted values are obtained from the from the line of best fitline of best fit..
IOP 301-TIOP 301-T
Indices and interpretation of validityIndices and interpretation of validity
Regression analysisRegression analysis
Linear regressionLinear regression It involves one criterion It involves one criterion
variable but may involve one variable but may involve one ((simple regressionsimple regression) or more ) or more than one predictor variable than one predictor variable ((multiple regressionmultiple regression).).
IOP 301-TIOP 301-T
Indices and interpretation of validityIndices and interpretation of validity
Regression analysisRegression analysis
IOP 301-TIOP 301-T
Indices and interpretation of validityIndices and interpretation of validity
Simple linear regressionSimple linear regression
Multiple linear regressionMultiple linear regression
bXaY ˆ
nn XbXbXbbY ...ˆ22110
IOP 301-TIOP 301-T
Reliability and ValidityReliability and Validity A valid test is always reliable (in order for a test tA valid test is always reliable (in order for a test t
o be valid, it needs to be reliable in the first placo be valid, it needs to be reliable in the first place)e)
A reliable test is not always valid.A reliable test is not always valid. Validity is more important than reliability.Validity is more important than reliability. To be useful, a measuring instrument (test, To be useful, a measuring instrument (test,
scale) must be both reasonably reliable scale) must be both reasonably reliable andand valid.valid.
Aim for validity first, and then try make the test mAim for validity first, and then try make the test more reliable little by little, rather than the other waore reliable little by little, rather than the other way around.y around.
IOP 301-TIOP 301-T
Reliability and ValidityReliability and Validity
IFIF THENTHEN
UnreliableUnreliable Reliable, but not Reliable, but not
validvalid Unreliable and Unreliable and
invalidinvalid Reliable and validReliable and valid
Test validity is undermined.Test validity is undermined. Test is not useful.Test is not useful.
Test is definitely NOT useful!Test is definitely NOT useful!
Test can be used with good Test can be used with good results.results.
IOP 301-TIOP 301-T
Optimising reliability and ValidityOptimising reliability and Validity The more questions the better (the number of
test items)
Ask questions several times in slightly different ways (homogeneity)
Get as many people as you can in your program (sample size n)
Get different kinds of people in your program (sample heterogeneity)
Linear relationship between the test and the criterion (Pearson correlation coefficient)
IOP 301-TIOP 301-T
Selecting and creating measuresSelecting and creating measures Define the construct(s) that you want to measure
clearly
Identify existing measures, particularly those with established reliability and validity
Determine whether those measures will work for your purpose and identify any areas where you may need to create a new measure or add new questions
Create additional questions/measures
Identify criteria that your measure should correlate with or predict, and develop procedures for assessing those criteria