The Theory of The Theory of Sampling and Sampling and MeasurementMeasurement
SamplingSampling
First step in implementing any First step in implementing any research design is to create a sample. research design is to create a sample.
We cannot study the theoretical We cannot study the theoretical population of all conceivable events population of all conceivable events (e.g., events that have not occurred), (e.g., events that have not occurred), nor can we usually study all instances nor can we usually study all instances of actual events. We select some of actual events. We select some instances to study and not others. instances to study and not others. Those we include are our Those we include are our samplesample. .
How our sample is selected is critical How our sample is selected is critical for external validity or generalizability.for external validity or generalizability.
Who do you want Who do you want to generalize to?to generalize to?Who do you want Who do you want to generalize to?to generalize to?
Groups in SamplingGroups in SamplingGroups in SamplingGroups in Sampling
Groups in SamplingGroups in SamplingGroups in SamplingGroups in SamplingThe theoretical The theoretical
populationpopulationThe theoretical The theoretical
populationpopulation
What population can What population can you get access to?you get access to?
What population can What population can you get access to?you get access to?
Groups in SamplingGroups in SamplingGroups in SamplingGroups in SamplingThe theoretical The theoretical
populationpopulationThe theoretical The theoretical
populationpopulation
Groups in SamplingGroups in SamplingGroups in SamplingGroups in SamplingThe Theoretical The Theoretical
PopulationPopulationThe Theoretical The Theoretical
PopulationPopulation
The study The study populationpopulationThe study The study populationpopulation
How can you get How can you get access to them?access to them?How can you get How can you get access to them?access to them?
Groups in SamplingGroups in SamplingGroups in SamplingGroups in SamplingThe theoretical The theoretical
populationpopulationThe theoretical The theoretical
populationpopulation
The study The study populationpopulationThe study The study populationpopulation
Groups in SamplingGroups in SamplingGroups in SamplingGroups in SamplingThe theoretical The theoretical
populationpopulationThe theoretical The theoretical
populationpopulation
The study The study populationpopulationThe study The study populationpopulation
The sampling The sampling frameframe
The sampling The sampling frameframe
Who is in your study?Who is in your study?Who is in your study?Who is in your study?
Groups in SamplingGroups in SamplingGroups in SamplingGroups in SamplingThe theoretical The theoretical
populationpopulationThe theoretical The theoretical
populationpopulation
The study The study populationpopulationThe study The study populationpopulation
The sampling The sampling frameframe
The sampling The sampling frameframe
Groups in SamplingGroups in SamplingGroups in SamplingGroups in SamplingThe theoretical The theoretical
populationpopulationThe theoretical The theoretical
populationpopulation
The study The study populationpopulationThe study The study populationpopulation
The sampling The sampling frameframe
The sampling The sampling frameframe
The sampleThe sampleThe sampleThe sample
Types of SamplesTypes of Samples
Probability Probability SamplingSampling Simple randomSimple random Stratified Stratified
randomrandom Cluster or area Cluster or area
randomrandom
Non-Probability Non-Probability SamplingSampling AccidentalAccidental Modal instanceModal instance ExpertExpert SnowballSnowball Case study Case study
(intentional (intentional selection)selection)
The Sampling The Sampling DistributionDistribution
The Sampling The Sampling DistributionDistribution
AverageAverageAverageAverage AverageAverageAverageAverage AverageAverageAverageAverage
4.44.24.03.83.63.43.23.0
15
10
5
0
The sampling The sampling distribution...distribution...The sampling The sampling distribution...distribution...
...is the distribution ...is the distribution of a statistic across of a statistic across an infinite number an infinite number
of samples.of samples.
SampleSample
4.44.24.03.83.63.43.23.0
5
0
5
0
SampleSample
4.44.24.03.83.63.43.23.0
5
0
5
0
SampleSample
4.44.24.03.83.63.43.23.0
5
0
5
0
Population ParameterPopulation ParameterPopulation ParameterPopulation Parameter
4.54.03.53.0
150
100
50
0
Self esteemSelf esteem
Fre
qu
ency
Fre
qu
ency
The population has The population has a mean of 3.75...a mean of 3.75...
The population has The population has a mean of 3.75...a mean of 3.75...
...and a ...and a standard unit standard unit
of .25.of .25.
...and a ...and a standard unit standard unit
of .25.of .25.
This meansThis meansAbout 64% of cases fall between 3.5 - 4.0.About 64% of cases fall between 3.5 - 4.0.
About 95% of cases fall between 3.25 - 4.25.About 95% of cases fall between 3.25 - 4.25.
about 99% of cases fall between 3.0 - 4.5about 99% of cases fall between 3.0 - 4.5
Sampling DistributionSampling DistributionSampling DistributionSampling Distribution
4.54.03.53.0
150
100
50
0
Self-esteemSelf-esteem
Fre
qu
ency
Fre
qu
ency
The population has The population has a mean of 3.75.a mean of 3.75.
The population has The population has a mean of 3.75.a mean of 3.75.
Sampling DistributionSampling DistributionSampling DistributionSampling Distribution
4.54.03.53.0
150
100
50
0
Self-esteemSelf-esteem
Fre
qu
ency
Fre
qu
ency
The population has The population has a mean of 3.75...a mean of 3.75...
The population has The population has a mean of 3.75...a mean of 3.75...
...and a ...and a standard standard
error of .25.error of .25.
...and a ...and a standard standard
error of .25.error of .25.
Inferring Population Inferring Population from Samplefrom Sample
Inferring Population Inferring Population from Samplefrom Sample
4.54.03.53.0
150
100
50
0
Self esteemSelf esteem
Fre
qu
ency
Fre
qu
ency
The sample has a The sample has a mean of 3.75...mean of 3.75...
The sample has a The sample has a mean of 3.75...mean of 3.75...
...and a ...and a standard standard deviation deviation
of .25.of .25.
...and a ...and a standard standard deviation deviation
of .25.of .25.
This meansThis means64% chance true population mean falls between 3.5 - 4.0.64% chance true population mean falls between 3.5 - 4.0.
95% chance true population mean falls between 3.25 - 4.25.95% chance true population mean falls between 3.25 - 4.25.
99% chance true population mean falls between 3.0 - 4.599% chance true population mean falls between 3.0 - 4.5
Figure 3.4 Labor Repression and Growth in Figure 3.4 Labor Repression and Growth in the Asian Cases, 1970-1981the Asian Cases, 1970-1981
Philippines
Thailand
Indonesia
Singapore
Taiw anMalaysiaKorea
-4
-2
0
2
4
6
8
10
0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0 4.5 5.0
Labor Repression
Growth in GDP per capita
Figure 3.5 Labor Repression and Growth in the Figure 3.5 Labor Repression and Growth in the Full Universe of Developing Countries,1970-1981Full Universe of Developing Countries,1970-1981
Israel Philippines
Syria
Iran
Korea Taiw an
BrazilMexico
Malaysia
Jamaica
Uganda
Botsw ana
Singapore
-4
-2
0
2
4
6
8
10
0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0 4.5 5.0
Labor Repression
Growth in GDP per capita
MeasurementMeasurement
Operationalization is the process of Operationalization is the process of translating theoretical constructs into translating theoretical constructs into observable indicators.observable indicators.
Construct validity and reliability are the Construct validity and reliability are the criteria we use to evaluate how well you criteria we use to evaluate how well you have operationalized your concepts.have operationalized your concepts.
Both matter regardless of the level of Both matter regardless of the level of measurement and whether you are using measurement and whether you are using qualitative or quantitative indicators.qualitative or quantitative indicators.
The Hierarchy of LevelsThe Hierarchy of Levels
NominalNominalNominalNominal
IntervalIntervalIntervalInterval
RatioRatioRatioRatio
Attributes are only named; weakest
Attributes can be ordered
Distance is meaningful
Absolute zero
OrdinalOrdinalOrdinalOrdinal
Nominal MeasurementNominal Measurement The values “The values “namename” the attribute ” the attribute
uniquely.uniquely. The name does The name does notnot imply any imply any
ordering of the cases.ordering of the cases.
Ordinal MeasurementOrdinal MeasurementWhen attributes can be When attributes can be rank-rank-
orderedordered…… Distances between attributes Distances between attributes do do
not have any meaningnot have any meaning..
Interval MeasurementInterval MeasurementWhen When distancedistance between attributes has between attributes has
meaning, for example, temperature meaning, for example, temperature (in Fahrenheit) -- distance from 30-(in Fahrenheit) -- distance from 30-40°F is same as distance from 70-40°F is same as distance from 70-80°F80°F
Note that Note that ratios don’t make any ratios don’t make any sense -sense -- 80°F is not - 80°F is not twicetwice as hot as as hot as 40°F.40°F.
Ratio MeasurementRatio Measurement Has an Has an absolute zero absolute zero that is that is
meaningfulmeaningful Can construct a meaningful Can construct a meaningful ratioratio
(fraction), for example, number of (fraction), for example, number of clients in past six monthsclients in past six months
Construct ValidityConstruct Validity
Key problem is that we have abstract Key problem is that we have abstract theoretical construct – power, democracy, theoretical construct – power, democracy, development, corruption, etc. – that we development, corruption, etc. – that we can never observe directly.can never observe directly.
Yet, to test propositions requires that we Yet, to test propositions requires that we have some indicator for the construct – or have some indicator for the construct – or at least have proxies that we can argue at least have proxies that we can argue are capturing some attributes of the are capturing some attributes of the construct.construct.
Our indicator is an analogy (to an Our indicator is an analogy (to an analogy).analogy).
Assessing Construct Assessing Construct ValidityValidity
Translation ValidityTranslation Validity Face Validity: plausible on its “face”Face Validity: plausible on its “face” Content Validity: matches lists of Content Validity: matches lists of
attributesattributes Criterion-related ValidityCriterion-related Validity
Predictive Validity: predicts accuratelyPredictive Validity: predicts accurately Concurrent Validity: distinguishes Concurrent Validity: distinguishes
appropriately between groupsappropriately between groups Convergent ValidityConvergent Validity Discriminant ValidityDiscriminant Validity
The Convergent PrincipleThe Convergent PrincipleThe Convergent PrincipleThe Convergent Principle
AlternativeAlternative measures of a measures of a construct construct shouldshould be strongly be strongly
correlated.correlated.
How It WorksHow It WorksHow It WorksHow It WorksTheoryTheoryTheoryTheory
Self-esteemSelf-esteemconstructconstruct
Item 1Item 1 Item 2Item 2 Item 3Item 3 Item 4Item 4
You You theorizetheorize that the that the items all reflect self-esteem.items all reflect self-esteem.
You You theorizetheorize that the that the items all reflect self-esteem.items all reflect self-esteem.
How It WorksHow It WorksHow It WorksHow It WorksTheoryTheoryTheoryTheory
ObservationObservationObservationObservation
Self-esteemSelf-esteemconstructconstruct
Item 1Item 1 Item 2Item 2 Item 3Item 3 Item 4Item 4
1.001.00 .83.83 .89.89 .91.91.83.83 1.001.00 .85.85 .90.90.89.89 .85.85 1.001.00 .86.86.91.91 .90.90 .86.86 1.001.00
The correlations provide evidence The correlations provide evidence that the items all that the items all convergeconverge
on the same construct.on the same construct.
The correlations provide evidence The correlations provide evidence that the items all that the items all convergeconverge
on the same construct.on the same construct.
Convergent Validity in Convergent Validity in Measures of “Democracy”Measures of “Democracy”
19851985 | polity2 pollib civlib | polity2 pollib civlib regreg
-------------+-------------------------------------------------+------------------------------------
polity2 | 1.0000 -0.9148 -0.8770 -0.8601 polity2 | 1.0000 -0.9148 -0.8770 -0.8601
pollib | -0.9148 1.0000 0.9176 0.8440 pollib | -0.9148 1.0000 0.9176 0.8440
civlib | -0.8770 0.9176 1.0000 0.8053 civlib | -0.8770 0.9176 1.0000 0.8053
reg | -0.8601 0.8440 0.8053 1.0000reg | -0.8601 0.8440 0.8053 1.0000
Convergent Validity in Convergent Validity in Measures of “Education”Measures of “Education”
19851985 | 1| 1 22 33 44 55 66-------------+-------------------------------------------------------------------+------------------------------------------------------Ed. spending Ed. spending | 1.0000 -0.1217 0.2415 0.3563 0.0214 | 1.0000 -0.1217 0.2415 0.3563 0.0214
0.01950.0195Illiteracy (%) Illiteracy (%) | -0.1217 1.0000 -0.5797 -0.7306 -0.8569 - | -0.1217 1.0000 -0.5797 -0.7306 -0.8569 -
0.61960.6196Cohort to Grade 4 | 0.2415 -0.5797 1.0000 0.4419 0.6553 0.3654Cohort to Grade 4 | 0.2415 -0.5797 1.0000 0.4419 0.6553 0.3654% Grade School % Grade School | 0.3563 -0.7306 0.4419 1.0000 0.6230 | 0.3563 -0.7306 0.4419 1.0000 0.6230
0.36120.3612% Secondary School | 0.0214 -0.8569 0.6553 0.6230 1.0000 0.7576% Secondary School | 0.0214 -0.8569 0.6553 0.6230 1.0000 0.7576% College % College | 0.0195 -0.6196 0.3654 0.3612 0.7576 1.0000 | 0.0195 -0.6196 0.3654 0.3612 0.7576 1.0000
The Discriminant The Discriminant PrinciplePrinciple
The Discriminant The Discriminant PrinciplePrinciple
Measures of Measures of differentdifferent constructs should constructs should notnot
correlate highly with each correlate highly with each other.other.
How It WorksHow It WorksHow It WorksHow It WorksTheoryTheoryTheoryTheory
Self-esteemSelf-esteemconstructconstruct
SESE11 SESE22
Locus-of-controlLocus-of-controlconstructconstruct
LOCLOC11 LOCLOC22
How It WorksHow It WorksHow It WorksHow It WorksTheoryTheoryTheoryTheory
Self- esteemSelf- esteemconstructconstruct
SESE11 SESE22
Locus-of-controlLocus-of-controlconstructconstruct
LOCLOC11 LOCLOC22
You theorize that you have two distinguishable constructs.
How It WorksHow It WorksHow It WorksHow It WorksTheoryTheoryTheoryTheory
Self-esteemSelf-esteemconstructconstruct
SESE11 SESE22
Locus-of-controlLocus-of-controlconstructconstruct
LOCLOC11 LOCLOC22
ObservationObservationObservationObservation
rrSESE11, , LOCLOC11
= .12 = .12
rrSESE11, , LOCLOC22
= .09 = .09
rrSESE22, , LOCLOC11
= .04 = .04
rrSESE22, , LOCLOC22
= .11 = .11
The correlations provide evidence that the items on the two tests discriminate.
TheoryTheoryTheoryTheory
Self-esteemSelf-esteemconstructconstruct
SESE11 SESE22 SESE33
Locus-of-controlLocus-of-controlconstructconstruct
LOCLOC11 LOCLOC22 LOCLOC33
We have two constructs. We have two constructs. We want to measure We want to measure self-esteemself-esteem
and and locus of control.locus of control.
For each construct, we develop threeFor each construct, we develop threescale items; our theory is that itemsscale items; our theory is that items
within the construct will converge andwithin the construct will converge andItems across constructs will discriminate.Items across constructs will discriminate.
TheoryTheoryTheoryTheory
ObservationObservationObservationObservation
Self-esteemSelf-esteemConstructConstruct
SESE11 SESE22 SESE33
Locus-of-controlLocus-of-controlconstructconstruct
LOCLOC11 LOCLOC22 LOCLOC33
1.001.00 .83.83 .89.89 .02.02 .12.12 .09.09.83.83 1.001.00 .85.85 .05.05 .11.11 .03.03.89.89 .85.85 1.001.00 .04.04 .00.00 .06.06.02.02 .05.05 .04.04 1.001.00 .84.84 .93.93.12.12 .11.11 .00.00 .84.84 1.001.00 .91.91.09.09 .03.03 .06.06 .93.93 .91.91 1.001.00
SESE11
SESE22
SESE33
LOCLOC11
LOCLOC22
LOCLOC33
SESE11 SESE22 SESE33 LOCLOC11 LOCLOC22 LOCLOC33
GreenGreen and and redredcorrelations arecorrelations are
Convergent;Convergent;yellow areyellow are
Discriminant.Discriminant.
TheoryTheoryTheoryTheory
ObservationObservationObservationObservation
Self-esteemSelf-esteemconstructconstruct
SESE11 SESE22 SESE33
Locus-of-controlLocus-of-controlconstructconstruct
LOCLOC11 LOCLOC22 LOCLOC33
1.001.00 .83.83 .89.89 .02.02 .12.12 .09.09.83.83 1.001.00 .85.85 .05.05 .11.11 .03.03.89.89 .85.85 1.001.00 .04.04 .00.00 .06.06.02.02 .05.05 .04.04 1.001.00 .84.84 .93.93.12.12 .11.11 .00.00 .84.84 1.001.00 .91.91.09.09 .03.03 .06.06 .93.93 .91.91 1.001.00
SESE11
SESE22
SESE33
LOCLOC11
LOCLOC22
LOCLOC33
SESE11 SESE22 SESE33 LOCLOC11 LOCLOC22 LOCLOC33
The correlations support bothThe correlations support bothconvergenceconvergence and and discriminationdiscrimination,,and therefore and therefore constructconstruct validity. validity.
What Is Reliability?What Is Reliability?What Is Reliability?What Is Reliability?
The “repeatability” of a measureThe “repeatability” of a measure The “consistency” of a measureThe “consistency” of a measure The “dependability” of a measureThe “dependability” of a measure
True Score TheoryTrue Score Theory
12
34
5
12
34
5
3Scan a multitu
de of inform
ation and decide
what is im
portant.
12
34
5
12
34
5
12
34
5
12
34
5
12
34
5
1Manage tim
e effectively
2Manage resources effectively.
3Scan a multitu
de of information and
decide what is im
portant.
4Decide how to manage multip
le tasks.
5Organize the work when directions are not specific
.
1Manage tim
e effectively
Rating Sheet
ObservedObservedscorescore
ObservedObservedscorescore ==
TrueTrueabilityabilityTrueTrue
abilityability++ RandomRandom
errorerrorRandomRandom
errorerror
TT ee++XX
The Error ComponentThe Error Component
TT ee++XX
Two components:Two components:
• Random errorRandom error• Random errorRandom error
• Systematic errorSystematic error• Systematic errorSystematic error
eerr
eess
The Revised True Score The Revised True Score ModelModel
TT eerr++XX eess++
Random ErrorRandom Error
XX
Fre
qu
ency
Fre
qu
ency
The distribution of X The distribution of X with no random errorwith no random errorThe distribution of X The distribution of X with no random errorwith no random error
Random ErrorRandom Error
XX
Fre
qu
ency
Fre
qu
ency
The distribution of X The distribution of X with no random errorwith no random errorThe distribution of X The distribution of X with no random errorwith no random error
The distribution of X The distribution of X with random errorwith random errorThe distribution of X The distribution of X with random errorwith random error
Notice that random error doesn’t Notice that random error doesn’t affect the average, only the affect the average, only the
variability variability around the average.around the average.
Systematic ErrorSystematic Error
XX
Fre
qu
ency
Fre
qu
ency
The distribution of X The distribution of X with no systematic errorwith no systematic errorThe distribution of X The distribution of X with no systematic errorwith no systematic error
Systematic ErrorSystematic Error
XX
Fre
qu
ency
Fre
qu
ency
The distribution of X The distribution of X with no systematic errorwith no systematic errorThe distribution of X The distribution of X with no systematic errorwith no systematic error
The distribution of X The distribution of X with systematic errorwith systematic errorThe distribution of X The distribution of X with systematic errorwith systematic error
Notice that systematic error doesNotice that systematic error doesaffect the average; we call affect the average; we call
this a this a biasbias..
If a Measure Is If a Measure Is Reliable...Reliable...
If a Measure Is If a Measure Is Reliable...Reliable...
XX11XX11 XX22XX22
We should see that a person’s score on the same test We should see that a person’s score on the same test given twice is given twice is similarsimilar (assuming the trait being (assuming the trait being
measured isn’t changing).measured isn’t changing).
If a Measure Is If a Measure Is Reliable...Reliable...
If a Measure Is If a Measure Is Reliable...Reliable...
XX11XX11 XX22XX22
T + eT + e11 T + eT + e22
Recall from true score theory that...Recall from true score theory that...
But, if the scores are similar, But, if the scores are similar, whywhy are they similar? are they similar?
If a Measure Is If a Measure Is Reliable...Reliable...
If a Measure Is If a Measure Is Reliable...Reliable...
XX11XX11 XX22XX22
T + eT + e11 T + eT + e22
The only thing common to the two measures is the true The only thing common to the two measures is the true score, T. Therefore, the score, T. Therefore, the true score true score must determine the must determine the
reliability.reliability.
Reliability Is...Reliability Is...Reliability Is...Reliability Is...
a a ratioratioa a ratioratio
variance of the true scoresvariance of the true scores
variance of the measurevariance of the measure
var(T)var(T)
var(X)var(X)
Reliability Is...Reliability Is...Reliability Is...Reliability Is...
a a ratioratioa a ratioratio
variance of the true scoresvariance of the true scores
variance of the measurevariance of the measure
We can measure the variance of the observed score, X.We can measure the variance of the observed score, X.The greater the variance, the less reliable the measure.The greater the variance, the less reliable the measure.
This Leads Us to...This Leads Us to...This Leads Us to...This Leads Us to...
We cannot calculate reliability We cannot calculate reliability exactly; we can only exactly; we can only estimateestimate it. it.
Each estimate attempts to capture Each estimate attempts to capture the consequences of the true score the consequences of the true score in different ways.in different ways.
We want both We want both Reliability and Reliability and
ValidityValidity
Reliability and ValidityReliability and Validity
Reliable but not validReliable but not validReliable but not validReliable but not valid
Reliability and ValidityReliability and Validity
Valid but not reliableValid but not reliableValid but not reliableValid but not reliable
Reliability and ValidityReliability and Validity
Neither reliable nor validNeither reliable nor validNeither reliable nor validNeither reliable nor valid
Reliability and ValidityReliability and Validity
Reliable and validReliable and validReliable and validReliable and valid
Assignment #1Assignment #1 Assess the validity and reliability of the Assess the validity and reliability of the
IRIS-3 International Country Risk Guide.IRIS-3 International Country Risk Guide. Can examine a single instance, compare Can examine a single instance, compare
instances, analyze the full variation in the instances, analyze the full variation in the dataset, compare with additional measures, dataset, compare with additional measures, or use any other form of assessment. May or use any other form of assessment. May use outside sources of data, history, or use outside sources of data, history, or analysis (but document).analysis (but document).
The only restriction is that the paper must The only restriction is that the paper must be empirical and examine issues of validity be empirical and examine issues of validity and reliability. and reliability.
3-5 pages. Be concise. 3-5 pages. Be concise. Due Monday 10/24 at beginning of class. Due Monday 10/24 at beginning of class.
Top Related