Part 2: Quantitative Methods October 9, 2006. Validity Face –Does it appear to measure what it...
-
date post
20-Dec-2015 -
Category
Documents
-
view
217 -
download
1
Transcript of Part 2: Quantitative Methods October 9, 2006. Validity Face –Does it appear to measure what it...
![Page 1: Part 2: Quantitative Methods October 9, 2006. Validity Face –Does it appear to measure what it purports to measure? Content –Do the items cover the domain?](https://reader035.fdocuments.net/reader035/viewer/2022062714/56649d535503460f94a2f8bb/html5/thumbnails/1.jpg)
Part 2: Quantitative Methods
October 9, 2006
![Page 2: Part 2: Quantitative Methods October 9, 2006. Validity Face –Does it appear to measure what it purports to measure? Content –Do the items cover the domain?](https://reader035.fdocuments.net/reader035/viewer/2022062714/56649d535503460f94a2f8bb/html5/thumbnails/2.jpg)
Validity
• Face– Does it appear to measure what it purports to
measure?
• Content– Do the items cover the domain?
• Construct– Does it measure the unobservable attribute
that it purports to measure?
![Page 3: Part 2: Quantitative Methods October 9, 2006. Validity Face –Does it appear to measure what it purports to measure? Content –Do the items cover the domain?](https://reader035.fdocuments.net/reader035/viewer/2022062714/56649d535503460f94a2f8bb/html5/thumbnails/3.jpg)
Validity
• Criterion– Predictive – Concurrent
• Consequential
![Page 4: Part 2: Quantitative Methods October 9, 2006. Validity Face –Does it appear to measure what it purports to measure? Content –Do the items cover the domain?](https://reader035.fdocuments.net/reader035/viewer/2022062714/56649d535503460f94a2f8bb/html5/thumbnails/4.jpg)
Types of validity (cont.)
The construct
The instrument
Here the instrument samples some and only of the construct
![Page 5: Part 2: Quantitative Methods October 9, 2006. Validity Face –Does it appear to measure what it purports to measure? Content –Do the items cover the domain?](https://reader035.fdocuments.net/reader035/viewer/2022062714/56649d535503460f94a2f8bb/html5/thumbnails/5.jpg)
Types of validity
The instrument
The construct
Here the instrument samples all and more of the construct
![Page 6: Part 2: Quantitative Methods October 9, 2006. Validity Face –Does it appear to measure what it purports to measure? Content –Do the items cover the domain?](https://reader035.fdocuments.net/reader035/viewer/2022062714/56649d535503460f94a2f8bb/html5/thumbnails/6.jpg)
The construct
The instrument
Here the instrument fails to sample ANY of the construct
![Page 7: Part 2: Quantitative Methods October 9, 2006. Validity Face –Does it appear to measure what it purports to measure? Content –Do the items cover the domain?](https://reader035.fdocuments.net/reader035/viewer/2022062714/56649d535503460f94a2f8bb/html5/thumbnails/7.jpg)
The construct
The instrument
Here the instrument samples some but not all of the construct
![Page 8: Part 2: Quantitative Methods October 9, 2006. Validity Face –Does it appear to measure what it purports to measure? Content –Do the items cover the domain?](https://reader035.fdocuments.net/reader035/viewer/2022062714/56649d535503460f94a2f8bb/html5/thumbnails/8.jpg)
Perfection!
The construct and the instrument!
![Page 9: Part 2: Quantitative Methods October 9, 2006. Validity Face –Does it appear to measure what it purports to measure? Content –Do the items cover the domain?](https://reader035.fdocuments.net/reader035/viewer/2022062714/56649d535503460f94a2f8bb/html5/thumbnails/9.jpg)
Reliability and Validity
![Page 10: Part 2: Quantitative Methods October 9, 2006. Validity Face –Does it appear to measure what it purports to measure? Content –Do the items cover the domain?](https://reader035.fdocuments.net/reader035/viewer/2022062714/56649d535503460f94a2f8bb/html5/thumbnails/10.jpg)
In groups of 3 to 4
• Sampling– What is the target population?– What sampling procedure was used?– Do you think the sample is representative?
• Why or why not?
• Measurement– What types of reliability and validity evidence
are provided?– What else would you like to know?
![Page 11: Part 2: Quantitative Methods October 9, 2006. Validity Face –Does it appear to measure what it purports to measure? Content –Do the items cover the domain?](https://reader035.fdocuments.net/reader035/viewer/2022062714/56649d535503460f94a2f8bb/html5/thumbnails/11.jpg)
Ways to Classify Instruments
• Who Provides the Information?– Themselves: Self-report data – Directly or indirectly: from the subjects of the study – From informants (people who are knowledgeable
about the subjects and provide this information)
![Page 12: Part 2: Quantitative Methods October 9, 2006. Validity Face –Does it appear to measure what it purports to measure? Content –Do the items cover the domain?](https://reader035.fdocuments.net/reader035/viewer/2022062714/56649d535503460f94a2f8bb/html5/thumbnails/12.jpg)
Types of Researcher-completed Instruments
• Rating scales• Interview schedules• Tally sheets• Flowcharts
• Performance checklists
• Observation forms
![Page 13: Part 2: Quantitative Methods October 9, 2006. Validity Face –Does it appear to measure what it purports to measure? Content –Do the items cover the domain?](https://reader035.fdocuments.net/reader035/viewer/2022062714/56649d535503460f94a2f8bb/html5/thumbnails/13.jpg)
Excerpt from a Behavior Rating Scale for Teachers
Instructions: For each of the behaviors listedbelow, circle the appropriate number, using the following key: 5 = Excellent, 4 = Above Average, 3 = Average, 2 = Below Average,1 = Poor.
A. Explains course material clearly.1 2 3 4 5
B. Establishes rapport with students.1 2 3 4 5
C. Asks high-level questions.1 2 3 4 5
D. Varies class activities.1 2 3 4 5
![Page 14: Part 2: Quantitative Methods October 9, 2006. Validity Face –Does it appear to measure what it purports to measure? Content –Do the items cover the domain?](https://reader035.fdocuments.net/reader035/viewer/2022062714/56649d535503460f94a2f8bb/html5/thumbnails/14.jpg)
Excerpt from a Graphic Rating Scale
Instructions: Indicate the quality of the student’s participationin the following class activities by placing an X anywhere alongeach line.
AlwaysFrequently Occasionally Seldom Never
1. Listens to teacher’s instructions.
Always Frequently Occasionally Seldom Never
2. Listens to the opinions of other students.
Always Frequently Occasionally Seldom Never
3. Offers own opinions in class discussions.
![Page 15: Part 2: Quantitative Methods October 9, 2006. Validity Face –Does it appear to measure what it purports to measure? Content –Do the items cover the domain?](https://reader035.fdocuments.net/reader035/viewer/2022062714/56649d535503460f94a2f8bb/html5/thumbnails/15.jpg)
Sample Observation Form
![Page 16: Part 2: Quantitative Methods October 9, 2006. Validity Face –Does it appear to measure what it purports to measure? Content –Do the items cover the domain?](https://reader035.fdocuments.net/reader035/viewer/2022062714/56649d535503460f94a2f8bb/html5/thumbnails/16.jpg)
Discussion Analysis Tally Sheet
![Page 17: Part 2: Quantitative Methods October 9, 2006. Validity Face –Does it appear to measure what it purports to measure? Content –Do the items cover the domain?](https://reader035.fdocuments.net/reader035/viewer/2022062714/56649d535503460f94a2f8bb/html5/thumbnails/17.jpg)
Performance Checklist Noting Student Actions
![Page 18: Part 2: Quantitative Methods October 9, 2006. Validity Face –Does it appear to measure what it purports to measure? Content –Do the items cover the domain?](https://reader035.fdocuments.net/reader035/viewer/2022062714/56649d535503460f94a2f8bb/html5/thumbnails/18.jpg)
Types of Subject-completed Instruments
• Questionnaires• Self-checklists• Attitude scales• Personality
inventories
• Achievement/aptitude tests
• Performance tests• Projective devices
![Page 19: Part 2: Quantitative Methods October 9, 2006. Validity Face –Does it appear to measure what it purports to measure? Content –Do the items cover the domain?](https://reader035.fdocuments.net/reader035/viewer/2022062714/56649d535503460f94a2f8bb/html5/thumbnails/19.jpg)
Example of a Self-Checklist
![Page 20: Part 2: Quantitative Methods October 9, 2006. Validity Face –Does it appear to measure what it purports to measure? Content –Do the items cover the domain?](https://reader035.fdocuments.net/reader035/viewer/2022062714/56649d535503460f94a2f8bb/html5/thumbnails/20.jpg)
Example of Items from a Likert Scale
![Page 21: Part 2: Quantitative Methods October 9, 2006. Validity Face –Does it appear to measure what it purports to measure? Content –Do the items cover the domain?](https://reader035.fdocuments.net/reader035/viewer/2022062714/56649d535503460f94a2f8bb/html5/thumbnails/21.jpg)
Example of the Semantic Differential
![Page 22: Part 2: Quantitative Methods October 9, 2006. Validity Face –Does it appear to measure what it purports to measure? Content –Do the items cover the domain?](https://reader035.fdocuments.net/reader035/viewer/2022062714/56649d535503460f94a2f8bb/html5/thumbnails/22.jpg)
Pictorial Attitude Scale for Use with Young Children
![Page 23: Part 2: Quantitative Methods October 9, 2006. Validity Face –Does it appear to measure what it purports to measure? Content –Do the items cover the domain?](https://reader035.fdocuments.net/reader035/viewer/2022062714/56649d535503460f94a2f8bb/html5/thumbnails/23.jpg)
Sample Items from a Personality Inventory
![Page 24: Part 2: Quantitative Methods October 9, 2006. Validity Face –Does it appear to measure what it purports to measure? Content –Do the items cover the domain?](https://reader035.fdocuments.net/reader035/viewer/2022062714/56649d535503460f94a2f8bb/html5/thumbnails/24.jpg)
Sample Items from an Achievement Test
![Page 25: Part 2: Quantitative Methods October 9, 2006. Validity Face –Does it appear to measure what it purports to measure? Content –Do the items cover the domain?](https://reader035.fdocuments.net/reader035/viewer/2022062714/56649d535503460f94a2f8bb/html5/thumbnails/25.jpg)
Sample Item from an Aptitude Test
![Page 26: Part 2: Quantitative Methods October 9, 2006. Validity Face –Does it appear to measure what it purports to measure? Content –Do the items cover the domain?](https://reader035.fdocuments.net/reader035/viewer/2022062714/56649d535503460f94a2f8bb/html5/thumbnails/26.jpg)
Sample Items from an Intelligence Test
![Page 27: Part 2: Quantitative Methods October 9, 2006. Validity Face –Does it appear to measure what it purports to measure? Content –Do the items cover the domain?](https://reader035.fdocuments.net/reader035/viewer/2022062714/56649d535503460f94a2f8bb/html5/thumbnails/27.jpg)
Item Formats
• Questions used in a subject-completed instrument can take many forms but are classified as either selection or supply items.
• Examples of selection items are:• True-false items
• Matching items
• Multiple choice items
• Interpretive exercises
• Examples of supply items are:• Short answer items
• Essay questions
![Page 28: Part 2: Quantitative Methods October 9, 2006. Validity Face –Does it appear to measure what it purports to measure? Content –Do the items cover the domain?](https://reader035.fdocuments.net/reader035/viewer/2022062714/56649d535503460f94a2f8bb/html5/thumbnails/28.jpg)
Norm-Referenced vs. Criterion-Referenced Instruments
• All derived scores give meaning to individual scores by comparing them to the scores of a group.
• The group used to determine derived scores is called the norm group and the instruments that provide such scores are referred to as norm-referenced instruments.
• An alternative to the use of achievement or performance instruments is to use a criterion-referenced test.
• This is based on a specific goal or target (criterion) for each learner to achieve.
• The difference between the two tests is that the criterion referenced tests focus more directly on instruction.
![Page 29: Part 2: Quantitative Methods October 9, 2006. Validity Face –Does it appear to measure what it purports to measure? Content –Do the items cover the domain?](https://reader035.fdocuments.net/reader035/viewer/2022062714/56649d535503460f94a2f8bb/html5/thumbnails/29.jpg)
Experimental Research
![Page 30: Part 2: Quantitative Methods October 9, 2006. Validity Face –Does it appear to measure what it purports to measure? Content –Do the items cover the domain?](https://reader035.fdocuments.net/reader035/viewer/2022062714/56649d535503460f94a2f8bb/html5/thumbnails/30.jpg)
The (Never-Ending) Search for Causation
• Establishing causation among variables :
» Produces increased understanding of those variables
» Results in the ability to manipulate conditions in order
to produce desired changes
![Page 31: Part 2: Quantitative Methods October 9, 2006. Validity Face –Does it appear to measure what it purports to measure? Content –Do the items cover the domain?](https://reader035.fdocuments.net/reader035/viewer/2022062714/56649d535503460f94a2f8bb/html5/thumbnails/31.jpg)
Experimental Research
• Can demonstrate cause-and-effect very convincingly
• Very stringent research design requirements
• Experimental design requires:
» Random assignment to groups (experimental and
control)
» Independent treatment variable that can be applied to
the experimental group
» Dependent variable that can be measured in all groups
![Page 32: Part 2: Quantitative Methods October 9, 2006. Validity Face –Does it appear to measure what it purports to measure? Content –Do the items cover the domain?](https://reader035.fdocuments.net/reader035/viewer/2022062714/56649d535503460f94a2f8bb/html5/thumbnails/32.jpg)
Quasi-Experimental Research
• Used in place of experimental research when random
assignment to groups is not feasible
• Otherwise, very similar to true experimental research
![Page 33: Part 2: Quantitative Methods October 9, 2006. Validity Face –Does it appear to measure what it purports to measure? Content –Do the items cover the domain?](https://reader035.fdocuments.net/reader035/viewer/2022062714/56649d535503460f94a2f8bb/html5/thumbnails/33.jpg)
Fundamentals of Experimental and Quasi-Experimental Research
• Cause and effect:
» Incorporates a temporal element—the cause is a
condition that exists prior to the effect; effect is a
condition that occurs after the cause
» There exists a “logical connection”—cause-and-effect
is demonstrated when manipulation of the independent
variable results in differences in the dependent variable
(as evidenced by comparing the experimental group to
the control group)
![Page 34: Part 2: Quantitative Methods October 9, 2006. Validity Face –Does it appear to measure what it purports to measure? Content –Do the items cover the domain?](https://reader035.fdocuments.net/reader035/viewer/2022062714/56649d535503460f94a2f8bb/html5/thumbnails/34.jpg)
What Aids Our Causal Arguments?
• Theory– "causes certainly are connected to effects; but this is because our
theories connect them, not because the world is held together by cosmic glue. The world may be glued together by imponderables, but that is irrelevant for understanding causal explanation." Hanson, 1958.
• Temporal Elements
• Design– "No causation without manipulation" Rubin &
Holland
![Page 35: Part 2: Quantitative Methods October 9, 2006. Validity Face –Does it appear to measure what it purports to measure? Content –Do the items cover the domain?](https://reader035.fdocuments.net/reader035/viewer/2022062714/56649d535503460f94a2f8bb/html5/thumbnails/35.jpg)
Fundamentals of Experimental and Quasi-Experimental Research
• Random selection and random assignment :
» Distinguish between “selection” and “assignment”
» Random selection helps to assure population validity
» If you incorporate random assignment
Experimental research
» If you do not use random assignmentQuasi-experimental research
![Page 36: Part 2: Quantitative Methods October 9, 2006. Validity Face –Does it appear to measure what it purports to measure? Content –Do the items cover the domain?](https://reader035.fdocuments.net/reader035/viewer/2022062714/56649d535503460f94a2f8bb/html5/thumbnails/36.jpg)
Fundamentals of Experimental and Quasi-Experimental Research (cont’d.)
• When to use experimental research design :
» If you strongly suspect a cause-and-effect relationship
exists between two conditions, and
» The independent variable can be introduced to
participants and can be manipulated, and
» The resulting dependent variable can be measured for
all participants
![Page 37: Part 2: Quantitative Methods October 9, 2006. Validity Face –Does it appear to measure what it purports to measure? Content –Do the items cover the domain?](https://reader035.fdocuments.net/reader035/viewer/2022062714/56649d535503460f94a2f8bb/html5/thumbnails/37.jpg)
Internal and External Validity
• “Validity of research” refers to the degree to which the
conclusions are accurate and generalizable
• Both experimental and quasi-experimental research are
subject to threats to validity
• If threats are not controlled for, they may introduce error
into the study, which will lead to misleading conclusions
![Page 38: Part 2: Quantitative Methods October 9, 2006. Validity Face –Does it appear to measure what it purports to measure? Content –Do the items cover the domain?](https://reader035.fdocuments.net/reader035/viewer/2022062714/56649d535503460f94a2f8bb/html5/thumbnails/38.jpg)
Threats to External Validity
• External validity—extent to which the results can be
generalized to other groups or settings
» Population validity—degree of similarity among
sample used, population from which it came, and target
population
» Ecological validity—physical or emotional situation or
setting that may have been unique to the experiment
» If the treatment effects can be obtained only under a limited
set of conditions or only by the original researcher the findings
have low ecological validity.
![Page 39: Part 2: Quantitative Methods October 9, 2006. Validity Face –Does it appear to measure what it purports to measure? Content –Do the items cover the domain?](https://reader035.fdocuments.net/reader035/viewer/2022062714/56649d535503460f94a2f8bb/html5/thumbnails/39.jpg)
Threats to External Validity
• Selection bias – if sample is biased you cannot generalize to the
population.• Reactive effects
– Experimental setting - differs from natural setting.– Testing – pretest influences how subjects respond to
the treatment.• Multiple-treatment inference
– If the subjects are exposed to more than one treatment, then the findings could only be generalized to individuals exposed to the same treatments in the same order of presentation.
![Page 40: Part 2: Quantitative Methods October 9, 2006. Validity Face –Does it appear to measure what it purports to measure? Content –Do the items cover the domain?](https://reader035.fdocuments.net/reader035/viewer/2022062714/56649d535503460f94a2f8bb/html5/thumbnails/40.jpg)
Threats to Internal Validity
• Internal validity—extent to which differences on the
dependent variable are a direct result of the manipulation
of the independent variable» History—when factors other than treatment can exert influence
over the results; problematic over time
» Maturation—when changes occur in dependent variable that may be due to natural developmental changes; problematic over time
» Testing—pretest may give clues to treatment or posttest and may result in improved posttest scores
» Instrumentation – Nature of outcome measure has changed.
![Page 41: Part 2: Quantitative Methods October 9, 2006. Validity Face –Does it appear to measure what it purports to measure? Content –Do the items cover the domain?](https://reader035.fdocuments.net/reader035/viewer/2022062714/56649d535503460f94a2f8bb/html5/thumbnails/41.jpg)
Threats to Internal Validity (cont’d.)
» Regression – Tendency of extreme scores to be nearer to the mean at retest
» Differential selection of participants—participants are not selected/assigned randomly
» Attrition (mortality)—loss of participants
» Experimental treatment diffusion – Control conditions receive experimental treatment.
![Page 42: Part 2: Quantitative Methods October 9, 2006. Validity Face –Does it appear to measure what it purports to measure? Content –Do the items cover the domain?](https://reader035.fdocuments.net/reader035/viewer/2022062714/56649d535503460f94a2f8bb/html5/thumbnails/42.jpg)
Experimental and Quasi-Experimental Research Designs
• Commonly used experimental design notation :
» X1 = treatment group
» X2 = control/comparison group
» O = observation (pretest, posttest, etc.)
» R = random assignment
![Page 43: Part 2: Quantitative Methods October 9, 2006. Validity Face –Does it appear to measure what it purports to measure? Content –Do the items cover the domain?](https://reader035.fdocuments.net/reader035/viewer/2022062714/56649d535503460f94a2f8bb/html5/thumbnails/43.jpg)
Common Experimental Designs
• Single-group pretest-treatment-posttest design:
O X O
» Technically, a pre-experimental design (only one
group; therefore, no random assignment exists)
» Overall, a weak design
»Why?
![Page 44: Part 2: Quantitative Methods October 9, 2006. Validity Face –Does it appear to measure what it purports to measure? Content –Do the items cover the domain?](https://reader035.fdocuments.net/reader035/viewer/2022062714/56649d535503460f94a2f8bb/html5/thumbnails/44.jpg)
Common Experimental Designs (cont’d.)
• Two-group treatment-posttest-only design:
R X1 O
R X2 O
» Here, we have random assignment to experimental,
control groups
» A better design, but still weak—cannot be sure that
groups were equivalent to begin with
![Page 45: Part 2: Quantitative Methods October 9, 2006. Validity Face –Does it appear to measure what it purports to measure? Content –Do the items cover the domain?](https://reader035.fdocuments.net/reader035/viewer/2022062714/56649d535503460f94a2f8bb/html5/thumbnails/45.jpg)
Common Experimental Designs (cont’d.)
• Two-group pretest-treatment-posttest design:
R O X1 O
R O X2 O
» A substantially improved design—previously
identified errors have been reduced
![Page 46: Part 2: Quantitative Methods October 9, 2006. Validity Face –Does it appear to measure what it purports to measure? Content –Do the items cover the domain?](https://reader035.fdocuments.net/reader035/viewer/2022062714/56649d535503460f94a2f8bb/html5/thumbnails/46.jpg)
Common Experimental Designs (cont’d.)
• Solomon four-group design:
R O X1 O
R O X2 O
R X1 O
R X2 O
» A much improved design—how??
» One serious drawback—requires twice as many
participants
![Page 47: Part 2: Quantitative Methods October 9, 2006. Validity Face –Does it appear to measure what it purports to measure? Content –Do the items cover the domain?](https://reader035.fdocuments.net/reader035/viewer/2022062714/56649d535503460f94a2f8bb/html5/thumbnails/47.jpg)
Common Experimental Designs (cont’d.)
• Factorial designs:
R O X1 O
R O X2 O
R O X1 O
R O X2 O
» Incorporates two or more factors
» Enables researcher to detect differential differences (effects apparent only on certain combinations of levels of independent variables)
![Page 48: Part 2: Quantitative Methods October 9, 2006. Validity Face –Does it appear to measure what it purports to measure? Content –Do the items cover the domain?](https://reader035.fdocuments.net/reader035/viewer/2022062714/56649d535503460f94a2f8bb/html5/thumbnails/48.jpg)
Common Experimental Designs (cont’d.)
• Single-participant measurement-treatment-measurement designs:
O O O | X O X O | O O O
» Purpose is to monitor effects on one subject
» Results can be generalized only with great caution
![Page 49: Part 2: Quantitative Methods October 9, 2006. Validity Face –Does it appear to measure what it purports to measure? Content –Do the items cover the domain?](https://reader035.fdocuments.net/reader035/viewer/2022062714/56649d535503460f94a2f8bb/html5/thumbnails/49.jpg)
Common Quasi-Experimental Designs
• Posttest-only design with nonequivalent groups:
X1 O
X2 O
» Uses two groups from same population
» Questions must be addressed regarding equivalency of groups prior to introduction of treatment
![Page 50: Part 2: Quantitative Methods October 9, 2006. Validity Face –Does it appear to measure what it purports to measure? Content –Do the items cover the domain?](https://reader035.fdocuments.net/reader035/viewer/2022062714/56649d535503460f94a2f8bb/html5/thumbnails/50.jpg)
Common Quasi-Experimental Designs (cont’d.)
• Pretest-posttest design with nonequivalent groups:
O X1 O
O X2 O
» A stronger design—pretest may be used to establish
group equivalency
![Page 51: Part 2: Quantitative Methods October 9, 2006. Validity Face –Does it appear to measure what it purports to measure? Content –Do the items cover the domain?](https://reader035.fdocuments.net/reader035/viewer/2022062714/56649d535503460f94a2f8bb/html5/thumbnails/51.jpg)
Similarities Between Experimental and Quasi-Experimental Research
• Cause-and-effect relationship is hypothesized
• Participants are randomly assigned (experimental) or
nonrandomly assigned (quasi-experimental)
• Application of an experimental treatment by researcher
• Following the treatment, all participants are measured on
the dependent variable
• Data are usually quantitative and analyzed by looking for
significant differences on the dependent variable