Variables and Control

Basic Principles of Research DesignBasic Principles of Research Design

Overview of Tonight’s Lecture Variables and Control

2 variables we can control (subject + situation)

Research designs are rules about controls and variables evaluated in terms of ValidityValidity (answers the question set

to address) Internal Validity (relevant variables are controlled

to answer question); 8 internal validity factors External Validity (findings can be generalized to

other settings and populations); 4 external validity factors

Observations are evaluated in terms of ReliabilityReliability

Basic Principles of Research Design

One of the most important aspects of One of the most important aspects of research in CDIS is the careful selection of research in CDIS is the careful selection of the variables that are to be controlled or the variables that are to be controlled or systematically varied.systematically varied.

After a problem has been identified and the After a problem has been identified and the purpose specified, there must be careful purpose specified, there must be careful consideration of which variables are most consideration of which variables are most relevant to the problem and the purpose.relevant to the problem and the purpose.

2 Purposes of Research Designs

Answer the research questionAnswer the research question

Reduce or eliminate contamination of Reduce or eliminate contamination of extraneous variablesextraneous variables

In experimental designs, this means …. Manipulate the IV in order to answer the Manipulate the IV in order to answer the

questionquestion

Arrange the experiment so that extraneous Arrange the experiment so that extraneous variables are controlledvariables are controlled

In descriptive designs, this means ….

Select the variables for observation in order Select the variables for observation in order to answer the questionto answer the question

make the observations in a systematic and make the observations in a systematic and unobtrusive manner so that the criterion unobtrusive manner so that the criterion variables are not confounded by the variables are not confounded by the extraneous variablesextraneous variables

RESEARCH DESIGNS are the rules for observation

- research designs are rules about controls and variables and controlled variation

Research design is described in terms of: systematic variation of the IVsystematic variation of the IV observation of the effects of the IV on the observation of the effects of the IV on the

DVDV systematic control of other variables that systematic control of other variables that

might produce variation in the DVmight produce variation in the DV

Researchers systematically vary the IV by Researchers systematically vary the IV by controllingcontrolling its variation its variation

a detailed explanation of a detailed explanation of controls controls and and variablesvariables is essential to understanding the is essential to understanding the basic principles or rules of research designbasic principles or rules of research design

Control and Variability

Control is a restriction of natural variationControl is a restriction of natural variation represent opposite extremes of ORDER and represent opposite extremes of ORDER and

DISORDERDISORDER both are essential to researchboth are essential to research IV is controlled variationIV is controlled variation

2 Controlled Variables in Human Communication Research Subject (participant) variablesSubject (participant) variables

characteristics of subjectscharacteristics of subjects Situation variablesSituation variables

characteristics of the situation in which characteristics of the situation in which the observations are madethe observations are made

In carefully controlled studies, subject and situation variables are controlled.

- only the phenomena of interest are allowed to vary.

- Control of the subject and situation variables permits researchers to study the effects of the variables in which they are interested.

Controls are essential for achieving the purposes of research.

- if the variables under study are not isolated by controlling the relevant variables, alternative explanations of the findings cannot be ruled out.

- the extent to which each kind of variation is controlled depends on the research design.

Control of Subject Variables The type of communication disorder to be The type of communication disorder to be

studied is controlled by the selection of studied is controlled by the selection of subjects who have that type of disordersubjects who have that type of disorder

Subject variables (e.g., age, gender, history Subject variables (e.g., age, gender, history of tx, severity) are controlled by holding of tx, severity) are controlled by holding them constantthem constant

If these variables were not controlled, the If these variables were not controlled, the effects of the disorder might be effects of the disorder might be confoundedconfounded by the effects of the uncontrolled variablesby the effects of the uncontrolled variables

Control of Situation Variables The research design also controls the The research design also controls the

situation variablessituation variables These include variables such as time of day These include variables such as time of day

and instructions given to subjectsand instructions given to subjects The need for situational controls varies with The need for situational controls varies with

the type of designthe type of design EX: If you’re interested in determining how EX: If you’re interested in determining how

sounds are detected in real-life situations, you sounds are detected in real-life situations, you will not control background noise.will not control background noise.

Criteria for Evaluating Research Designs A research design is evaluated in its ability A research design is evaluated in its ability

to achieve its 2 purposes (answer the to achieve its 2 purposes (answer the question AND reduce/eliminate question AND reduce/eliminate contamination of extraneous variables) in contamination of extraneous variables) in terms of INTERNAL and EXTERNAL terms of INTERNAL and EXTERNAL VALIDITYVALIDITY

Therefore, research designs are evaluated in Therefore, research designs are evaluated in terms of VALIDITY (internal and external)terms of VALIDITY (internal and external)

Internal Validity

Ability to isolate effects of the IVAbility to isolate effects of the IV means that we can believe that the means that we can believe that the

manipulation of the IV manipulation of the IV diddid cause the cause the observed change in the DVobserved change in the DV

a study is internally valid when relevant a study is internally valid when relevant variables have been controlled and the only variables have been controlled and the only variables that affect the DV’s are the IV’svariables that affect the DV’s are the IV’s

Internal Validity (con’t)

When uncontrolled variables can affect the When uncontrolled variables can affect the DV, the effects of the IV are DV, the effects of the IV are confoundedconfounded with the effects of the uncontrolled with the effects of the uncontrolled variablesvariables

internal validity indicates that the research internal validity indicates that the research design accomplished what it was suppose to design accomplished what it was suppose to accomplish, i.e., answer the question.accomplish, i.e., answer the question.

External Validity Means that the findings from the study can be Means that the findings from the study can be

generalizedgeneralized to other populations and settings to other populations and settings external validity is the extent to which the external validity is the extent to which the

effects of the IVs on the DVs in the research effects of the IVs on the DVs in the research situation apply to the natural settings.situation apply to the natural settings.

Indicates the degree to which the findings from Indicates the degree to which the findings from the study can be generalizedthe study can be generalized

if findings from a study do not external if findings from a study do not external validity, they may not be of any practical use validity, they may not be of any practical use or value.or value.

The primary thing in evaluating experimental research is whether you can believe with enough confidence that the change in the DV was caused by manipulation of the IV

- the fewer the alternative explanations for the observed change in the DV, the greater the internal validity of the study.

8 Factors that Affect Internal Validity historyhistory maturationmaturation test-practice effectstest-practice effects instrumentationinstrumentation statistical regressionstatistical regression differential selection of subjectsdifferential selection of subjects mortalitymortality Hawthorne effectHawthorne effect

History

Events that occur between the first and Events that occur between the first and subsequent measurements subsequent measurements in additionin addition to the to the experimental variable (“in the gap”)experimental variable (“in the gap”)

these are the events that occur outside the these are the events that occur outside the experimental setting and therefore are experimental setting and therefore are outside the control of the experimenteroutside the control of the experimenter EX: prior treatment received in another EX: prior treatment received in another

setting; history of ear infectionssetting; history of ear infections

History (con’t)

Descriptive studies: longitudinal studies Descriptive studies: longitudinal studies more susceptible to history effects than more susceptible to history effects than cross-sectional studiescross-sectional studies

Experimental studies: one group pretest-Experimental studies: one group pretest-posttest design is susceptible to history posttest design is susceptible to history effectseffects

Maturation Changes in the subjects that cannot be Changes in the subjects that cannot be

controlled by the experimenter (“time-tied”)controlled by the experimenter (“time-tied”) these changes may cause effects that are these changes may cause effects that are

attributed incorrectly to the experimental attributed incorrectly to the experimental variablevariable EX: age changes, changes in biological EX: age changes, changes in biological

or psychological processes that take place or psychological processes that take place over time; even short-term changes, such over time; even short-term changes, such as changes in motivation, interest, as changes in motivation, interest, attentionattention

Solution to maturational effects in studies: Keep experimental sessions relatively shortKeep experimental sessions relatively short randomize or counterbalance conditionsrandomize or counterbalance conditions randomly assign subjects to experimental randomly assign subjects to experimental

and control groupsand control groups

Test-Practice Effects The effect of taking a test may influence the The effect of taking a test may influence the

subjects’ performance when they retake the subjects’ performance when they retake the testtest practice with the test, familiarity with the practice with the test, familiarity with the

test items, reduction of test anxiety, etc.test items, reduction of test anxiety, etc. pretest-posttest designs are especially pretest-posttest designs are especially

vulnerable to this problemvulnerable to this problem must be aware of the effect of testing on must be aware of the effect of testing on

behavior when evaluating findings from behavior when evaluating findings from researchresearch

Instrumentation

Changes in the calibration of an instrument or Changes in the calibration of an instrument or changes in the observers or scorers may produce changes in the observers or scorers may produce changes in the observed measurements.changes in the observed measurements. EX: (a) faulty or changing calibration of equipment; (b) EX: (a) faulty or changing calibration of equipment; (b)

rating scales, questionnaires, attitude inventories that rating scales, questionnaires, attitude inventories that have not been standardized or are not reliable; (c) biases have not been standardized or are not reliable; (c) biases of judges or observers used to collect data for the study; of judges or observers used to collect data for the study; (d) poor test environment which may contaminate (d) poor test environment which may contaminate observations due to distractions, noise, interruptions, etc.observations due to distractions, noise, interruptions, etc.

Statistical Regression

The migration of extreme scores toward the meanThe migration of extreme scores toward the mean atypical scores become more typical (elastic band)atypical scores become more typical (elastic band)

did change in performance occur because of tx did change in performance occur because of tx or because of regression to the mean?or because of regression to the mean?

occurs because subjects elected on basis of their occurs because subjects elected on basis of their extreme scores.extreme scores. Solution: random selection of subjects and Solution: random selection of subjects and

random assignment of subjects to groupsrandom assignment of subjects to groups

Differential Selection of Subjects Inherent differences between subjects assigned to Inherent differences between subjects assigned to

groups may account for the treatment effects groups may account for the treatment effects rather than the treatment itself.rather than the treatment itself. EX: SES, IQEX: SES, IQ Solution: subjects should be equal on Solution: subjects should be equal on

important dimensions prior to the study and important dimensions prior to the study and systematic differences between comparison systematic differences between comparison groups must be minimized with the exception groups must be minimized with the exception of the criterion variable under study. Further, of the criterion variable under study. Further, there should be no overlap between groups on there should be no overlap between groups on the variable that presumably distinguishes the the variable that presumably distinguishes the groups from one another.groups from one another.

Mortality

Differential loss of subjects between Differential loss of subjects between experimental and control groups or between experimental and control groups or between comparison groupscomparison groups EX: subjects who drop out may be EX: subjects who drop out may be

different in important respects from those different in important respects from those subjects who continue to participate in subjects who continue to participate in the study.the study.

Hawthorne Effect

Changes in a subject’s behavior that occur Changes in a subject’s behavior that occur because the subject knows he/she is because the subject knows he/she is participating in a study (“on stage”)participating in a study (“on stage”) Solution: apply Hawthorne effect to Solution: apply Hawthorne effect to

control group; begin sessions with a brief control group; begin sessions with a brief period of acclimation to establish rapport.period of acclimation to establish rapport.

Finally, you can have an interaction of any one or combination of the above factors that affect internal validity.

External Validity (Generalizability)

External validity is qualitatively different from internal External validity is qualitatively different from internal validity.validity.

If a study is internally invalid, then the results are If a study is internally invalid, then the results are meaningless.meaningless.

If, however, the study is externally invalid, the results If, however, the study is externally invalid, the results are only limited to the degree to which the internally are only limited to the degree to which the internally valid results can be generalizedvalid results can be generalized

A study can be internally valid, but externally invalid. A study can be internally valid, but externally invalid. However, a study cannot be externally valid, but However, a study cannot be externally valid, but internally invalidinternally invalid

4 Factors that Affect External Validity Subject selectionSubject selection reactive or interactive effects of pretestingreactive or interactive effects of pretesting reactive arrangementsreactive arrangements multiple treatment interferencemultiple treatment interference

Subject Selection

The representativeness of subjects selected The representativeness of subjects selected from the population to which the researcher from the population to which the researcher hopes to generalize the findings.hopes to generalize the findings. The relevant characteristics of the The relevant characteristics of the

subjects that determine their degree of subjects that determine their degree of representativeness should be specified in representativeness should be specified in the article to allow readers to evaluate the the article to allow readers to evaluate the generality of the results to other subjectsgenerality of the results to other subjects

Subject Selection (con’t)

Investigators should report what the Investigators should report what the relevant characteristics were in selecting the relevant characteristics were in selecting the subjects AND what the characteristics were subjects AND what the characteristics were of the subjects on those relevant of the subjects on those relevant characteristics (e.g., age, gender, severity, characteristics (e.g., age, gender, severity, etc.)etc.)

this factor affects the generalizability of this factor affects the generalizability of findings to other subjectsfindings to other subjects

Reactive or Interactive Effects of Pretesting The degree to which a reactive pretest may The degree to which a reactive pretest may

interact with an IV in determining the subjects’ interact with an IV in determining the subjects’ performance on the DVperformance on the DV that is, subjects who take a reactive pretest may that is, subjects who take a reactive pretest may

react to an experimental treatment in a different react to an experimental treatment in a different way from people who have not been exposed to way from people who have not been exposed to the pretest.the pretest.

The test itself may change the subjects’ The test itself may change the subjects’ responses.responses.

Reactive or Interactive Effects of Pretesting (con’t)

Therefore, the effect of pretesting may Therefore, the effect of pretesting may limit the generalization that can be made limit the generalization that can be made to people who have not been pretestedto people who have not been pretested

this factor affects the generalizability of this factor affects the generalizability of findings to other measures and subjectsfindings to other measures and subjects

Reactive Arrangements The degree to which the setting of the The degree to which the setting of the

research is reactive or interacts with the IV research is reactive or interacts with the IV in determining the subjects’ performance on in determining the subjects’ performance on the DV.the DV. For example, the child may respond in a For example, the child may respond in a

certain way with the clinician in the certain way with the clinician in the therapy room, but not in their classroomtherapy room, but not in their classroom

this factor affects the generalizability of this factor affects the generalizability of findings to other settings.findings to other settings.

Multiple Treatment Interference The degree to which various parts of multiple The degree to which various parts of multiple

treatment interact with each other in determining treatment interact with each other in determining the subjects’ performance on the DV.the subjects’ performance on the DV.

This is likely to occur when more than one This is likely to occur when more than one treatment is administered to the same subjectstreatment is administered to the same subjects this factor affects the generalizability of this factor affects the generalizability of

findings to other treatments.findings to other treatments. The results of a multiple treatment study can The results of a multiple treatment study can

only be generalized to people who would only be generalized to people who would receive the same sequence and number of receive the same sequence and number of treatments.treatments.

The 4 factors that affect external validity can affect generalization in 1 of 4 ways:

Generalizability to other subjectsGeneralizability to other subjects generalizability to other settingsgeneralizability to other settings generalizability to other measurementsgeneralizability to other measurements generalizability to other treatmentsgeneralizability to other treatments

In sum…. Research designs are evaluated in terms of Research designs are evaluated in terms of

validity (internal and external) validity (internal and external)

Observations are evaluated in terms of Observations are evaluated in terms of reliabilityreliability a measure is a measure is validvalid to the extent that it to the extent that it

measures what it claims to measure, and measures what it claims to measure, and reliablereliable to the extent that it is consistent to the extent that it is consistent and repeatable.and repeatable.

Reliability

Refers to the precision of measurement and Refers to the precision of measurement and can be assessed by examining the can be assessed by examining the consistency or stability of a test or measure.consistency or stability of a test or measure.

Measurements of DVs are reliable to the Measurements of DVs are reliable to the extent that the same measurements would extent that the same measurements would be obtained if the study were repeated.be obtained if the study were repeated.

Reliability of measurements can be estimated in several ways Test-retest reliabilityTest-retest reliability split-half reliabilitysplit-half reliability alternate form reliabilityalternate form reliability observer reliabilityobserver reliability

Test-Retest Reliability

Assessed by obtaining the same Assessed by obtaining the same measurement twice and comparing the two measurement twice and comparing the two measurementsmeasurements

Split-Half Reliability

Assessed when measurements consist of a Assessed when measurements consist of a series of itemsseries of items

the items are split in half and the resulting the items are split in half and the resulting measurements are compared.measurements are compared.

Alternate Form Reliability

Assessed when alternate forms of Assessed when alternate forms of measurement are comparedmeasurement are compared

Observer reliability

Assessed when the judgement of two or Assessed when the judgement of two or more observers are compared.more observers are compared.

The design is evaluated in terms of validity and the observationsobservations are evaluated in terms of reliabilityreliability.