11

Research MethodsResearch Methods

TechniquesTechniques

StrengthsStrengths

ProblemsProblems

22

ExperimentsExperiments

Experiments involve manipulation and Experiments involve manipulation and control. They use:control. They use:

IVs and DVs.IVs and DVs. Experimental and control groupsExperimental and control groups Pretesting and posttestingPretesting and posttesting They seek to understand causation. The They seek to understand causation. The

IV is thought to lead to, affect, the DVIV is thought to lead to, affect, the DV

33

•Logic of experimentsLogic of experiments

(IV)(IV)

DV Experimental group DVDV Experimental group DV

DV Control group DVDV Control group DV

Ideally, random assignment of subjects to Ideally, random assignment of subjects to experimental and control groupsexperimental and control groups

44

Logic of experiments (continued)Logic of experiments (continued)

If random assignment (randomization) is If random assignment (randomization) is used, the groups are likely to be used, the groups are likely to be equivalent. The P is that the groups will equivalent. The P is that the groups will be the same, differences canceled out.be the same, differences canceled out.

Logic or rationale: If the control and Logic or rationale: If the control and experimental group are the same on the experimental group are the same on the DV at the start of the study, and the only DV at the start of the study, and the only difference is that the experimentaldifference is that the experimental

55

Logic (continued)Logic (continued)

group receives the IV and the control group receives the IV and the control group does not, thengroup does not, then

If there are differences between the If there are differences between the groups on the DV at the end of the groups on the DV at the end of the study, then they must be due to the IV.study, then they must be due to the IV.

66

Equivalent groupsEquivalent groups

Importance of equivalent groupsImportance of equivalent groups Randomization best, but not always Randomization best, but not always

feasiblefeasible Might match on important variables (sex Might match on important variables (sex

race, SES, intelligence, etc..)race, SES, intelligence, etc..) Problem with matching: might not match Problem with matching: might not match

on some important variable, and the on some important variable, and the groups will not be equivalentgroups will not be equivalent

77

Some designsSome designs

Classic experimentClassic experiment One group pre-post testOne group pre-post test Static-group comparisonsStatic-group comparisons

88

Threats to validity (extraneous)Threats to validity (extraneous)

Internal validity: variables which might Internal validity: variables which might affect the DV, aside from the IVaffect the DV, aside from the IV

History: events which occur during the History: events which occur during the course of the studycourse of the study

Maturation: changes that take place Maturation: changes that take place because of the aging processbecause of the aging process

Testing: changes that occur because of Testing: changes that occur because of pretestingpretesting

99

Threats (continued)Threats (continued)

Selection: nonequivalent groupsSelection: nonequivalent groups Regression: problematic when extreme Regression: problematic when extreme

groups are used. People with very low groups are used. People with very low or very high scores tend to score closer or very high scores tend to score closer to the mean on retesting. (holds true for to the mean on retesting. (holds true for other phenomenon, such as inheritance other phenomenon, such as inheritance to traits)to traits)

Mortality: dropping out of a studyMortality: dropping out of a study

1010

External validity (generalization)External validity (generalization)

One study shows that the IV affects the One study shows that the IV affects the DV. Will this be true in other settings DV. Will this be true in other settings with other subjects?with other subjects?

Interaction of testing and IVInteraction of testing and IV Experimenter effects, reactive Experimenter effects, reactive

arrangementsarrangements Hawthorne studyHawthorne study

1111

Advantages of experimentsAdvantages of experiments

We can more directly observe the We can more directly observe the effects of one variable on the other, effects of one variable on the other, holding other variables constantholding other variables constant

Sometimes can be done with little time Sometimes can be done with little time and moneyand money

Experiments can be more easily Experiments can be more easily replicated than other methodsreplicated than other methods

1212

Disadvantages of experimentsDisadvantages of experiments

ArtificialityArtificiality Example: TV and aggressionExample: TV and aggression Example: Zimbardo studyExample: Zimbardo study

1313

SurveysSurveys

Ask people what they think, feel, didAsk people what they think, feel, did Initial steps in surveysInitial steps in surveys Selection of population and sampleSelection of population and sample Development of the questions into a Development of the questions into a

questionnairequestionnaire Pretesting and reviewPretesting and review

1414

Survey methodsSurvey methods

Group administeredGroup administered MailedMailed Face to face interviewsFace to face interviews Telephone interviewsTelephone interviews Each has its advantages and Each has its advantages and

disadvantagesdisadvantages

1515

Group administeredGroup administered

Assemble subjects together, have them Assemble subjects together, have them fill out the questions. A proctor is fill out the questions. A proctor is available to answer questionsavailable to answer questions

Advantages: economical, quick, Advantages: economical, quick, problems can be handled immediatelyproblems can be handled immediately

Problem: it may not be possible to Problem: it may not be possible to assemble subjects. Works best with assemble subjects. Works best with groups who will be together anywaygroups who will be together anyway

1616

Mailed surveysMailed surveys

Mail out: (1) questionnaire; (2) cover Mail out: (1) questionnaire; (2) cover letter explaining the purpose of the letter explaining the purpose of the study, who is doing it, and what the study, who is doing it, and what the results will be used for; and (3) a self-results will be used for; and (3) a self-addressed “stamped” envelope.addressed “stamped” envelope.

bulk ratebulk rate on the return envelope: stamps, on the return envelope: stamps,

metering, business reply permitsmetering, business reply permits

1717

Mailed (continued)Mailed (continued)

business reply permits : more business reply permits : more expensive to purchase, but you only expensive to purchase, but you only pay for those surveys returned.pay for those surveys returned.

Follow-ups--reminder postcards vs. new Follow-ups--reminder postcards vs. new questionnaire, cover letter and envelopequestionnaire, cover letter and envelope

1818

Advantages of mailed surveysAdvantages of mailed surveys

Relatively inexpensiveRelatively inexpensive Fairly fast resultsFairly fast results Geographic accessibility--can reach Geographic accessibility--can reach

almost anyonealmost anyone

1919

DisadvantagesDisadvantages

Low response rates, with the potential Low response rates, with the potential for biased resultsfor biased results

No control over the subjectsNo control over the subjects No control over misinterpretations of the No control over misinterpretations of the

questionsquestions Will subjects be truthful?Will subjects be truthful? More impersonal than other methodsMore impersonal than other methods

2020

Face to face interviewsFace to face interviews

Training of interviewersTraining of interviewers IdentificationIdentification How to dress (depends on subjects)How to dress (depends on subjects) Approaching people (appointment vs. Approaching people (appointment vs.

showing up)showing up) How to administer questions and record How to administer questions and record

responsesresponses

2121

Face to face (continued)Face to face (continued)

Practice sessionsPractice sessions Anticipating as many situations as Anticipating as many situations as

possible and handling problemspossible and handling problems Need to monitor interviewersNeed to monitor interviewers

2222

Advantages of face to faceAdvantages of face to face

High response ratesHigh response rates Control over subjectsControl over subjects Can “probe” and ask more complicated Can “probe” and ask more complicated

questionsquestions Deals better with less educated or more Deals better with less educated or more

illiterate populationsilliterate populations Better for establishing rapport and getting Better for establishing rapport and getting

more informationmore information

2323

Disadvantages of face to faceDisadvantages of face to face

Time-consumingTime-consuming ExpensiveExpensive Interviewer biasInterviewer bias Some people may lie to face to face Some people may lie to face to face

interviewersinterviewers

2424

Telephone interviewsTelephone interviews

Same procedures as face to face, but Same procedures as face to face, but over the telephoneover the telephone

Use of computer systemsUse of computer systems Random digit dialingRandom digit dialing Coding data as questions are answeredCoding data as questions are answered

2525

Advantages of telephoneAdvantages of telephone

FastFast Less expensive than face to faceLess expensive than face to face Can still ask more complicated Can still ask more complicated

questions and probequestions and probe

2626

DisadvantagesDisadvantages

More impersonalMore impersonal Can terminate the interview more easilyCan terminate the interview more easily response rate not as high as face to response rate not as high as face to

face, but higher than mailedface, but higher than mailed Cannot be very longCannot be very long

2727

Observational ResearchObservational Research

Problem with surveys: what people say Problem with surveys: what people say and what they actually do are not and what they actually do are not always the same. always the same.

Additionally, there may be important Additionally, there may be important information that no one thought to askinformation that no one thought to ask

Therefore it is sometimes useful to Therefore it is sometimes useful to observe behavior as it occurs, not ask observe behavior as it occurs, not ask about it or do experimentsabout it or do experiments

2828

Distinctions in observationsDistinctions in observations

Laboratory vs. fieldLaboratory vs. field One subject vs. a groupOne subject vs. a group Unstructured vs. structuredUnstructured vs. structured Unstructured: observe a variety of Unstructured: observe a variety of

aspects of environment and behavioraspects of environment and behavior Structured: interested in a particular Structured: interested in a particular

behaviorbehavior

2929

Distinctions (continued)Distinctions (continued)

Unstructured: Margaret Meade’s Unstructured: Margaret Meade’s studies, prison subculture studiesstudies, prison subculture studies

Structured: police brutality, police Structured: police brutality, police discretion at the time of arrestdiscretion at the time of arrest

Participant vs. non-participant observerParticipant vs. non-participant observer Participant: joins in the activities in the Participant: joins in the activities in the

environment. May be a genuine member environment. May be a genuine member (Becker as a jazz musician)(Becker as a jazz musician)

3030

Distinctions (continued)Distinctions (continued)

Might be pretend to be a member of the Might be pretend to be a member of the group. Problem: is it ethical to group. Problem: is it ethical to deceive? (Rosenhan study, “end of the deceive? (Rosenhan study, “end of the world” study)world” study)

With participant, the observer might With participant, the observer might become too involved to be objective, become too involved to be objective, and might influence the group’s and might influence the group’s behaviorbehavior

3131

Distinctions (continued)Distinctions (continued)

Nonparticipant researcherNonparticipant researcher Might conceal him/herself. Problem: Might conceal him/herself. Problem:

people’s right to privacy (Not a problem people’s right to privacy (Not a problem if observing in a public place, where if observing in a public place, where people do not have an expectation of people do not have an expectation of privacy)privacy)

Might identify self as researcher. Might identify self as researcher.

3232

DistinctionsDistinctions

Problem: Might be rejected. Also, Problem: Might be rejected. Also, people might act differently if they know people might act differently if they know they are being observed.they are being observed.

The pros and cons must be considered The pros and cons must be considered in such observational studiesin such observational studies

3333

What do observers study?What do observers study?

Practices (behaviors)Practices (behaviors) Episodes and encountersEpisodes and encounters RolesRoles RelationshipsRelationships Groups and organizationsGroups and organizations

3434

Steps in observational researchSteps in observational research

LiteratureLiterature InformantsInformants ““Getting in” and rapport, explanation of Getting in” and rapport, explanation of

researchresearch Sampling: often nonprobability--Sampling: often nonprobability--

convenience, quota, or snowball.convenience, quota, or snowball. ObservationsObservations

3535

Steps in observation (continued)Steps in observation (continued)

Unstructured: field notesUnstructured: field notes Structured: record events in categories Structured: record events in categories

as they occuras they occur Unstructured interviews--probe and try Unstructured interviews--probe and try

to determine what is going onto determine what is going on Categorizing observationsCategorizing observations Examples: history of group, biographiesExamples: history of group, biographies

3636

Steps in observation (continued)Steps in observation (continued)

religion, rule-breaking, roles.religion, rule-breaking, roles. Finding patterns, deviations in the Finding patterns, deviations in the

patternspatterns Writing up resultsWriting up results More difficult than other methodsMore difficult than other methods

3737

Advantages of observationAdvantages of observation

Depth of understanding gainedDepth of understanding gained Flexibility of the methodFlexibility of the method Rich source of hypotheses for other Rich source of hypotheses for other

types of studiestypes of studies

3838

Disadvantages of observationDisadvantages of observation

Might misinterpretMight misinterpret Difficulties of interpretationDifficulties of interpretation Small number of subjectsSmall number of subjects Sampling problems--will the Sampling problems--will the

observations be representativeobservations be representative Time-consumingTime-consuming ExpensiveExpensive

3939

Unobtrusive researchUnobtrusive research

Looking at traces left behindLooking at traces left behind Example: looking at well-worn floors in Example: looking at well-worn floors in

museums to determine which exhibits museums to determine which exhibits are most popularare most popular

Our only way of being able to study Our only way of being able to study prehistoric manprehistoric man

Problem: dangers of misinterpretationProblem: dangers of misinterpretation

4040

Record ResearchRecord Research

With the advent of the written word, we With the advent of the written word, we can study behavior. Our only method can study behavior. Our only method for studying the past.for studying the past.

For example, has America always had a For example, has America always had a crime problem, particularly violent crime problem, particularly violent crime, or has this always been part of crime, or has this always been part of our culture?our culture?

Have child rearing practices changed?Have child rearing practices changed?

4141

Content analysisContent analysis

Use of documents to study a behavior Use of documents to study a behavior such as crime: books, magazines, such as crime: books, magazines, songs, newspapers, letters, diariessongs, newspapers, letters, diaries

Example: fear of crime in Switzerland Example: fear of crime in Switzerland studied by examining newspapers and studied by examining newspapers and Parliamentary debates for concern Parliamentary debates for concern about crime (In contrast to the U.S., the about crime (In contrast to the U.S., the topic was not raised often)topic was not raised often)

4242

Content analysis (continued)Content analysis (continued)

Themes put into categoriesThemes put into categories Need to distinguish manifest (actual) Need to distinguish manifest (actual)

and latent (hidden meanings) contentand latent (hidden meanings) content Use of computers to search for words in Use of computers to search for words in

documents (advantages and documents (advantages and disadvantages)disadvantages)

4343

Record (archival) researchRecord (archival) research

Use of existing statistics and recordsUse of existing statistics and records government documentsgovernment documents computer tapes of agenciescomputer tapes of agencies agency filesagency files Examples: Do judges sentence females Examples: Do judges sentence females

differently than males? What percentage differently than males? What percentage of cases are plea bargained? etc..of cases are plea bargained? etc..

4444

Advantages of study of recordsAdvantages of study of records

InexpensiveInexpensive Possible to study processes occurring Possible to study processes occurring

over long periods of timeover long periods of time No effect on those being studied; records No effect on those being studied; records

are being kept anywayare being kept anyway Can be used for historical and Can be used for historical and

comparative usescomparative uses Large numbers of subjectsLarge numbers of subjects

4545

Disadvantages of record Disadvantages of record researchresearch

Records may have gaps, be incompleteRecords may have gaps, be incomplete Ecological fallacy: patterns at a group Ecological fallacy: patterns at a group

level may be different than those at the level may be different than those at the individual. Example: Protestant individual. Example: Protestant countries have higher suicide rates than countries have higher suicide rates than Catholic countries. Are Protestants Catholic countries. Are Protestants committing suicides--could be Catholic committing suicides--could be Catholic minorityminority

4646

Disadvantages--record researchDisadvantages--record research

Problems with statisticsProblems with statistics Official statistics--unreported crimeOfficial statistics--unreported crime Crimes depend on laws, which varyCrimes depend on laws, which vary Numbers depend on discretion of officials Numbers depend on discretion of officials

and their norms, at arrest, plea and their norms, at arrest, plea bargaining and convictions, sentences, bargaining and convictions, sentences, probation/parole revocationprobation/parole revocation

Necessary to look behind the numbersNecessary to look behind the numbers

4747

Evaluation ResearchEvaluation Research

Determining the effects of programs, Determining the effects of programs, agencies, policies and lawsagencies, policies and laws

Four componentsFour components Needs assessmentNeeds assessment MonitoringMonitoring OutcomesOutcomes Cost/benefit and cost/effectivenessCost/benefit and cost/effectiveness

4848

Needs assessmentNeeds assessment

Is the service/change/intervention Is the service/change/intervention needed?needed?

Who is the target population? i.e.., who Who is the target population? i.e.., who do we want to reach the most? What do we want to reach the most? What are their characteristics?are their characteristics?

What problems can be anticipated as What problems can be anticipated as the change is made? i.e.., halfway the change is made? i.e.., halfway houses expect neighborhood resistancehouses expect neighborhood resistance

4949

Needs (continued)Needs (continued)

Can problems be addressed Can problems be addressed successfully?successfully?

Ways of assessing: records, surveys, Ways of assessing: records, surveys, key informants and gatekeeperskey informants and gatekeepers

Without needs assessment, Without needs assessment, inappropriate interventions might be inappropriate interventions might be mademade

5050

MonitoringMonitoring

Looks at the processes taking place as Looks at the processes taking place as the intervention is madethe intervention is made

Are the services being provided? In the Are the services being provided? In the case of a law, is it being utilized?case of a law, is it being utilized?

Is the intervention reaching the target Is the intervention reaching the target population? i.e., if your intervention is population? i.e., if your intervention is school based and your targets are school based and your targets are truants, it will not worktruants, it will not work

5151

Monitoring (continued)Monitoring (continued)

Are there unanticipated side effects?Are there unanticipated side effects? Is the intervention working/happening?Is the intervention working/happening? Assessing: records, interviews with Assessing: records, interviews with

people carrying out the interventions, people carrying out the interventions, interviews with people affected by the interviews with people affected by the interventionintervention

5252

Outcomes researchOutcomes research

What are the goals and outcomes What are the goals and outcomes expected? Usually multiple goalsexpected? Usually multiple goals

Is the intervention achieving those Is the intervention achieving those outcomes?outcomes?

Assessing: experiments or quasi-Assessing: experiments or quasi-experiments (nonequivalent control experiments (nonequivalent control groups, before-after), time series groups, before-after), time series designsdesigns

5353

Cost benefit & cost effectivenessCost benefit & cost effectiveness

Cost-benefit: monetary costs of the Cost-benefit: monetary costs of the intervention are compared to the intervention are compared to the monetary savings as the result of the monetary savings as the result of the programprogram

Cost effectiveness: costs are compared Cost effectiveness: costs are compared to the outcomesto the outcomes

Example: bail programExample: bail program

5454

Cost benefit/cost effectivenessCost benefit/cost effectiveness

Example: Halfway house programExample: Halfway house program Costs must be estimatedCosts must be estimated Effectiveness a more difficult issueEffectiveness a more difficult issue How successful does a program have to How successful does a program have to

be to be “worth it?” a 5% reduction in be to be “worth it?” a 5% reduction in recidivism? 10% ? etc.. compared to recidivism? 10% ? etc.. compared to what costs?what costs?

5555

Strengths of evaluationStrengths of evaluation

Determining what works and what does Determining what works and what does not, rather than just trying intervention not, rather than just trying intervention after interventionafter intervention

Determining what the unanticipated Determining what the unanticipated effects are (ex: net-widening)effects are (ex: net-widening)

Determining why programs do not work, Determining why programs do not work, which can be valuable (example: laws which can be valuable (example: laws and prosecutors)and prosecutors)

5656

Problems in evaluationProblems in evaluation

Problems of uncontrolled experimentsProblems of uncontrolled experiments Measurement problemsMeasurement problems Lack of control over agenciesLack of control over agencies Results ignored, or might be unpopular Results ignored, or might be unpopular

(for example, pornography studies)(for example, pornography studies)