CHAPTER 2 Personality Assessment, Measurement, and Research Design © 2015 M. Guthrie Yarwood.

© 2015 M. Guthrie Yarwood

CHAPTER 2

Personality Assessment, Measurement, andResearch Design


Outline

I. 4 Sources of personality data

II. Reliability / Validity in Personality

III. A little more on observer reports


I. Sources of Personality Data

Self-Report Data (S-Data)Observer-Report Data (O-Data) Test-Data (T-Data) Life-Outcome Data (L-Data)


Self-Report Data (S-Data)

Information provided by a person, such as through a survey or interview

Limitations of S-data?


O-Data

Information provided by someone else about another person Professional personality assessors People who actually know the target person

Naturalistic vs. Artificial Observation

Limitations?


Test-Data (T-Data)

Information provided by standardized tests or testing situations

Situation designed to elicit behaviors that serve as indicators of personality


Test-Data Creativity Example

What are unusual uses for common objects – bricks, knives, newspapers?

Answers to hypothetical events What would happen if people went blind? What would happen is people shrank to 12 inches tall?

(Paul Silvia’s work)


Test-Data: Other Examples

Mechanical recording devices, e.g., “Actometer” used to assess children’s activity

Physiological data

Projective Tests

Ex: Fairy Tale Test

Limitations?

http://www.fairytaletest.com/


Life-Outcome Data (L-Data)

Information that can be gleaned from events, activities, and outcomes in a person’s life that is available for public scrutiny—e.g., marriage, speeding tickets

Can serve as important source of “real life” information about personality

Ex: implicit egotism: people gravitate toward people, places, things that resemble the self

(Pelham and colleagues’ work)


Issues in Personality Assessment

Links among different data sourcesFallibility of personality measurement

All sources of data have limitations Results that replicate through “triangulation” are

most powerful


You are a personality psychologist and would like to measure the personality trait risk-taking (i.e., sensation seeking).

How could you measure risk-taking using each of the four data sources?S-DataO-DataT-DataL-Data


Evaluation of Personality Measures How do we know whether our scale is a “good scale?”

Types of ErrorsReliability ValidityThreats to Reliability and Validity


Extraneous vs. Confounding Variables

Extraneous variables are variables that may compete with the independent variable in explaining the outcome of a study.

Confounding variable: an extraneous variable that does indeed influence the dependent variable.

A confounding variable systematically varies or influences the independent variable and also influences the dependent variable.


Random vs. Constant Errors

Random Errors (unsystematic): extraneous variables whose average influence on the outcome is the same in both (or all) conditions (Aronson et al., 1990) Affects reliability AND validity

Constant (systematic) Errors: influences all the scores in one condition in the same direction and has no effect or a different effect on the other condition. Affects ONLY validity Confounding variables


Clarification Check!

IV: Type of show Condition 1: Extraverts watch comedy

show Condition 2: Extraverts watch neutral

show

DV: Self-reported emotions Extraverts report more positive emotions

in Comedy Condition than in Neutral Condition


Clarification Check! – Random Errors

Extraverts in conditions 1 and 2 are placed in very hot rooms. The hot temperature lowers their self-reported levels of positive emotions.

But, we still find that extraverts report more positive emotions when watching the comedy show than the neutral show.

Accurate: We find differences in self-reported positive emotions (on 5-point scale): Comedy Cond. = 4.8, Neutral Cond. = 2.5

Random Errors: In presence of hot temperature, we find differences in self-reported positive emotions Comedy Cond. = 3.0, Neutral Cond. = 1.5


Clarification Check! – Random Errors

Extraverts in conditions 1 and 2 are placed in very hot rooms. The hot temperature lowers their self-reported levels of positive emotions.

But, we still find that extraverts report more positive emotions when watching the comedy show than the neutral show.

Accurate: We find differences in self-reported positive emotions (on 5-point scale): Comedy Cond. 1 = 4.8, Neutral Cond. 2 = 2.5

Random Errors: In presence of hot temperature, we find differences in self-reported positive emotions Comedy Cond = 3.0, Neutral Cond.2 = 1.5

Hot temperature lowers the scores in both

conditions!!


Clarification Check! – Constant Errors

Extraverts in Comedy Cond. are in a 70◦ room. Extraverts in Neutral Cond. are placed in a 78◦ room. In Neutral Cond. only, the hot temperature lowers self-reported levels of positive emotions.

We find that the comedy show results in more positive emotions than the neutral show.

Accurate: We do not find differences in self-reported positive emotions Comedy Cond. = 4.8, Neutral Cond. = 4.6

Constant Error: We find differences in positive emotions due to the hot temperature, not due to the manipulation! Comedy Cond. = 4.8, Neutral Cond. = 2.5


Clarification Check! – Constant Errors

Extraverts in Comedy Cond. are in a 70◦ room. Extraverts in Neutral Cond. are placed in a 78◦ room. In Neutral Cond. only, the hot temperature lowers self-reported levels of positive emotions.

We find that the comedy show results in more positive emotions than the neutral show.

Accurate: We do not find differences in self-reported positive emotions Comedy Cond. = 4.8, Neutral Cond. = 4.6

Constant Error: We find differences in positive emotions due to the hot temperature, not due to the manipulation! Comedy Cond. = 4.8, Neutral Cond. = 2.5

Hot temperature lowers the scores in Condition 2, but not Condition 1!!


Reliability

The ability of a test to measure an attribute consistently Does this extraversion scale measure the true

level of each participant’s extraversion (over time)?

Are coders following the same method?

To achieve reliability we want to reduce random error.


Threats to Reliability (random error)

Participant Characteristics: fatigue, motivation, boredom

Testing situation: time of day, room temperature

Testing Instrument: instructions, rating scale, items, reading level.

Experimenter Characteristics and Errors: interactions with participants; incorrect observations

of participants; incorrect coding of behavior


Estimating Reliability

Test-Retest Coefficient

Parallel-Forms Coefficient

Internal Consistency Coefficient

Interrater (interobserver) Reliability


Validity

Validity: The degree to which a test or measurement accurately measures or reflects what it claims to measure.

Internal Validity: Did the experimental treatments make a difference in this specific experimental instance?

External Validity: Generalizability; To what populations, settings, treatment variables, and measurement variables can the effect be generalized? **Never completely answerable


Internal Validity

Degree to which test measures what it claims to measure

5 types of internal validity Face validity Predictive or criterion validity Convergent validity Discriminant validity Construct validity


Threats to Internal Validity (errors)

Affected by random and constant errors

Random (unsystematic) Errors: same errors that affect reliability

Constant Errors (systematic): errors that affect measurement of variable; does not affect reliability


Knowledge Check!You are conducting a study on the personality traits associated with the frequency of exercising.

For your study, which of the following poses a threat to validity, but not reliability? A. All the participants are bored.B. The construction outside the laboratory

window is very loud.C. You recruited participants from the Rec Hall.D. The experimenter who greets all

participants is very rude.


Knowledge Check!You are conducting a study on the personality traits associated with the frequency of exercising.

For your study, which of the following poses a threat to validity, but not reliability? A. All the participants are bored.B. The construction outside the laboratory

window is very loud.C. You recruited participants from the Rec

Hall.D. The experimenter who greets all participants

is very rude.


Threats to Internal Validity – MRS SMITH

MaturationRegression to the MeanSelection of SubjectsSelection by Maturation InteractionMortalityInstrumentationTestingHistory


Regression to the Mean Example

In an experiment involving reading instruction, subjects grouped because of poor pre-test reading scores show considerably greater gain than do the groups who scored average and high on the pre-test.




Poor Average / Mean

HighPre-test




Poor Average / Mean

HighPre-test

After reading Instruction


Selection by Maturation Interaction

Group Pre-test Post-Test (after head start)

Head Start Intervention

Middle-class children 65/100

Disadvantaged children

65/100

Control Group – No Intervention

Middle-class children 65/100

Disadvantaged children

65/100



Group Pre-test

Post-Test: 6 months

Post-Test: 12 months



65/100

70/100 75/100 80/100


65/100

65/100 65/100 65/100



Group Pre-test

Post-Test: 6 months




65/100

70/100 75/100 80/100


65/100

65/100 65/100 65/100

Ss in Intervention are middle class, while Ss in control group are disadvantaged. Over time, Intervention Ss show

improvement of post-test due to better health care, greater parental support, greater access to resources, etc.


Identify the Threat to Validity!

In a short experiment designed to investigate the effect of computer-based instruction, Ss missed some instruction because of a power failure at school.

A. HistoryB. Mortality C. TestingD. Instrumentation


Identify the Threat to Validity!

In a health experiment designed to determine the effect of various exercises, those Ss who find the exercise most difficult stop participating.

A. Selection of SubjectsB. Mortality C. TestingD. Maturation


Estimating Internal Validity

Content ValidityCriterion-Related Validity

Concurrent Validity Predictive Validity

Construct Validity Convergent Validity Discriminant Validity


Content Validity

Definition: whether the content of a test elicits a range of responses that are representative of the entire domain or universe of skills, understandings, and other behaviors a test is designed to measure.

To assess: compare tests’ content with an outline of specifications concerning subject matter to be covered in test.


Openness to Experience

?????

?????

?????

?????



Behaviors?

Perceptions?

Thoughts/Cognitions?

Feelings/Emotions?


Criterion-Related Validity

Definition: procedures in which the test scores of a group of people are compared with ratings, classifications, or other measures of performance.


Concurrent Validity – A type of criterion-related validity

Concurrent Validity: when a test is administered to people in various categories, to determine whether test scores of people in 1 category are significantly different from people in other categories. Clinical vs. non-clinical group Different socioeconomic levels


Group A shows more openness to experience

Group B shows less openness to experience

Concurrent Validity


High IQ group shows more openness

to experience

Low IQ group shows less openness to

experience

Concurrent Validity


Predictive Validity – A type of criterion-related validity

Predictive Validity: how accurately test scores predict criterion scores.

Indicated by correlation between test score (the predictor) and a criterion of future performance (what the test predicts)


Predictive Validity


(Predictor)

?????

(Criterion)


Predictive Validity


College Major

Well-being, Psychological Adjustment



Construct-Related Validity

Definition: extent to which scale measures a particular construct or psychological concept

To assess: need to determine whether an assessment instrument that presumably measures a certain personality variable is actually doing so


Convergent Validity – A Type of Construct-Related Validity

Convergent Validity: the measure has high correlations with other measures or methods of measuring the same construct


Convergent Validity

Our Openness to Experience

Self-Report Scale

Same Construct, Same Measurement


Self-Report Scale

Same Construct,Different Measurement


Convergent Validity

Our Openness to Experience Self-

Report Scale

Costa & McCrae’s Self-Report OE

Dimension

Observer-Report OE Dimension

Our Openness to Experience Self-

Report Scale


Discriminant Validity – A Type of Construct-Related Validity

Discriminant Validity: the measure has low correlations with measures of different constructs


Discriminant Validity


Self-Report Scale

Different Construct, Same Measurement

Different Construct Different Measurement


Self-Report Scale

OR




Self-Report Scale

Self-Report Sensation Seeking

Scale

Clinical Diagnosis of Schizotypal

PersonalityDisorder


Self-Report Scale




Self-Report Scale

Self-Report Sensation Seeking

Scale

Clinical Diagnosis of Schizotypal

PersonalityDisorder


Self-Report Scale

These correlations should be significant,

but lower than correlations for

convergent validity.


To assess convergent/discriminant validity

The same construct using the same method (c)The same construct using different methods

(c)Different constructs using the same method

(d)Different constructs using different methods

(d)

Note: c = convergent; d=discriminant


Reliability & Validity

A test can be reliable, but not valid

In other words, to be valid a test must first be reliable.

NO VALDITY WITHOUT RELIABILITY!


Reliability Internal Validity

Test-Retest Coefficient

Parallel-Forms Coefficient

Internal Consistency Coefficient

Interrater (interobserver) Reliability

Content ValidityCriterion-Related

Validity Concurrent Validity Predictive Validity

Construct Validity Convergent Validity Discriminant Validity

Reliability and Validity: Summary


A LITTLE MORE ON OBSERVER REPORTS (O-DATA)


•Self-report

•Observer report 1

Correlation 1



Correlation 2

•Self-report

•Several different observers

Correlation 3, 4, 5…


Aggregating Scores

Averaging the self-report and the observer report/s provides a clearer picture of personality than the self or observer report alone (Kolar et al., 1996)


Let’s discuss the purpose of observer reports

What is the purpose of an observer report?Compared to a self-report measure, what

results would you expect from an observer report?

What would you think if you did not obtain these results?

Can you think of any threats to validity that observer reports may pose?


Aggregating Scores

A positive correlation between an observer report and a self-report, would provide evidence for which type of validity?

A. ConvergentB. PredictiveC. ConcurrentD. Discriminant


Recall Reflection Post and TIPI!

Disagree

Strongly

(1)

Disagree

Moderately

(2)

Disagree a

little

(3)

Neither

Agree nor

Disagree

(4)

Agree a

little

(5)

Agree

Moderately

(6)

Agree

Strongly

(7)

1 Extraverted,

enthusiastic

2 Reserved, quiet r

3 Critical,

quarrelsomer

4 Sympathetic,

warm

5 Dependable,

self-disciplined

6 Disorganized,

carelessr

7 Anxious, easily

upset r

8 Calm,

emotionally

stable

9 Open to new

experiences,

complex

10 Conventional,

uncreativer


Self Scores!

Extra

vers

ion

Agree

ablene

ss

Consc

ient

ious

ness

Emot

iona

lly S

tabl

e

Openn

ess to

Exp

erienc

e1

2

3

4

5

6

7

3.96

5.135.54

4.46

5.71

TIPI Dimensions


Observer Scores!

Extra

vers

ion

Agree

ablene

ss

Consc

ient

ious

ness

Emot

iona

lly S

tabl

e

Openn

ess to

Exp

erienc

e1

2

3

4

5

6

7

4.294.92 5.13

4.38

5.25

TIPI Scores


Based on the following graphs, which dimension do you think shows the weakest correlation between self and observer scores?

A. ExtraversionB. AgreeablenessC. ConscientiousnessD. Emotional StabilityE. Openness to Experience


1 2 3 4 5 6 71

2

3

4

5

6

7


SELF SCORE

OB

SE

RV

ER

SC

OR

E

1 2 3 4 5 6 71234567

Emotional StabilityLOW NEUROTICISM

SELF SCORE

OB

SE

RV

ER

SC

OR

E

1 2 3 4 5 6 71

2

3

4

5

6

7

Conscientiousness

SELF SCORE

OB

SE

RV

ER

SC

OR

E

1 2 3 4 5 6 71

3

5

7

AGREEABLENEESS

SELF SCOREOB

SE

RV

ER

SC

OR

E

1 2 3 4 5 6 71234567

EXTRAVERSION

SELF SCORE

OB

SE

RV

ER

SC

OR

E


Based on the following graphs, which dimension do you think shows the weakest correlation between self and observer scores?

A. ExtraversionB. AgreeablenessC. ConscientiousnessD. Emotional StabilityE. Openness to Experience


LET’S CORRELATE THE SCORES!

TIPI DIMENSION TIPI TRAIT r

1EXTRAVERSION

Extraverted, enthusiastic

.742 Reserved, quiet r

3AGREEABLENESS

Critical, quarrelsomer

.484 Sympathetic, warm

5

CONSCIENTIOUSNESSDependable, self-disciplined

.526 Disorganized, carelessr

7EMOTIONAL STABILITY

(LOW NEUROTICISM)

Anxious, easily upset r

.078 Calm, emotionally stable

9 OPENNESS TO

EXPERIENCE

Open to new experiences,

complex .3910 Conventional, uncreativer

Note. Red = Self; Yellow = Observer


1 2 3 4 5 6 7 8 9 10 11 121

2

3

4

5

6

7

EXTRAVERSION

STUDENT

SC

OR

E

1 2 3 4 5 6 7 8 9 10 11 121

2

3

4

5

6

7

AGREEABLENESS

STUDENT

SC

OR

E

1 2 3 4 5 6 7 8 9 10 11 121

2

3

4

5

6

7

CONSCIENTIOUSNESS

STUDENT

SC

OR

E

1 2 3 4 5 6 7 8 9 10 11 121

2

3

4

5

6

7

EMOTIONAL STABILITY(LOW NEUROTICISM)

STUDENT

SC

OR

E

1 2 3 4 5 6 7 8 9 10 11 121

2

3

4

5

6

7

OPENNESS TO EXPERIENCE

STUDENT

SC

OR

E


Summary and Evaluation

Decisions about data source and research design depend on (1) the purpose of study and (2) threats to validity/reliability

There is no perfect data sourceThere is no perfect research designAssessing threats to reliability and validity will

assist in selecting a data source and research design.

Observer reports improve validity and reduce social desirability concerns


Reminder – Paper Topic!!

Find a group member (can do on ANGEL!)Select a topic!Submit Paper topic to drop-box

Due: Friday, January 30th at 9 AM ETPaper Topic Outline available on ANGEL.Guidance: Meet with Michelle or Celina

Questions???


Chapter 3 (Dispositional Domain) Survey

HEXACO Follow link to access survey Score your survey according to the instructions on

ANGEL

Big Five Model Access through Chapter 3 Survey on course website

CHAPTER 2 Personality Assessment, Measurement, and Research Design © 2015 M. Guthrie Yarwood.

Documents

Transcript of CHAPTER 2 Personality Assessment, Measurement, and Research Design © 2015 M. Guthrie Yarwood.