1

Work in the 21st CenturyChapter 2

Methods and Statistics

in I-O Psychology

2

Module 2.1: Science

• What is science?– Approach that involves the understanding, prediction,

and control of some phenomenon of interest• Science has common methods• Science is a logical approach to investigation

– Based on a theory, hypothesis, or basic interest

• Science depends on data– Gathered in a laboratory or the field

3

Common Methods (cont'd)

• Research must be communicable, open, & public

– Research published in journals, reports, or books

1) Methods of data collection described

2) Data reported

3) Analyses displayed for examination

4) Conclusions presented

4

Common Methods (cont'd)

• Scientists set out to disprove theories or hypotheses– Goal: Eliminate all plausible explanations

except one

• Scientists are objective– Expectation that researchers will be objective &

not influenced by biases or prejudices

5

Role of Science in Society

• Expert witnesses in a lawsuit– Permitted to voice opinions about

organizational practices

– Often a role assumed by I-O psychologists

6

Module 2.1 (cont'd)

• Why do I-O psychologists engage in research?– Better equip HR professionals in making

decisions in organizations

– Provide an aspect of predictability to HR decisions

7

Module 2.2: Common Research Designs in I-O Psychology

Table 2.1 Common Research Designs in I-O Psychology

8

Research Designs (cont'd)

– Experimental• Random assignment of participants to conditions• Conducted in a laboratory or the workplace

– Non-experimental• Does not include manipulation or assignment to different conditions

– 2 common designs:

• Observational design: Observes and records behavior

• Survey/Questionnaire design (most common)

– Quasi-experimental• Non-random assignment of participants to conditions

9

Methods of Data Collection

• Quantitative methods– Rely on tests, rating

scales, questionnaires, & physiological measures

– Yield results in terms of numbers

C. Borland/PhotoLink/Getty Images

10

Methods of Data Collection: Qualitative & Quantitative Research

• Qualitative methods– Include procedures like observation, interview, case

study, & analysis of written documents– Generally produce flow diagrams & narrative

descriptions of events/processes

• Quantitative methods– Rely on tests, rating scales, and physiological

measures– Yield numerical results

11

Quantitative & Qualitative Research (cont’d)

• Not mutually exclusive

• Triangulation– Examining converging information from

different sources (qualitative and quantitative research).

12

Generalizability in Research

Generalizability:

• Application of results from one study or sample to other participants or situations

• The more areas a study includes, the greater its generalizability

• Every time a compromise is made, the generalizability of results is reduced

13

Sampling Domains for I-O Research

Figure 2.1: Sampling Domainsfor I-O Research

14

Control in Research

• Experimental control– Eliminates influences that could make results

less reliable or harder to interpret

• Statistical control– Statistical techniques used to control for the

influence of certain variables

15

Ethical Behavior inI-O Psychology

• Ethical standards of the APA

• SIOP book of 61 cases (Lowman, 1998)– Cases illustrate ethical issues that are likely to

arise in I-O psychology

• Joel Lefkowitz (2003) published a recent book on values and ethics in I-O psychology

16

Table 2.2 Potential Roles Available to the I-O Psychologist and Other HR Managers with Respect to Ethical Problems

17

Module 3: Data Analysis

• Descriptive statistics– Summarize, organize, describe sample of data

Frequency Distribution:– Horizontal axis = Scores running low to high– Vertical axis = Indicates frequency of

occurrence

18

Describing a Score Distribution

• Measures of central tendency

• Mean • Mode• Median

Ryan McVay/Getty Images

19

Describing Score Distribution (cont'd)

• Variability– Standard deviation

• Lopsidedness or skew– Mean is affected by high or low

scores, median is not– Mean pulls in direction of skew

Ryan McVay/Getty Images

20

Descriptive Statistics:Two Score Distributions (N = 30)

Figure 2.2 Two Score Distribution (N = 30)

21

Two Score Distributions (N = 10)

Figure 2.3. Two Score Distributions (N = 10)

22

Inferential Statistics

• Aid in testing hypotheses & making inferences from sample data to a larger sample/population

• Include t-test, F-test, chi-square test

23

Statistical Significance

• Defined in terms of a probability statement

• Threshold for significance is often set at .05 or lower

• Significance refers only to confidence that result is NOT due to chance, not strength of an association or importance of results.

24

Statistical Power

• Likelihood of finding statistically significant difference when true difference exists

• The smaller the sample size, the lower the power to detect a true or real difference

25

Concept of Correlation

Positive Linear Correlation

Figure 2.4Correlation betweenTest Scores andTraining Grades

26

Concept of Correlation (cont'd)

• Scatterplot– Displays correlational relationship between 2

variables

• Regression– Straight line that best “fits” the scatterplot and

describes the relationship between the variables in the graph

27

Correlation Coefficient

• Statistic or measure of association

• Reflects magnitude (numerical value) & direction (+ or –) of relationship between 2 variables

• Ranges from 0.00 and 1.00

28

Correlation Coefficient

• Positive correlation → As one variable increases, other variable also increases & vice versa

• Negative correlation → As one variable increases, other variable decreases & vice versa

29

Scatterplots of Various Degrees of Correlation

Figure 2.6. Scatterplots of Various Degrees of Correlation

30

Curvilinear Relationship

• If correlation coefficient is .00, one cannot conclude that there is no association between variables

• A curvilinear relationship might better describe the association

31

Curvilinear Correlation

Figure 2.7An Example ofa CurvilinearRelationship

32

Multiple Correlation

• Multiple correlation coefficient– Overall linear association between

several variables & a single outcome variable

33

Meta-Analysis

• Statistical method for combining results from many studies to draw a general conclusion

• Statistical artifacts– Characteristics of a particular study that distort

the results– Sample size is typically the most influential

statistical artifact

34

Module 4: Interpretation

• Reliability– Consistency or stability of a measure

– Test-retest reliability• Calculated by correlating measurements

taken at Time 1 with measurements taken at Time 2

35

High and LowTest-Retest Reliability

Figure 2.8Examples of High and Low Test-Retest Reliability: Score Distributions of Individuals Tested on Two Different Occasions

36

Reliability (cont'd)

• Equivalent forms reliability– Calculated by correlating measurements

from a sample of individuals who complete 2 different forms of same test

• Internal consistency– Assesses how consistently items of a test

measure a single construct

37

Reliability (cont'd)

• Inter-rater reliability– Can calculate various statistical indices to

show level of agreement among raters• Values in the range of .70 to .80 represent

reasonable reliability

• Generalizability theory• Simultaneously considers all types of error

in reliability estimates

38

Validity

• Whether measurements taken accurately & completely represent what is to be measured

• Predictor– Test chosen or developed to assess identified abilities or

other characteristics (KSAOs)

• Criterion– Outcome variable describing important performance

domain

39

Figure 2.9: Validation Process from Conceptual and Operational Levels

Figure 2.9

40

Criterion-Related Validity

• Correlate a test score (predictor) with a performance measure; resulting correlation often called a validity coefficient

• Predictive validity design– Time lag between collection of test data &

criterion data– Test often administered to job applicants

41

Criterion-Related Validity (cont'd)

• Concurrent validity design– No time lag between collection of test data &

criterion data– Test administered to current employees,

performance measures collected at same time– Disadvantage: No data about those not

employed by the organization

42

Content-Related Validity

• Demonstrates that content of selection procedure represents adequate sample of important work behaviors & activities or worker KSAOs defined by job analysis

• I-O Psychologists can use incumbents/SMEs to gather content validity evidence

43

Construct-Related Validity

• Investigators gather evidence to support decisions or inferences about psychological constructs

• Construct - concept or characteristic that a predictor is intended to measure; examples include intelligence, extraversion, and integrity

44

A Model for Construct Validity

Figure 2.10.A Model forConstruct Validity

45

Construct Validity Model of Strength and Endurance Physical Factors

Figure 2.11