Improving the Effectiveness of Interviewer Administered Surveys

though Refusal Avoidance Training

Grace E. O’Neill

Presented by Anne Russell

U.S. Census Bureau

ICES-III - June 19, 2007

2

Outline of Presentation

• Background

• Respondent focus groups

• Refusal avoidance training

• Training content

• Discussion

• Future developments

3

Background

4

Respondent contact staff

• Interviewers: outgoing calls to gain cooperation and to gather data

• Clerks: incoming calls to provide basic information and resend forms

• Analysts: professional staff who place outgoing calls concerning data errors or nonresponse follow-up

5

Respondent contact training

• Interviewers: Centralized training on telephone skills and refusal avoidance and conversion

• Clerks: Shadow experienced clerks with informal discussion

• Analysts: Basic training on respondent contact techniques

6

Why clerks?

• Self-administered paper/ electronic forms

• Establishment respondents need less encouragement to participate

• Nature of telephone calls

7

However…

Over time, the role of the clerk has evolved into the role of an interviewer

• Less cooperative respondents

• More data collected over the telephone

• More nonresponse follow-up telephone calls

Yet, their training has not evolved

8

Respondent Focus Groups

9

Focus Groups

• Monthly Trade Surveys

• Wholesale and retail firms

• Conducted by outside firm

• Six focus groups, 44 participants

• Respondent’s views and impressions

• Ways to improve survey

10

Focus Groups

Finding

• Inexperienced clerks needed skills to increase participation by respondents, especially on voluntary surveys

Solution

• Refusal avoidance training

11

Refusal Avoidance Training

12

Refusal Avoidance Training

• Groves and McGonagle (2001)

– Assemble respondent concerns

– Develop responses

– Train interviewers to classify concerns

– Train interviewers to provide quick and appropriate responses

13

Refusal Avoidance Training

• Interactive

• Cooperative learning

• Flexible

• Survey specific

• Provides telephone skills, refusal avoidance techniques, and improves communication

14

Training specifics

• Monthly Trade Surveys (MTS) and Quarterly Services Survey (QSS)

• 43 MTS clerks, 14 QSS clerks

• Mix of tenure and survey experience

• Supervisor and survey manager involvement

• Eight hours over two days

15

Training Preparation

• Adapt household- based training to establishment-based survey

• Adapt training to the MTS or QSS

• Identify respondent concerns and develop solutions

• FAQ Job Aid

16

Training Content

17

Refusal avoidance training modules

Module 1: IntroductionModule 2: Survey SpecificModule 3: Shared ExperienceModule 4: Preparing a Telephone CallModule 5: Telephone SkillsModule 6: Identify, Analyzing, and Dealing with

Reluctance and RefusalModule 7: Recovering from Negative CallsModule 8: Wrap-up and Evaluation

18

Module 1

Introduction

• Introduces training to clerks

• Introduces the trainer and clerks to each other

• Provides training schedule

19

Module 2

Survey Specific

• Introduces survey specific content

• Conducted by survey manager or survey staff

20

Module 3

Shared Experience

• Identify clerks’ biggest concerns and difficulties

• Develops solutions

• Concerns are used in a later module

21

Module 3 example

• Trainer: “What are some of the things respondents say when you talk to them?”

• Clerks: “Why should I do this,” “I’m not a wholesaler” etc.

• Group: Identify appropriate responses • Trainer: Make sure identified concerns

are addressed and adds new concerns to list

22

Module 4

Preparing for the Telephone Call

• Asks clerks how they prepare for telephone calls

• Assess what tools clerks use to find information about company and what tools they might need

23

Module 5

Telephone Skills

• Assess clerks active listening skills

• Discuss tone

• Discuss mechanics of placing a telephone call

24

Module 6

Identifying, Analyzing and Dealing with Reluctance and Refusal

• Classify concerns identified in Module 3 as reluctance and/ or refusal

• Further discuss solutions

• Paired practice during class

• Reviewed updated FAQ Job Aid

25

Module 7

Recovering from Negative Calls

• Discusses recovering from refusals and other negative calls

• Helps clerks to evaluate negative experiences

26

Module 8

Wrap-up and Evaluation

• Review main training points

• Clarify any remaining concerns

• Evaluations by clerks

27

Results

28

Results

Clerk evaluation

• Usefulness of workshop

• Usefulness of skills learned

• Increased confidence, preparation, and communication

29

Results

Response rates

Caveats:

• Not experimentally tested

• Confounded by other survey conditions

• Don’t know how many potential refusals clerks prevented

30

Results: MTS

Initially lower from previous response period

• Loss of clerks

• Misclassification of refusals

31

Results: MTS

Over time

• Good response rates, decrease in wholesale refusals

• Continued improvement

• Clerks have more responsibilities

32

Results: QSS

Decline in response rates

Imputation remained stable

• More companies refused

• Larger companies continued to report

33

Results: QSS

• Communication has improved

• Clerks are open about sharing concerns with survey managers

• Bi-weekly telephone calls with call center

• Annual refresher training

34

Discussion

35

Benefits

• Centralized dissemination of skills and information

• Practice occurs in a test environment

• Provides training at regular intervals

• Proactive training instead of reactive training

36

Benefits

• Communication between clerks, supervisors, and survey managers

• Clerks feel invested in data collection process

• Survey managers gain direct insight into data collection process

37

Costs

• Staff time writing and delivering training

• Telephone coverage

• Monetary cost of training

• Difficult to provide conclusive evidence

38

Future Developments

39

Future Developments

• Formalized process for clerk training

• Data capture of call concerns

• Follow-up evaluation by clerks

• Analyst training

40

Thank You

Please feel free to contact the author at:

Grace E. O’Neill

Email: grace.e.oneill@census.gov

Phone: 301-763-3537