The Response Process Model as a Tool for Evaluating Business Surveys Deirdre Giesen Statistics...
-
Upload
alexandra-dolan -
Category
Documents
-
view
215 -
download
0
Transcript of The Response Process Model as a Tool for Evaluating Business Surveys Deirdre Giesen Statistics...
The Response Process Model as a Tool for Evaluating Business Surveys
Deirdre Giesen
Statistics Netherlands
Montreal, June 20th 2007
ICES III
Outline
–Questionnaire testing at Stat Netherlands–Collecting data on the response process –Reviewing field visits to reflect on response process model –Data used –Preliminary results
Questionnaire testing
– Recently more attention for establishment data collection (efficiency, response burden and quality)– One of the strategies: improving questionnaires–Focus of Question Lab: Response Burden and Data Quality– Preferably: multi method evaluation– Favorite method: company visits to collect data response process
Response process model for business surveys
1. Encoding of information in company records or memory
2. Selection and identification of the respondent(s).3. Assessment of priority
4. Comprehension of the data request5. Retrieval of relevant information from records or
memory6. Judgment of the adequacy of the response7. Communication of the response
8. Release of the data
Sudman, S., Willimack D.K., Nichols E. & Mesenbourg T. (2000), Willimack, D.K. & Nichols E. (2001)
Collecting data on the response process
– on site – methodologist and field officer – mix of observing and reconstructing– if necessary: general interview – standard protocol with adaptations– detailed visit reports
– video taping
Pilot study by Hak & Van Sebille (2002)
Standard protocol
1. Introduction
2. General questions
3. Observation or reconstruction response process
4. Evaluating
5. Correcting data and answering questions
Review of field visits:
1. Which problems did we find that caused data error and/or response burden?
2. How are these problems linked to the different steps of the model ?
3. To what extent were the steps of the model useful for describing and understanding the process of responding to a business survey?
Evaluation studies reviewed
Name Mode # reports reviewed
SBS2003 Paper 10
Transport Electronic 3
Producer Prices Electronic 5
International trade Electronic 7
Sourcing Electronic 5
Respondents and visitsreviewed
– retrospective interviews (6), observations (15) and general interviews about response process (9) – respondents from size classes 0 to 9– respondents from retail, wholesale, service, manufacturing, building, transport and external accountants
Encoding
Problems found
– Lack of available information important and source of response burden and data error
– Important to distinguish lack of information and (ease of) accessibility of information
Recommendations – Change information request if possible– Assist respondent with data collection
Selection and Identification of Respondents
Problems found
– electronic forms extra difficulties –distribution from SN to firm – characteristics of respondent– distribution within firm– change of respondents
Selection and Identification of Respondents
Recommendations
– information on who to contact– instrument design should allow for easy forwarding of (parts of) the exact questionnaire– data request and specific arrangements with firm should be documented in a way understandable for new respondent
Assessment of priorities
Problems found
– Timely and correct completion is generally not a high priority– Most respondents hardly see any reward or benefits for their effort– Some respondents may deliberately provide wrong data to prevent response burden
Assessment of priorities
Recommendations
– design questionnaires for quick readers and clickers – adapt data collecting strategies: reminders, quality control, incentives and penalties– improve general communication to stress importance of contribution to national statistics
Comprehension
Many problems found at several levels– General design and goal of the study – Overall design of the instrument – Specific questions
Recommendations – Improve communication ´around´ questionnaire– Develop tailored questionnaires for small businesses in lay language– Many suggestions to improve wording, order, layout of total instrument and specific questions
Retrieval
Problems found
– Using the wrong sources – Lack of access to or cooperation from sources– Lack of knowledge of sources – Compiling errors (also if automated)– Excessive response burden for certain tasks– Retrieval strategies vary
Retrieval
Recommendations – ask less detailed information if possible– explicitly allow estimates for known difficult variables – design materials to make internal data collection easier and more accurate– stress more clearly about which unit respondents should be reporting on
Judgment
Often difficult to distinguish retrieval and judgment problems
Problems found– Checking the questionnaire can cause high response burden– Motivation to do so lacks – Instrument design can hinder easy checking and editing– Few confidentiality issues
Judgment
Recommendations– Stimulate respondents to check their answers by automated checks in electronic instruments and quality control combined with feedback after submission of the data. – Design instruments to facilitate checking and editing the data by respondent
Communication
Problems found– many usability issues in e-forms that made communicating a specific answer difficult – data error through obligatory fields – electronic sending of questionnaire important source of response burden en data error
Communication
Recommendations– many specific technical and usability issues that need to be addressed– allow empty fields for known difficult variables– ideas for electronic questionnaire functionalities that will make communication of answer easier
Release of the data
No problems with response burden or data error problems found in this step
Problems ´outside´ model
– Filing of report for internal use – Lack of feedback after data has been submitted
Conclusions
– response process model is helpful framework to find problems with data collection– some problems can only be discovered when studying the actual response process in detail– response process does not end with release of the data, model might be extended to take this into account