Eurostat Questionnaire design. Presented by Paweł Lańduch Central Statistical Office, Statistical...

Eurostat

Questionnaire design

Presented by

• Paweł Lańduch • Central Statistical Office, Statistical Office in

Poznań, Poland.


Modules• Main theme,• Electronic questionnaire design,• Editing during data collection,• Testing the questionnaire .


• Business surveys – special population– the response process is complex and burdensome, since

preparing the required data entails a mixture of organizational and individual tasks. Data are mostly quantitative and their acquisition requires access to business records;

– the predominant role of self-administered questionnaires and the role of the respondent. In the case of establishments, the respondent is usually a representative, who acts as a data provider in an organizational environment,


• Business surveys – special population– the mandatory character of reporting as opposed to

mostly voluntarily aspect of household surveys,– concepts and definitions are technical, complex, and

often based on legal or regulatory considerations. Practices, terminology and standards used by businesses in their daily operations, the accounting standards, for example, are the context that needs to be taken into account. This calls for detailed instructions that accompany questions.


• Business surveys – special population– the distribution is skewed in favor of larger

establishments, – timeliness is often given priority over quality.– the longitudinal character of surveys and overall

reluctance to changes in measurement instruments used.


• Response processDefinition - the result of the interaction between a

respondent and a questionnaire (Edwards & Cantor, 1991, Measurement Error in Surveys)

Houshold surveys versus business surveys individual organisation using memory using records cognition cognition + organisation retrieval retrieval + records look up

• Concepts and variables (example) Short term statistics– Term – Industrial production sale value in current prices

(Sa)– Variables

Revenues from products sale (Sw) Value of goods production not including in sales (Sz) Excise imposed on products (Da) Subsidy for products (recevied for products)(Dp)

Sa=Sw+Sz-Da+Dp


• Choosing the mode of data collection- Interviewer administered - Self-administered- Mixing modes- Impact of computer technology – web forms

- Actual burden versus perceived burden

- Numerical character of data- One of the most burdensome phase – data retrieval


• Building the prototype questionnaire– Content and non-content elements

Introductory letters, Cognitive aspects Layout, navigation instructions technological challanges

– Heavy reliance on instructions– Guidelines (examples: Economic Directorate Guidelines

on Questionnaire Design. Washington, DC: U.S. Census Bureau (2008).


– Guidelines cont. examples: Forms Design Standards Manual, Australian Bureau of Statistics (2010),

– Handbook of Recommended Practices for Questionnaire Development and Testing in the European Statistical System, Eurostat (2006)

- Creating standards as a framework for developing, testing and evaluation, also for technical aspects

- Example (Web environment – hollistic view, dynamics, one point for collection, one-size-fits-all ?)

- Measurement errors-Quality Cost-Response Burden-User satisfaction


• Impact on response process • Impact on response burden• Impact on quality• New features and opportunities but also challages • Electronic versus paper

Layout – paging, scrolling, should the electronic versions be smiliar in appearance to those on paper ?

Editing – moving closer to the respondent, cost reduction, reducing burden, seeking better quality

Questionnaire design – Electronic questionnaire

Automatic routing – eliminate routing errors, previous answers determine next questions presented,

Calculations – expecting by respondents, are elements of data consistency and validity

Progress indicator – possibility of checking the completion status, there are also arguments against placing it

Navigation – seeking easy way for moving forward and backward, possibility of interutption and resuming

Instructions – close at hands, examples: hyperlinks and pop-ups

Security – important for respondents espacially in busines surveys

Questionnaire design – Electronic questionnaire cont.

Testing for usability - Gestalt psychology for visual design- Cognitive psychology

- designer’s perspective versus user’s perspective

- user-centre design- designing for errors- visibility, mapping and feedback- standardizing

- heuristic approach (Jacob Nielsen)


Example: paper questionaire and its electronic counterpart


Microsoft Office as an example of standardizing a toolbar in a suite of programs



The effect of automatic routing (the gap in numbering)

- Possibility to move some of editing activities (so far as a post-collection process) on collection stage.

- Reduction of costs and data quality improvement- Potentials for lowering burden - Build-in editing rules called edit checks or edits – a respondent is

notified about errors while entering responses- Types of implemented edit rules – hard edits and soft edits

There is an expectation collected data are of high quality and contain no errors – THAT MAY NOT BE TRUE

Questionnaire design – Editing during data collection

Presenting messages to respondents (examples from CSO of Poland)


Using icons in messagesMarking hard edits failures (errors)

Soft edits (warnings)

Text of message as a pop-up box or in a list (examples)


Red text: Error: The information you provided does not match our criteria.

Various types of edits for various types of items:- Text – length, format check- Numbers – range, algebraical relations, balance check- a reponse choosing from a predefined list- Required item check- A rule allowing only some sort of characters e.g. numeric- Checking the relationship between two or more items


Considerations when choosing a strategy for edit rules in questionnaires

- How many edit rules will be effective and not resulting in item non-response ?

- Conservative aproach versus extensive usage of edit rules

- Which way of preseting messages is efficent to be recognized by respondents ?

- Even if the messages are perceived they should be understood


Considerations when choosing a strategy for edit rules in questionnaires (cont.)

- How data with unresolved edits should be treated ?- How to test and evaluate edits in the questionnaire ?- Should be permitted to submit data with unresolved edits ?

Some hints from usability testing:– user should be at the center of design, the task should

be under the user’s control– Words „error” and „mistake should be avoided


Some hints from usability testing (cont.):– Edit rules should be triggered immidiately as they can,

but with a possibilty of a repetead execution,– The message should be provided with attributes for easy

localization of the problematic item– Clear signification to the respondent the obligation to resolve all

edits or to provide a comment before submission– Deliberation before providing data from earlier periods to avoid

acquiescence bias


– European Statistics Code of Practice, Principle 8, Indicator 8.2: In the case of statistical surveys, questionnaires are systematically tested prior to the data collection.

The continuous process– A draft questionnaire review pretesting revision

– Data collection evaluation revision pretesting

– pretesting / field testing

Questionnaire design – Testing the questionnaire

Questionnaire design – Testing the questionnaire- CASM Cognitive aspects of survey methodology - „the subject” thought process of response

Think aloud Verbal probing

Social surveys Business surveys at business location presurvey visits expansion of cognitive

interviewing

Record keeping studies- One of the most burdensome step in the response process –

retrieval of the demanded data - Data may not exist in business records - Data may be kept in different places- Cognitive background- Different persons with various degree of access to data


Problems with data retrieval estimation rough estimation blunder non-response

Usability testing

Visual aspect• Fonts• Graphics• Visual clutter

User interface• Navigation• Skipping routes


- Usability testing (cont.)Software

the process (planning, analysis, implementation, assesement, closure)

functional tests (black box)

structural tests (white box)

- Testing in the field (after data have been collected)Pilot tests – dress rehearsal

Debriefing respondents – formal and informal - the source of information about the questionnaire

Debriefing survey stuff – formal and informal – as intermediaries between respondents and survey agency have valuable knowledge for the questionnaire evaulation


• Testing in the field (after data have been collected)Post-collection data evaluation – a tool for assessing data quality,

– Measuring non-response– Detecting outliers– Data editing failures– Comparing with data from other sources



Qualitative methodsPretesting and cognitive pretesting can lead to changes

and improvement

Quantitative methodsSeeking measures how changes have worked

Setting standards for testing• Minimal standards• Extended standards• Best practices

Response process as a tool for the questionnaire evaluation

• Based on steps of the response process in business surveys

• Based on a topic list of questions• Qualitative method for improvements


Thank you very much for your attention

Eurostat Questionnaire design. Presented by Paweł Lańduch Central Statistical Office, Statistical...

Documents

Transcript of Eurostat Questionnaire design. Presented by Paweł Lańduch Central Statistical Office, Statistical...