Recommended Practices for the design of business surveys … S17CP4.pdf · Recommended Practices...
Transcript of Recommended Practices for the design of business surveys … S17CP4.pdf · Recommended Practices...
Recommended Practices for the
design of business surveys
questionnaires Stefania Macchia
(Istat, National Accounts and Economic Statistics Department)
NTTS 2015 Conference
Bruxelles, 10-12 March
2
ISTAT Business Statistical Portal implements a new approach for the organization and
management of data collection processes. It constitutes a single entry point for web-
based data collection from enterprises according to a “business-centric” perspective.
Process 1
Businesses register
Process n
Businesses
Statistical Portal
Data capturing systems
Communication
The context
High integration among all the phases of the productive processes: survey design, data
collection, frame, design of electronic questionnaires, elaboration, estimations, informative
«return»
Rationalization and intensification of the informative flows among «stakeholders» involved in
statistical processes (production lines, statistical registers, data capturing systems, dissemination and
communication, statistical units)
Introduction of new management procedures characterised by high interrelation degree among Portal,
production lines, production support facilities
High potential in terms of systemic efficiency
Main
Fea
ture
s:
Harmonisation of business surveys questionnaires
3
The new Business Portal architecture implies the acceleration of the harmonization
of survey questionnaires, both in terms of concepts treated and features of
electronic questionnaires, with the perspective of:
reducing respondent burden in terms of duplication of information requested
adopting a common language of communication with enterprises through a
standard approach for questionnaires design, which takes into consideration the
peculiarities of the different business surveys questionnaires (SBS and STS),
with attention to the variables definitions management and to the structure of
questions to collect information
enhancing the level of quality (in terms of response rates, but also
homogeneity of collected data).
Manual of Recommended Practices
for the design of business surveys
questionnaires
Manual of Recommended Practices for the design of
business surveys questionnaires
Not all the aspects of the structured process to design questionnaires have been treated
because, in this context, the aim was not to plan new surveys ex novo, but to restructure
and harmonize surveys to be included in Business Portal.
The manual is concentrated on:
The definition of the questionnaire schemes
All the aspects of the questionnaire design related to the data capturing technique
4
GSBPM v.5:
2.2 Design variable descriptions
2.3 Design collection
Questionnaire Electronic Questionnaire
design design
‘The sub-process 2.2 may need to run in parallel with sub-
process 2.3, as the definition of the variables to be collected, and
the choice of collection instrument may be inter-dependent’
Recommended Practices for the design of business
surveys questionnaires
In defining Recommended Practices, it has been considered how some aspects of the
design of questionnaires and of the features to be implemented in the electronic
questionnaire have an impact on the steps of the response process, represented in a
‘hybrid response model’ based on a cognitive approach.
5
Encoding in Memory / Record Formation
Selection / Identification of Respondent(s)
Assessment of Priorities (Motivation)
Release of the Data
Comprehension
Retrieval
Judgement
Communication.
The questions wording, their structuring, the layout
and all the instructions and information at disposal
of respondents related to each question impact on
the comprehension and retrieval processes.
The skipping and consistency rules implemented in
electronic questionnaire support for the judgement
step
the electronic questionnaire design impact on the
communication process, as it predefines the
structures according to which responses have to be
given.
the questionnaire design impact on the Encoding
in memory/Record formation step because as more
the way data are stored in the respondent data base is
considered in structuring questions, as more the effort
needed to provide data is contained.
Ibrid response model (2000 Sudman, 2001 Wilimack,
Nichols)
Manual of Recommended Practices for the design of
business surveys questionnaires
6
First section of the Manual:
General principles and particular ones related to the CAWI technique (shortly described)
Principles defined according to a strategy which should:
Encourage responses
Contain responses errors
Reduce the effort necessary to provide answers
Give respondents a feedback
Very important general principle, consistent with an integrated approach:
contain information to be collected, using sources already available like:
Administrative/fiscal sources
Data collected in other surveys
Official registers.
Manual of Recommended Practices
7
The set of recommendations provided are organised in paragraphs according to specific
homogeneous themes:
1) Design of questionnaire sections/Web pages
Suggestions are given on how designing the sections, taking into considerations the Web
pages, which in turn, can be composed by more than one computer screen-shots.
2) Questions design
Identification of the structure to be used to provide the answer: suggestions are provided on
quantitative questions (how to manage decimal values, scale factors, measure units, etc.),
closed questions (when using radio-buttons, drop-down-boxes, check-boxes), free texts
questions to be coded (when it is preferred to use assisted coding functions) and, finally,
questions organised in tables/arrays.
3) Use of classifications
When classifications are used to classify textual responses, suggestions are given on
different algorithms for textual matching which can be used.
Manual of Recommended Practices
8
4) The structure for variables definitions
Harmonising the definitions in terms of their content is surely the first problem to be faced
but how this content is expressed and structured has in impact on the comprehension
process as well. For this reason a common modular structure for definitions of variables
has been defined
5) The management of skipping rules
The automatic management of skipping rules is a strength of data collection computer
assisted techniques. Concerning business surveys questionnaires, different suggestions
are given for skipping rules related to questions belonging to the same section/web page or
to different section/web pages.
6) The checking plan
Suggestions are given on the management of the consistency rules, taking into
consideration aspects, like: rules regarding variables of the same web-page or different
ones, number of checking rules on a single web-page, relation between skipping and
checking rules, etc.
7) The screen design
Standards already defined for software package used in the Business Portal architecture
have been adopted. Some details have been provided on graphic symbols to be used to link
definitions, instructions, helps, etc.
Manual of Recommended Practices
9
8) The guide and instructions for respondents
A set of documents is always provided to respondents (e.g. brief description of the survey,
normative aspects concerning confidentiality, compilation guide, etc.). Common structures,
templates and access modes have been designed for these documents, because they can
have an impact on the comprehension process as well.
9) Questions to estimate the perceived respondent burden
The migration of surveys questionnaires in the new generalised sw system for electronic
questionnaires implementation is a good occasion to collect information on respondent
burden directly from respondents. For this purpose a set of questions has been proposed,
which distinguish possible burden due to difficulty in retrieving the requested information
and in filling in the questionnaire. The responses to these questions could be analysed
together with other indications derivable from paradata.
10) The use of paradata for data collection monitoring and optimising questionnaire
design
The Business Portal already produces some indicators for the monitoring purpose. A set of
other indicators has been identified which can be used by the researcher to optimise the
questionnaire design, specifically when some ‘alerts’ are highlighted by the answers given to
questions on the respondent burden (e.g. number of entries for web-pages, connection time
for the entire questionnaire or for each web-page, length of navigation routings, etc.).
10 10
The structure for variables definitions The harmonisation of the
contents to be observed
In business statistics some variables can be collected in different surveys
questionnaires. This can depend on different informative needs (structural and short-
term surveys), while in other situations it could sometime result to be redundant.
On the other hand, big business agencies are often involved in more surveys.
Definitions of the same variables could be different:
in terms of their content because EU regulations contain different specifications for structural
and short-term surveys, or
in terms of wording, because expressed differently by each survey responsible.
Contents harmonisation in terms of variables definitions and questions wording is
the pre-requirement for a two steps activity:
Improvement of the harmonisation and comparability of collected data
Reduction of redundancies and respondent burden
11 11
A common modular structure for definitions of variables has been defined,
which will be a standard for all variables definitions.
The same informative content can be presented to respondents through different wordings, different
structures and different layouts.
All these aspects have an impact on the comprehension process.
The structure for variables definitions The harmonisation of the
contents to be observed
A Definition (core of the definition = real meaning, expressed in a
simple, unambiguous and synthetic way)
B Inclusions/Exclusions clauses
C National peculiarities
D Peculiarities of sectors/surveys
This structure is a useful tool for the improvement of harmonisation and comparability of collected
data. It makes it easier:
converging on the core of the definition
identifying differences of informative needs and, consequently, structuring questions so as to
reduce redundancies in data collection
12 12
Two variables, used by different survey, have been considered with the
purpose of harmonising definitions and testing the standard structure:
Number of employees
Turnover
Sources used to produce harmonised definitions
Eurostat Regulations on SBS and STS surveys
Definitions already used in Istat surveys (SBS and STS)
Indications produced by ESSNet project on consistency (WP3)
Definitions of Istat Business register
Definitions of Business census.
The structure for variables definitions The harmonisation of the
contents to be observed
REFERENCES
14
• J.Bethlehem, S.Biffignandi (2011) Handbook of Web Surveys, eds Wiley
• Eurostat (2006) Handbook of Recommended Practices for Questionnaire Development and
Testing in the European Statistical System
http://ec.europa.eu/eurostat/ramon/statmanuals/files/Handbook_of_Practices_for_Quest.pdf
• Eurostat (2014) Memobust handbook - Handbook on Methodology of Modern Business
Statistics http://www.cros-portal.eu/content/memobust
• N.Fazio, M.Murgia, A.Nunnari (2013) The Business Statistical Portal: a new way of
organising and managing data collection process for business surveys in ISTAT Proceedings
of the UNECE Seminar on Data Collection, Ginevra
• Sudman, S., D. K. Willimack, E. Nichols and T.L.Mesenbourg (2000) “Exploratory
Research at the U.S. Census Bureau on the Survey Response Process in Large Companies,” Proceedings of the Second International Conference on Establishment Surveys, American
Statistical Association,