Joint EC-OECD Workshop on International Development of Business and Consumer Tendency Surveys...

Joint EC-OECD Workshop on International Joint EC-OECD Workshop on International Development of Business and Consumer Tendency Development of Business and Consumer Tendency

SurveysSurveys

Brussels Brussels

14-15 November 200514-15 November 2005

Task Force on Harmonisation of Survey Operation and Task Force on Harmonisation of Survey Operation and Technical DesignTechnical Design

Efficient Sample Design and Weighting Efficient Sample Design and Weighting MethodologiesMethodologies

Analysis of Key Issues and RecommendationsAnalysis of Key Issues and Recommendations

Efficient Sample Design and Weighting MethodologiesEfficient Sample Design and Weighting Methodologies

• The usefulness of BTS and CS strongly depends on The usefulness of BTS and CS strongly depends on their statistical qualitytheir statistical quality

• Quality of survey data may be measured in terms of Quality of survey data may be measured in terms of (OECD, 2003): (OECD, 2003):

• reliability reliability • timeliness of releasetimeliness of release• comparability over timecomparability over time• transparency and accessibility to the userstransparency and accessibility to the users

• This task force has to do with sample design and This task force has to do with sample design and weighting methodologiesweighting methodologies

• A sound definition of these aspects increases the A sound definition of these aspects increases the reliability – and therefore the overall quality – of the reliability – and therefore the overall quality – of the datadata

Efficient Sample Design and Weighting MethodologiesEfficient Sample Design and Weighting Methodologies

• Business Tendency Surveys (BTS) are conducted on Business Tendency Surveys (BTS) are conducted on manufacturing, services, retail and construction manufacturing, services, retail and construction sectorssectors• Consumer Surveys (CS) measure households’ Consumer Surveys (CS) measure households’ opinion and expectations on personal and general opinion and expectations on personal and general economic situationeconomic situation• For both BTS and CS, key issues in Efficient For both BTS and CS, key issues in Efficient Sampling Design and Weighting Methodologies will be Sampling Design and Weighting Methodologies will be analyzedanalyzed• An overview of the current practices in these fields An overview of the current practices in these fields will be presented, from which it will be derived a will be presented, from which it will be derived a number of draft recommendations aimed at number of draft recommendations aimed at improving the overall quality of the surveysimproving the overall quality of the surveys

Efficient Sampling Design for Business Tendency Efficient Sampling Design for Business Tendency SurveysSurveys

• IIn the case of BTS, key Issues for an efficient sample n the case of BTS, key Issues for an efficient sample design are (see also Donzè, Etter, Sydow, Zellweger, design are (see also Donzè, Etter, Sydow, Zellweger, “Sample Design for Indystry Survey”, ECFIN/2003/A3-“Sample Design for Indystry Survey”, ECFIN/2003/A3-03):03):

• Identification of relevant Universe/reference populationIdentification of relevant Universe/reference population• Identification of the sample frameIdentification of the sample frame• Identification of the correct method for sample selectionIdentification of the correct method for sample selection• Treatment of missing dataTreatment of missing data


• The first step in setting up a BTS is the choice of the The first step in setting up a BTS is the choice of the Relevant Universe/reference populationRelevant Universe/reference population• Typically, it is represented by all the firms operating Typically, it is represented by all the firms operating in a given sector, as resulting from some in a given sector, as resulting from some official/statistical registerofficial/statistical register• Some firms may be excluded looking at their size Some firms may be excluded looking at their size (i.e., below a certain size threshold) or location, or on (i.e., below a certain size threshold) or location, or on the basis of their structural characteristic (i.e., the basis of their structural characteristic (i.e., exclusion of government bodies) exclusion of government bodies)


• The second step is the choice of the sample frame, The second step is the choice of the sample frame, having the goal of maximize sample coverage and having the goal of maximize sample coverage and minimize coverage errorsminimize coverage errors• BTS are usually based on a panel of responding BTS are usually based on a panel of responding firms, that are re-interviewed each month firms, that are re-interviewed each month • Demographic and structural characteristic of the Demographic and structural characteristic of the respondents have to be known in order to build up respondents have to be known in order to build up the sample, the construction of the frame implying the sample, the construction of the frame implying the following steps:the following steps:

• identification of the appropriate frame listidentification of the appropriate frame list• eventual adoption of a cut off strategyeventual adoption of a cut off strategy• identification of the sample, reporting and response unitidentification of the sample, reporting and response unit• Updating of the frame listUpdating of the frame list


• The frame list may be derived from:The frame list may be derived from:• official or statistical registers official or statistical registers • membership lists of business associations and chamber of membership lists of business associations and chamber of commercecommerce

0%

10%

20%

30%

40%

50%

60%

non eu eu-25 total

Manufacturing Services Retail Construction

Statistical registers Business directory Governement registers and other


• The adoption of a cut off strategy may respond to the need of:The adoption of a cut off strategy may respond to the need of:• better focusing on the sector of interest (cut-off with respect to the better focusing on the sector of interest (cut-off with respect to the sector of activity)sector of activity)• ensuring a certain stability of the panel, excluding firms below a ensuring a certain stability of the panel, excluding firms below a certain threshold (cut-off with respect to size)certain threshold (cut-off with respect to size)

• The sample unit is the unit on which to perform the sample The sample unit is the unit on which to perform the sample selection procedure; the main choice is between having:selection procedure; the main choice is between having:

• the whole firmthe whole firm• establishments, local units, or kind of activity units (kau)establishments, local units, or kind of activity units (kau)

• Even if the firm is chosen as the sample unit, it is possible to Even if the firm is chosen as the sample unit, it is possible to have more reporting and response units within the firm, sending have more reporting and response units within the firm, sending the questionnaires to different establishments/local units/KAUs the questionnaires to different establishments/local units/KAUs within the firmwithin the firm


•

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

eu non Eu Total


Establishemnt Enterprise Mixed (establishment + kau)


• Finally, the list should be updated frequently in order to keep track of Finally, the list should be updated frequently in order to keep track of the changes in the structure of the reference Universe and avoid the changes in the structure of the reference Universe and avoid possible problems in terms of:possible problems in terms of:

• Under coverage (new firms entering the market) Under coverage (new firms entering the market) • ineligibility (old firms exiting the market) ineligibility (old firms exiting the market) • duplicate entries (errors)duplicate entries (errors)

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

50%

Continously more than 1time per year

yearly between 1 and 6years

other (ad hoc) not available



• The third step implies the choice of an appropriate method of The third step implies the choice of an appropriate method of sample selection sample selection • The sample should be representative of the relevant UniverseThe sample should be representative of the relevant Universe• In order to build up a representative sample, choices have to In order to build up a representative sample, choices have to be made relatively to: be made relatively to:

• The sampling method The sampling method • The sampling sizeThe sampling size

• BTS are usually based on a fixed panel of reporting unitsBTS are usually based on a fixed panel of reporting units• A fixed sample structure may rise representitiveness problems, A fixed sample structure may rise representitiveness problems, because the panel may loose its initial representativeness if it is because the panel may loose its initial representativeness if it is not updated regularly not updated regularly • For this reason, Institutes often use a rotating panel method, in For this reason, Institutes often use a rotating panel method, in which a a fixed percentage of units are replaced at regular which a a fixed percentage of units are replaced at regular intervalsintervals• More precisely, the largest and most important firms should More precisely, the largest and most important firms should ideally always be included, with smaller firms being rotated out ideally always be included, with smaller firms being rotated out on a regular basison a regular basis


• Possible methods of sample selection include:Possible methods of sample selection include:• non-statistically founded methods, such as:non-statistically founded methods, such as:

• comprehensive surveys with cut-offcomprehensive surveys with cut-off• purposive or quota samplingpurposive or quota sampling

• statistically funded methods, such as:statistically funded methods, such as:• simple stratified samplingsimple stratified sampling• stratified sampling (PPS, OAS)stratified sampling (PPS, OAS)

• Once selecting the sample, Institutes have to choose Once selecting the sample, Institutes have to choose appropriate sample sizesappropriate sample sizes

• Generally speaking, sample size will be chosen accordingly to a Generally speaking, sample size will be chosen accordingly to a predetermined desired maximum measurement errorpredetermined desired maximum measurement error


•

0%

5%

10%

15%

20%

25%

30%

35%

40%

Representativepanel

Quota Simple stratif iedsampling

Stratif iedsampling w ith

PPS

Stratif iedsampling w ith

Neymanallocation

Cluster andrandomsampling

Purposivechoice

Comprehensiveor mixed



• The fourth step in building up BTS implies the treatment of The fourth step in building up BTS implies the treatment of missing datamissing data• It is possible to distinguish between It is possible to distinguish between

• unit non response (entire interview is missing)unit non response (entire interview is missing)• item non response (some answer is missing)item non response (some answer is missing)

• If missing data occur with large, dominant firms the problem is If missing data occur with large, dominant firms the problem is severesevere• If there is missing data among the smallest firms, the problem If there is missing data among the smallest firms, the problem is less critical and missing data may easily be imputed on the is less critical and missing data may easily be imputed on the basis of the answer of similar firmsbasis of the answer of similar firms• Most commonly used method for dealing with missing data Most commonly used method for dealing with missing data problem is the use of follow-up techniques (re-interviewing with problem is the use of follow-up techniques (re-interviewing with telephone, fax or web based techniques)telephone, fax or web based techniques)• If missing data problems persist, re-weighting or imputation If missing data problems persist, re-weighting or imputation methods should be used methods should be used

Weighting Methods for Business Tendency SurveysWeighting Methods for Business Tendency Surveys

• Weighting is used to transform data for the realized sample into Weighting is used to transform data for the realized sample into estimates for the reference populationestimates for the reference population• Weights may be based on:Weights may be based on:

• information coming from the survey itself (size, output of the firm)information coming from the survey itself (size, output of the firm)• auxiliary sources (official/statistical data on the size of the reference auxiliary sources (official/statistical data on the size of the reference sector/region)sector/region)

• Weighting methods are usually based on either:Weighting methods are usually based on either:• one stage weight schemeone stage weight scheme• two stage weight schemetwo stage weight scheme

• In the one-stage scheme, a weight is associated with each In the one-stage scheme, a weight is associated with each reporting unit, in order to take into account its relative importance reporting unit, in order to take into account its relative importance inside the sampleinside the sample• In the second case, a unit-specific weight is used to calculate In the second case, a unit-specific weight is used to calculate strata results, further aggregating the strata with some external strata results, further aggregating the strata with some external sources in order to obtain industry aggregates. sources in order to obtain industry aggregates.

Weighting Methods for Business Tendency SurveysWeighting Methods for Business Tendency Surveys

•

0%

10%

20%

30%

40%

50%

60%

One-stage weightingprocedure

Two-stage weighting procedureNo weights/missing information


Efficient Sampling Design for Consumers SurveysEfficient Sampling Design for Consumers Surveys

• In the case of CS, in order to build up a representative sample choices In the case of CS, in order to build up a representative sample choices have to be made relatively to:have to be made relatively to:

• The sampling frameThe sampling frame• The sampling methodsThe sampling methods

• The construction of the sampling frame implies the following steps:The construction of the sampling frame implies the following steps:• identification of the appropriate frame listidentification of the appropriate frame list• eventual adoption of a cut off strategyeventual adoption of a cut off strategy• identification of the sample unit, reporting unit and response unitidentification of the sample unit, reporting unit and response unit• updating of the frame listupdating of the frame list

• In the identification of the appropriate frame list, it is crucial thatIn the identification of the appropriate frame list, it is crucial that• the right population is being sampled, the right population is being sampled, • all the members of the population have the same chance of being sampledall the members of the population have the same chance of being sampled

• In OECD countries, frame lists are usually based on:In OECD countries, frame lists are usually based on:• official population register (including every adult member of the population)official population register (including every adult member of the population)• telephone register (arising possible bias problems, to be solved adopting telephone register (arising possible bias problems, to be solved adopting random extraction of phone numbers or using other sample techniques for random extraction of phone numbers or using other sample techniques for those excluded from the directories)those excluded from the directories)


• A cut off strategy is often adopted A cut off strategy is often adopted • on the basis of age (cut off age varying often across countries in EU); on the basis of age (cut off age varying often across countries in EU); • in some countries, geographical cut offs are also applied in some countries, geographical cut offs are also applied

• Response unit may differ from the sample unit; typically, sample are Response unit may differ from the sample unit; typically, sample are devised to be representative of all households, with the selected devised to be representative of all households, with the selected respondent reporting on the household as a wholerespondent reporting on the household as a whole• The list should be updated frequently in order to monitor as close as The list should be updated frequently in order to monitor as close as possible the evolution of the relevant populationpossible the evolution of the relevant population

0%

5%

10%

15%

20%

25%

30%

35%

Continously 4 times a year yearly betw een 1and 6 years

more than 6years

not available


• Key issues in sampling extraction include Key issues in sampling extraction include •the choice of the appropriate sampling methodthe choice of the appropriate sampling method• the choice of the optimal size of the samplethe choice of the optimal size of the sample

• In CS usually a independent cross-section of household is extracted In CS usually a independent cross-section of household is extracted each month:each month:

• In EU a general strategy of simple random sampling is usedIn EU a general strategy of simple random sampling is used• In the US a rotating sampling design is usually applied, in which the In the US a rotating sampling design is usually applied, in which the respondent chosen in each drawing is re-interviewed six months later, in respondent chosen in each drawing is re-interviewed six months later, in order to provide a regular assessment of change in consumers’ attitudesorder to provide a regular assessment of change in consumers’ attitudes

• Most widely used methods of sampling extractions include:Most widely used methods of sampling extractions include:• simple stratified samplingsimple stratified sampling• multiple stage stratified sampling multiple stage stratified sampling • Random Digit Dialing methodsRandom Digit Dialing methods

• There is no consensus in the literature on the appropriate sample size There is no consensus in the literature on the appropriate sample size • In practice, sample size currently converges to about 2000, a size In practice, sample size currently converges to about 2000, a size supposed to provide acceptable confidence intervals for this type of supposed to provide acceptable confidence intervals for this type of surveysurvey

Weighting Methodologies for Consumers SurveysWeighting Methodologies for Consumers Surveys

• Information gathered from survey’ respondents may be Information gathered from survey’ respondents may be appropriately weighted to derive aggregate information on appropriately weighted to derive aggregate information on household’ opinion and expectationshousehold’ opinion and expectations• Weights may be based on Weights may be based on

• auxiliary information (demographic or socio-economic weights)auxiliary information (demographic or socio-economic weights)• inverse selection probabilities (sample weights)inverse selection probabilities (sample weights)

• Most commonly used weight variables are:Most commonly used weight variables are:• demographic characteristics of the household:demographic characteristics of the household:

• gender and age of the respondentgender and age of the respondent• region of residence and size of the townshipregion of residence and size of the township

• socio economic characteristics of the household:socio economic characteristics of the household:• economic occupationeconomic occupation• level of educationlevel of education• housing condition, type of area/municipalityhousing condition, type of area/municipality

• A number of Institutes do not use weights: this is appropriate A number of Institutes do not use weights: this is appropriate only when:only when:

•every household has an equal chance of selectionevery household has an equal chance of selection• there is no differential no responsethere is no differential no response

Weighting Methodologies for Consumers SurveysWeighting Methodologies for Consumers Surveys

•

0%

10%

20%

30%

40%

50%

60%

70%

80%

Sample weights Socio Demographiccharacteristic

No weights

Minimum requirements and recommendation for BTS: Minimum requirements and recommendation for BTS: sample design – the sample framesample design – the sample frame

• The frame lists The frame lists • Frame lists should include an as exhaustive as Frame lists should include an as exhaustive as possible account of active firms for the survey of possible account of active firms for the survey of interestinterest• As a consequence, the use of official or statistical As a consequence, the use of official or statistical registers of active firms is recommended over that registers of active firms is recommended over that of – more partial – business or membership of – more partial – business or membership registersregisters

• Cut offCut off strategiesstrategies• Institutes are advised to use cut-off strategies in Institutes are advised to use cut-off strategies in order to stabilize the panel (size cut off) and for a order to stabilize the panel (size cut off) and for a precise identification of the survey objectives precise identification of the survey objectives (branch cut off)(branch cut off)

Minimum requirements and recommendation for BTS: Minimum requirements and recommendation for BTS: sample design – the sample framesample design – the sample frame

• Sample units and reporting unitsSample units and reporting units• Establishments may be considered the ideal choice for the sample Establishments may be considered the ideal choice for the sample unit; however, it may be difficult to gather information at this levelunit; however, it may be difficult to gather information at this level• Use of KAU is advisable if we are particularly interested in the Use of KAU is advisable if we are particularly interested in the industrial structureindustrial structure• Use of local units is advisable if we are particularly interested in Use of local units is advisable if we are particularly interested in the regional structure Sample frame: reporting unitsthe regional structure Sample frame: reporting units• Even if the firms is identified as the sample unit, it is advisable – if Even if the firms is identified as the sample unit, it is advisable – if possible – to have different reporting units within the firmpossible – to have different reporting units within the firm

• Response unitsResponse units• In any case it is strongly recommended that the Institutes ensure In any case it is strongly recommended that the Institutes ensure that the same response unit answer the questionnaire every monththat the same response unit answer the questionnaire every month

• Updating of the listsUpdating of the lists• As a minimum requirement, frame lists should be updated as soon As a minimum requirement, frame lists should be updated as soon as a new census of active firms is availableas a new census of active firms is available

Minimum requirements and recommendation for Minimum requirements and recommendation for BTS: sample design – sampling methodsBTS: sample design – sampling methods

• As a recommendation, a fixed panel should be used…As a recommendation, a fixed panel should be used…• … … established on a statistically founded basis…established on a statistically founded basis…• … … using a rotating pattern of updating …using a rotating pattern of updating …• … … with a fixed percentage of participants being replaced at with a fixed percentage of participants being replaced at regular intervalsregular intervals

• As a minimum requirement, sampling extraction should be As a minimum requirement, sampling extraction should be based on sound probabilistic considerationsbased on sound probabilistic considerations

• The use of exhaustive sampling is possible for small countries The use of exhaustive sampling is possible for small countries or for a sub-set of the sampleor for a sub-set of the sample

• Avoiding of purposive or ad hoc sampling methods is strongly Avoiding of purposive or ad hoc sampling methods is strongly recommendedrecommended

• Different probabilistic methods of sample selection may be Different probabilistic methods of sample selection may be used; as a general consideration, the more heterogeneous is the used; as a general consideration, the more heterogeneous is the population, the more is advisable the use of stratification based population, the more is advisable the use of stratification based sampling methodssampling methods

Minimum requirements and recommendation for Minimum requirements and recommendation for BTS: sample design – treatment of missing dataBTS: sample design – treatment of missing data

• Institutes should define what procedures are used Institutes should define what procedures are used for the treatment of item and unit non response for the treatment of item and unit non response (missing data)(missing data)• As a minimum requirement, institutes are advised: As a minimum requirement, institutes are advised:

• to closely monitor the impact of missing data (especially to closely monitor the impact of missing data (especially for large firms)for large firms)• to use follow up techniques in order to reduce their impact to use follow up techniques in order to reduce their impact (fax, telephone, web remainder)(fax, telephone, web remainder)

•The use of imputation methods to deal with The use of imputation methods to deal with remaining missing data should be considered with remaining missing data should be considered with care, in order to avoid possible distortionscare, in order to avoid possible distortions• The use of re-weighting techniques, taking into The use of re-weighting techniques, taking into account different composition of the panel in adjacent account different composition of the panel in adjacent surveys, may be advisable to reduce the biassurveys, may be advisable to reduce the bias

Minimum requirements and recommendation for Minimum requirements and recommendation for BTS: weighting methodsBTS: weighting methods

• The use of weights is strongly The use of weights is strongly recommended in order to improve the recommended in order to improve the precision of the estimatesprecision of the estimates• As a minimum requirement the use of As a minimum requirement the use of a simple – one stage – system of a simple – one stage – system of weights is suggestedweights is suggested• Two stage (or multiple stage) Two stage (or multiple stage) weighting procedures are advisable for weighting procedures are advisable for heterogeneous population, especially in heterogeneous population, especially in large countrieslarge countries

Minimum requirements and recommendation for Minimum requirements and recommendation for CS: sample design – the sample frameCS: sample design – the sample frame

• Frame list should include an as exhaustive as Frame list should include an as exhaustive as possible account of the adult population possible account of the adult population • As a consequences, official census or As a consequences, official census or statistical registers are to be preferred to statistical registers are to be preferred to telephone registerstelephone registers• If telephone registers are used, appropriate If telephone registers are used, appropriate methods to correct for possible bias is methods to correct for possible bias is recommendedrecommended• Cut off strategies with respect to age are Cut off strategies with respect to age are advisable – this may call for further advisable – this may call for further harmonization in the EU harmonization in the EU • As a recommendation, frame lists should be As a recommendation, frame lists should be updated yearlyupdated yearly

Minimum requirements and recommendation for Minimum requirements and recommendation for CS: sample design – sampling methodsCS: sample design – sampling methods

• As a minimum requirement, random sampling As a minimum requirement, random sampling techniques have to be used in order to ensure survey techniques have to be used in order to ensure survey representitivenessrepresentitiveness

• In case of heterogeneous population, the use of In case of heterogeneous population, the use of stratified sampling methods should be preferred to stratified sampling methods should be preferred to simple random samplingsimple random sampling

• Finally, a major difference emerges between EU Finally, a major difference emerges between EU (using independent drawing of the sample each (using independent drawing of the sample each month) and the US (using a rotating sample design)month) and the US (using a rotating sample design)

• The adoption of the US method may possibly The adoption of the US method may possibly enhance research option available to analyst even in enhance research option available to analyst even in the EUthe EU

Minimum requirements and recommendation for Minimum requirements and recommendation for CS: weightingCS: weighting

• Weighting is recommended in order to Weighting is recommended in order to ensure better representitiveness ensure better representitiveness • Demographic characteristic of the Demographic characteristic of the households may be used as weights, households may be used as weights, considering among them:considering among them:

• age and genderage and gender• region of residence and size of the townshipregion of residence and size of the township

• Alternatively, socio-economic characteristics Alternatively, socio-economic characteristics may be used as weights:may be used as weights:

• type of occupationtype of occupation• Level of educationLevel of education• type of area municipalitytype of area municipality

THANK YOU FOR YOUR ATTENTION!THANK YOU FOR YOUR ATTENTION!

Task Force Members are: Richard Curtin (University of Task Force Members are: Richard Curtin (University of Michigan, United States), Isabelle De Greef (National Michigan, United States), Isabelle De Greef (National Bank of Belgium), Richard Etter (KOF / ETZ, Bank of Belgium), Richard Etter (KOF / ETZ, Switzerland), Christian Gayer (European Commission), Switzerland), Christian Gayer (European Commission), Marie Hormannova (CZSO, Czech Republic), Marco Marie Hormannova (CZSO, Czech Republic), Marco Malgarini (Institute for Studies and Economic Analysis Malgarini (Institute for Studies and Economic Analysis – ISAE, Italy), Rony Nilsson (OECD), Raymund Petz – ISAE, Italy), Rony Nilsson (OECD), Raymund Petz (GKI Research, Hungary), Takashi Sakuma (ESRI, (GKI Research, Hungary), Takashi Sakuma (ESRI, Japan), Philippe Scherrer (INSEE, France), Anna Stangl Japan), Philippe Scherrer (INSEE, France), Anna Stangl (Ifo Institute for Economic Research, Germany), (Ifo Institute for Economic Research, Germany), Andres Vertes (GKI Research, Humgary), Peter Weiss Andres Vertes (GKI Research, Humgary), Peter Weiss (European Commission), Jonathan Wood (European Commission), Jonathan Wood (Confederation of British Industry – CBI, United (Confederation of British Industry – CBI, United Kingdom).Kingdom).

Joint EC-OECD Workshop on International Development of Business and Consumer Tendency Surveys...

Documents

Transcript of Joint EC-OECD Workshop on International Development of Business and Consumer Tendency Surveys...