How to Design a Sample and Improve Response Rates

Alex Stannard Scottish GovernmentKevin Ralston University of Stirling

Plan

• Types of sample

• Sampling frames

• Improving response rates

• Group exercise

Census or sample?

Edinburgh City Council wants to know peoples thoughts on its leisure facilities.

A decision is made to send a questionnaire to every household in Edinburgh City. A 5% response rate is expected.

This is a census of all households in Edinburgh City

What is Sampling?

• Identify population

• Select members of population to sample

• Study selected members (the sample)

• Draw inferences about population from sample

Types of samples

• Non-Probability SamplesNot all members of the population have a

chance to be included in the sample. Selection method not random.

• Probability Samples Every member of the population has a known,

nonzero chance of being included in the sample. The selection method is random.

Probability Sampling

Example – 1000 households randomly selected across Argyll and Bute.

• More expensive and slower• Non response a problem – but resources can be

targeted and extent of non response bias can be estimated.

• Enables precision of final statistics to be assessed.• Sample selection method is objective, specified and

replicable.

Simple random sampling

• Sampling method completely random based on random numbers.

• Is easy to understand, but can be expensive

Example – every household in Aberdeen City assigned a number. 1,000 random numbers chosen between 1 and 204,683. These numbers identify which households are in sample.

• Tables of random numbers• www.random.org• Excel function ‘=rand()’

Systematic Sampling

• Uses a ‘random’ start on the sampling frame and then selects every i’th unit/person.

• Easy to understand, quick and easy to implement. • Can lead to some stratification depending on how the list

is ordered.• Can be expensive• Need to be careful on how list is ordered to avoid bias.

Systematic SamplingExample

Total units on sampling frame = 60

Want sample of 10

Interval size is 60/10 = 6

Select random start between 1 and 6

Select every sixth unit

1 16 31 462 17 32 473 18 33 484 19 34 495 20 35 506 21 36 517 22 37 528 23 38 539 24 39 54

10 25 40 5511 26 41 5612 27 42 5713 28 43 5814 29 44 5915 30 45 60

Sampling Frame

Stratified Sampling

• Units/people are aggregated into subgroups called strata. A certain number of units are sampled from each stratum.

• Guards against unusual samples• Stratification information has to be available

United Kingdom (60m)

England (51m) Northern Ireland (2m) Scotland (5m) Wales (3m)

Stratified Sampling

Proportionate Stratification• Chance of inclusion in sample is same for all

units/people regardless of strata

Population

United Kingdom 61 000 000

England 51 000 000

Northern Ireland 2 000 000

Scotland 5 000 000

Wales 3 000 000

Sample Size

6 100

5 100

200

500

300

Sampling Fraction

0.01%

0.01%

0.01%

0.01%

0.01%

Stratified Sampling

Disproportionate Stratification• Chance of a unit/person being included in the sample

depends on the strata they are in.• Often used to target small sub groups to help analysis

Population

United Kingdom 61 000 000

England 51 000 000

Northern Ireland 2 000 000

Scotland 5 000 000

Wales 3 000 000

Sample Size

6 100

3 100

1 000

1 000

1 000

Sampling Fraction

0.01%

0.01%

0.05%

0.02%

0.03%

Sampling Frames

• List of all units/people that could be included in sample

• Sample is only as good as the sampling frameo Eligible units/people not on frame cannot be selected

– leads to coverage erroro Units/people on frame more than once changes

probability of being selectedo Ineligible units/people on frame can lead to final

sample being smaller than intended.

Intended and achieved populations

Intended Population

Coverage Bias

Sampling Frame

Sampling Variance

Selected Sample

Non Response

Achieved Sample

Improving Response Rates

Improving Response Rates

• Incentives

• Advanced Letters & Reminders

• Respondent Burden

• Call Back

Incentives

• Incentives – monetary or gift, e.g. pen, gift token?

• Incentives generally improve response rates.

For postal surveys: If the budget is an issue then follow up mailings are preferred, if time is an issue incentives are preferred (Larson and Chow 2003).

– Robertson et al. (2005) found a Lottery scratch card incentive increased postal survey response by 9.6%.

– Prize draws may be ineffective - the incentive is abstract and distant from the participant (McCarty 2006).

Advanced Letters and Reminders

• Cost Effective – they work

• Sponsor – who is the survey for? People are more likely to provide information to the government (or local authority) than to a private company.

• Salience – emphasise the importance a topic may have to a respondent.

…but... Overemphasising importance or sensitivity may put some people off (Groves 2006).

Strike a balance.

Respondent Burden

• Burden –The right questions and only those that need to be asked.

• McCarty et al. (2006) found that a 10 minute increase in survey length results in a 7% decrease in response rates for telephone surveys.

Call Back

• Repeated call back improves response particularly used in face to face interviews.

– Minimum standard: rules about timing and number of call attempts before classification as non-contact. Training techniques to avoid refusal.

– Extended measures: passing on to a senior interviewer; phone call and appointment, letter and appointment; letter from study director.

(Lynn and Clarke 2002)

In postal surveys we can re-issue the questionnaire

General Recommendations

• Include incentives (if the budget is an issue follow ups are preferred)

• Issue advanced letters

• Keep respondent burden as low as possible

• Have a clear call back procedure

• Make it salient

Group Exercise