Post on 21-Mar-2016
description
Field test optionsinstrument, instructions,
non-response/ refusals, and Interviewer debriefing
Washington Group Regional Training Workshop
Rio de Janeiro19 – 20 September 2005
Margie SchneiderHSRC, South Africa
Recap What we have covered so far:
– Purpose of the questions and where they came from
– Core set of questions and extended set and Q X Q specifications
– Objectives of the test– Translation protocol
What we still need to go through: – How to design the test and related issues– Enumerator training– Plan for analysis and report writing
Different components of the testing process
• Pre-testing and expert review • Testing internal validity on people with
known disability status (quota sampling and linked to cognitive testing)
• Pilot testing – as for full Census or survey
• Full field test
Different sampling approaches will be discussed in relation to different testing protocols
Field Test options• 2 questions to be answered:
1. What set of questions should be used in field test?
2. What field test approach should be used?
Consider your own context, need, funding, skills, etc. in deciding
Question sets1. Core set – 4 + 2 (6 domains of
functioning): Compulsory2. Core + extended set on core domains :
highly recommended3. Core + extended set + further
additional questions: nice to have for analysis• 2 additional domains (learning and
interpersonal interactions)• Psychological distress
4. Country specific question set : nice to have for comparison with prior data
What approach to testing?• What factors to consider in
deciding: –Country needs–Capacity to undertake test–Funds available–Time available–Planned activities, e.g. other
surveys– ?
Census or national survey• Use of question set in planned
Census or national survey• Advantages:
– Large sample (5000 – 10 000 or whole pop)
– Minimal additional cost for disability questions
• Disadvantages:– Not sufficient space for extended set– Need to administer extended set and
cognitive test to sub-sample (costs of time and additional training)
Special study – small sample• Select sample on basis of known
‘disability’ status • Allows for construction of 2X2 table for
sensitivity and specificity calculations and identifying true and false positives and negatives – use of ‘gold standard’
• Suggest around 200 true positives and 200 true negatives
• Interviewer does not know status of respondent (unless visible and obvious)
Special study (contd.) • Advantages:
– Larger set of questions– Close observation of interview– Cost is not exorbitant
• Disadvantages:– Might not get full population
representation– Cost of establishing true status is high– What counts as the true positive?
(beyond the easily observable) – What is the ‘gold standard’?
Special study – large sample• Uses population based sample• Sample size based on expected
prevalence of the different types of activity limitations
• Prevalence rates are usually low and so need large sample to yield enough ‘disabled’ respondents
Special study – large sample (contd)
• Advantages:– Large data set– Detailed set of questions - more than
extended set– Provides an indication of prevalence
using core and extended sets• Disadvantages:
– Costly and time consuming– Requires capacity to run and analyse
Field test and cognitive test• All respondents have core set • One sub sample of respondents
have cognitive test (core + extended are embedded)
• A second sub-sample of respondents do – Core + extended– Core + extended + further Qs– Core + extended + country specific
questions• Some do all ?
Non-responses and refusals•How do you define these?
•How do you deal with these?
•Aim to: –Reduce non-response and
refusals–Manage them when they do
arise
Reducing NR and refusals• Use up to date sampling frame• Clearly defined reasons for data
collection• Preparing the way – prior contact,
letters, etc. • Adequate interviewer training• Allow budget for call-backs• Plan for follow up of NR and refusals• Separate refusals, part refusals, non-
contacts and sample loss (e.g. vacant dwellings)
Effect of each on analysis• Sample loss: does not create bias but
reduces sample size; need large enough initial sample to take these into account
• Refusals and non-contact: bias where these respondents might be different to those respondents reached
• Need to understand reasons for refusals and non-contacts – during pre-testing and pilot stage
Interviewer debriefing• Interviewers are well placed to
evaluate process and content at an early stage
• Interviewers should note comments in margins of questionnaires
• Different approaches: – Group discussion (focus group technique)– Interviewer rating forms– Standardised interviewer questionnaires– Combination of all three
Enumerator training
Washington Group Regional Training Workshop
Rio de Janeiro19 – 20 September 2005
Margie SchneiderHSRC, South Africa
General points• Select some disabled interviewers• Importance of all interviewers
having a good understanding of what disability is and is not and how it relates to the questions asked
• Avoid using term ‘disabled’ or ‘with disabilities’
• All interviews are face to face• Translation training
General points (contd.)• Confidentiality and understanding what
this means• Getting informed consent (ethics
clearance?)• Interviewers must be able to explain
purpose of survey• Read questions in set order and with set
wording• Editing in field of completed questionnaires• Submitting of completed interviews to
head office
Types of interviews• Direct: respondent answers for him or
herself• Interpreted: an interpreter
‘translates’ and respondent answers directly
• Facilitated: a third party assists in explaining (e.g. intellectually disabled person)
• Proxy: a person responds for another (e.g. child)
Interviewing disabled people
• Show respect and treat the person like anyone else
• Don’t use first names unless permitted
• Address the person directly (not their attendant)
• Ask how you can adapt your presentation to make it easier (no need to ask what is wrong with person)
Hearing difficulties• Lip reading• Lighting• Face the person• Get person’s attention before
speaking• Reduce background noise• Set the context – especially when
changing topics• Use written communication (literate)
Physical difficulties • Accessibility of building where
conducting interviews• Presence of attendant and
confidentiality issues• Get to same level (e.g. sitting for person
using wheelchair)• Person to be seated comfortably• Address person directly• Pointing may be difficult • Person may need breaks to move around
Visual difficulties• Large print and small print for
cue cards• Braille versions of cue cards• Good contrast printing for
pictures and print (black on white or yellow)
• Identify yourself and others in the room verbally
Communication difficulties • Clarify preferred mode of
communication• Repeat what you think was said
to clarify unclear speech• Limit to yes / no questions
Specific learning difficulties• Manage problems in spatial
orientation, hand-eye coordination
• Limit auditory, visual and tactile distractions
• Avoid written text • Explain carefully (if verbal
language skills are affected)
Intellectual difficulties • Be careful with informed consent• Explain terms simply• Listen carefully• Have familiar person (friend or
relative) close by• Use pictures or role play with
little human or animal figures
Emotional or mental health difficulties
• Side effects of medication• Break up interview if too fatigued • Give encouragement and support• Manage expressions of
frustration• Manage stress
Hidden difficulties• Might not come forward with
information because of fear of stigma
• Effect of medication• May need to break up interview