Asking Survey Questions in Many Tongues Paper presented at the ESS Launch Conference Brussels,...
-
Upload
morgan-kerr -
Category
Documents
-
view
219 -
download
3
Transcript of Asking Survey Questions in Many Tongues Paper presented at the ESS Launch Conference Brussels,...
![Page 1: Asking Survey Questions in Many Tongues Paper presented at the ESS Launch Conference Brussels, November 25-26, 2003 Janet A Harkness, ZUMA Germany.](https://reader035.fdocuments.net/reader035/viewer/2022062417/551525885503465e608b550d/html5/thumbnails/1.jpg)
Asking Survey Questionsin Many Tongues
Paper presented at theESS Launch Conference
Brussels, November 25-26, 2003
Janet A Harkness, ZUMAGermany
![Page 2: Asking Survey Questions in Many Tongues Paper presented at the ESS Launch Conference Brussels, November 25-26, 2003 Janet A Harkness, ZUMA Germany.](https://reader035.fdocuments.net/reader035/viewer/2022062417/551525885503465e608b550d/html5/thumbnails/2.jpg)
Overview
• English as survey lingua franca• The ESS translation context• ESS requirements and procedures• Common survey practice
and where the problems lie• The ESS and lessons learned
![Page 3: Asking Survey Questions in Many Tongues Paper presented at the ESS Launch Conference Brussels, November 25-26, 2003 Janet A Harkness, ZUMA Germany.](https://reader035.fdocuments.net/reader035/viewer/2022062417/551525885503465e608b550d/html5/thumbnails/3.jpg)
English as survey lingua franca
• We can all speak English• Source questionnaires often in English• Texts anchored to language• Language-driven challenges (two way)
![Page 4: Asking Survey Questions in Many Tongues Paper presented at the ESS Launch Conference Brussels, November 25-26, 2003 Janet A Harkness, ZUMA Germany.](https://reader035.fdocuments.net/reader035/viewer/2022062417/551525885503465e608b550d/html5/thumbnails/4.jpg)
The ESS Context: Facts and Figures• Round One fielded in 22 countries• Fielded in 20-21 languages, depending on how you count• All languages used as first by 5% and
more
• 5 countries fielded in more than one language (12 languages)
• Most often shared languages (to date): French Italian German.
• 9 countries shared one or more languages
![Page 5: Asking Survey Questions in Many Tongues Paper presented at the ESS Launch Conference Brussels, November 25-26, 2003 Janet A Harkness, ZUMA Germany.](https://reader035.fdocuments.net/reader035/viewer/2022062417/551525885503465e608b550d/html5/thumbnails/5.jpg)
The WP Context: Resources & Researchers
• ESF• EC and Partners• CCT • Scientific Advisory
Board• Participating
countries• Translation
Workpackage
Translation Expert Panel Dr. Hans Hönig,
Dr. Paul Kussmaul,
Beth-Ellen Pennell, MA,
Dr. Alisú Schoua-Glusberg
![Page 6: Asking Survey Questions in Many Tongues Paper presented at the ESS Launch Conference Brussels, November 25-26, 2003 Janet A Harkness, ZUMA Germany.](https://reader035.fdocuments.net/reader035/viewer/2022062417/551525885503465e608b550d/html5/thumbnails/6.jpg)
Key Workpackage Component
TRAPD
A framework for ESS translation and assessment to deal with the complex requirements
![Page 7: Asking Survey Questions in Many Tongues Paper presented at the ESS Launch Conference Brussels, November 25-26, 2003 Janet A Harkness, ZUMA Germany.](https://reader035.fdocuments.net/reader035/viewer/2022062417/551525885503465e608b550d/html5/thumbnails/7.jpg)
Translation framework TRAPD
Translation, Review, Adjudication, Pre-testing and Documentation
5 interrelated processes with protocols involved in producing a final translated version
TRAPD combines translation with assessment
![Page 8: Asking Survey Questions in Many Tongues Paper presented at the ESS Launch Conference Brussels, November 25-26, 2003 Janet A Harkness, ZUMA Germany.](https://reader035.fdocuments.net/reader035/viewer/2022062417/551525885503465e608b550d/html5/thumbnails/8.jpg)
Key Activities for Round One
• Develop framework for ESS translation and assessment: TRAPD
• Overview report on TRAPD
• Guidelines for NCs on translation and assessment procedures, team members ‘ roles.
• Strategies for languages shared across countries
• Documentation framework and guidelines
![Page 9: Asking Survey Questions in Many Tongues Paper presented at the ESS Launch Conference Brussels, November 25-26, 2003 Janet A Harkness, ZUMA Germany.](https://reader035.fdocuments.net/reader035/viewer/2022062417/551525885503465e608b550d/html5/thumbnails/9.jpg)
Additional practical measures
• Basic guidelines on organising translation and assessment, selecting & training translators
• Documentation template for problems, divergences, sharing harmonisation
• Source questionnaire annotated to help translators
• Consultation hotline
![Page 10: Asking Survey Questions in Many Tongues Paper presented at the ESS Launch Conference Brussels, November 25-26, 2003 Janet A Harkness, ZUMA Germany.](https://reader035.fdocuments.net/reader035/viewer/2022062417/551525885503465e608b550d/html5/thumbnails/10.jpg)
Ongoing work--Round One into Round Two
• Probe interviews with (external) translators on the source questionnaire
• Analysis of (external) translators‘ appraisals of selected questions from ESS translations
• Comparisons of selected translations for shared languages
![Page 11: Asking Survey Questions in Many Tongues Paper presented at the ESS Launch Conference Brussels, November 25-26, 2003 Janet A Harkness, ZUMA Germany.](https://reader035.fdocuments.net/reader035/viewer/2022062417/551525885503465e608b550d/html5/thumbnails/11.jpg)
In Sum
Funds, Guidelines, Specifications, Requirements, Illustrations and Examples Templates, Annotations, Hot-line, Presentations and Reports
Ongoing: cognitive-based work on the source questionnaires and translations
![Page 12: Asking Survey Questions in Many Tongues Paper presented at the ESS Launch Conference Brussels, November 25-26, 2003 Janet A Harkness, ZUMA Germany.](https://reader035.fdocuments.net/reader035/viewer/2022062417/551525885503465e608b550d/html5/thumbnails/12.jpg)
Isn’t this an unusual amount of effort?
Aren’t questionnaires easy to translate?
![Page 13: Asking Survey Questions in Many Tongues Paper presented at the ESS Launch Conference Brussels, November 25-26, 2003 Janet A Harkness, ZUMA Germany.](https://reader035.fdocuments.net/reader035/viewer/2022062417/551525885503465e608b550d/html5/thumbnails/13.jpg)
Questionnaires at first glance
• Easy questions, simple wording• Response categories often one or two
words• Simple and repeated instructions• Repetition of lexical and structural
elements throughout • Similar or identical questions in other
surveysAnd the work team often knows English
![Page 14: Asking Survey Questions in Many Tongues Paper presented at the ESS Launch Conference Brussels, November 25-26, 2003 Janet A Harkness, ZUMA Germany.](https://reader035.fdocuments.net/reader035/viewer/2022062417/551525885503465e608b550d/html5/thumbnails/14.jpg)
Questionnaires at second glance
• Questionnaires are complex texts•They look easier than they are• Good questions not easy to construct• Measurement-in-text poses special problems• Cross-disciplinary requirements (e.g., measurement, translation)• Interdisciplinary expertise neededA key problem: resulting challenges under-estimated
![Page 15: Asking Survey Questions in Many Tongues Paper presented at the ESS Launch Conference Brussels, November 25-26, 2003 Janet A Harkness, ZUMA Germany.](https://reader035.fdocuments.net/reader035/viewer/2022062417/551525885503465e608b550d/html5/thumbnails/15.jpg)
A Complex Text Type
• Text-with-blanks genre• Often largely generic • Texts within the text• Low Context- jump from one topic to
another• Cross-national aims encourage general
formulations• Computer applications can technically
complex source formats
![Page 16: Asking Survey Questions in Many Tongues Paper presented at the ESS Launch Conference Brussels, November 25-26, 2003 Janet A Harkness, ZUMA Germany.](https://reader035.fdocuments.net/reader035/viewer/2022062417/551525885503465e608b550d/html5/thumbnails/16.jpg)
One example of not-so-easy features
• Measurement properties lead to ‘surveyspeak’ and ‘scalespeak’
Surveyspeak and Scalespeak are textual traces of measurement(Harkness 1996)
![Page 17: Asking Survey Questions in Many Tongues Paper presented at the ESS Launch Conference Brussels, November 25-26, 2003 Janet A Harkness, ZUMA Germany.](https://reader035.fdocuments.net/reader035/viewer/2022062417/551525885503465e608b550d/html5/thumbnails/17.jpg)
Surveyspeak
Q: All things considered, how satisfied are you with your life as a whole nowadays ? (ESS, B13)
Q: How‘s life?
![Page 18: Asking Survey Questions in Many Tongues Paper presented at the ESS Launch Conference Brussels, November 25-26, 2003 Janet A Harkness, ZUMA Germany.](https://reader035.fdocuments.net/reader035/viewer/2022062417/551525885503465e608b550d/html5/thumbnails/18.jpg)
Q: When did you last visit a dentist?
Would you say that you are //very certain, somewhat certain, neither certain nor uncertain, somewhat uncertain or very uncertain// about the date you just gave?
A:
Conversational Follow-up: Uh-HuhReally ?Are you sure?
![Page 19: Asking Survey Questions in Many Tongues Paper presented at the ESS Launch Conference Brussels, November 25-26, 2003 Janet A Harkness, ZUMA Germany.](https://reader035.fdocuments.net/reader035/viewer/2022062417/551525885503465e608b550d/html5/thumbnails/19.jpg)
Scalespeak -- response categories
How important or unimportant is ----
Extremely ImportantSomewhat importantNeither important/or unimportantSomewhat unimportantExtremely unimportant
![Page 20: Asking Survey Questions in Many Tongues Paper presented at the ESS Launch Conference Brussels, November 25-26, 2003 Janet A Harkness, ZUMA Germany.](https://reader035.fdocuments.net/reader035/viewer/2022062417/551525885503465e608b550d/html5/thumbnails/20.jpg)
Scalespeak
Extremely unimportant (ESS E13)
Somewhat importantSomewhat unimportant
Somewhat difficult
Somewhat ugly
![Page 21: Asking Survey Questions in Many Tongues Paper presented at the ESS Launch Conference Brussels, November 25-26, 2003 Janet A Harkness, ZUMA Germany.](https://reader035.fdocuments.net/reader035/viewer/2022062417/551525885503465e608b550d/html5/thumbnails/21.jpg)
Response scales are difficult to translate
Differences across languages/ cultures in• quantification structures & potential negation
• saliency of concept (satisfaction, justness) • disclosure norms• response styles• numeracy and related notions• scaling norms (1-6; 1-10?)
![Page 22: Asking Survey Questions in Many Tongues Paper presented at the ESS Launch Conference Brussels, November 25-26, 2003 Janet A Harkness, ZUMA Germany.](https://reader035.fdocuments.net/reader035/viewer/2022062417/551525885503465e608b550d/html5/thumbnails/22.jpg)
Known Measurement Effects
vs. Allow/Not allow
Allow/ Forbid
![Page 23: Asking Survey Questions in Many Tongues Paper presented at the ESS Launch Conference Brussels, November 25-26, 2003 Janet A Harkness, ZUMA Germany.](https://reader035.fdocuments.net/reader035/viewer/2022062417/551525885503465e608b550d/html5/thumbnails/23.jpg)
So what about differences such as:
vs.Agree/
Not agree
Agree/ Disagree
Agree/Reject
or
![Page 24: Asking Survey Questions in Many Tongues Paper presented at the ESS Launch Conference Brussels, November 25-26, 2003 Janet A Harkness, ZUMA Germany.](https://reader035.fdocuments.net/reader035/viewer/2022062417/551525885503465e608b550d/html5/thumbnails/24.jpg)
Translations may not be able to match English scale properties
German: ‘Agree’ / ‘Not agree’/ ‘Reject’
English : Agree/ Not agree / Disagree
![Page 25: Asking Survey Questions in Many Tongues Paper presented at the ESS Launch Conference Brussels, November 25-26, 2003 Janet A Harkness, ZUMA Germany.](https://reader035.fdocuments.net/reader035/viewer/2022062417/551525885503465e608b550d/html5/thumbnails/25.jpg)
Despite these complexitiesviews on translationtranslatorsassessment and testingare often pre-theoreticaland procedures followed not grounded in research
![Page 26: Asking Survey Questions in Many Tongues Paper presented at the ESS Launch Conference Brussels, November 25-26, 2003 Janet A Harkness, ZUMA Germany.](https://reader035.fdocuments.net/reader035/viewer/2022062417/551525885503465e608b550d/html5/thumbnails/26.jpg)
Not by chance that the ESS work package focused on Survey translation process procedures
Specifications, briefing, & training materials for translators and for those hiring translators
Survey translation evaluation procedures
Providing access to know-how to implement translation components
![Page 27: Asking Survey Questions in Many Tongues Paper presented at the ESS Launch Conference Brussels, November 25-26, 2003 Janet A Harkness, ZUMA Germany.](https://reader035.fdocuments.net/reader035/viewer/2022062417/551525885503465e608b550d/html5/thumbnails/27.jpg)
These are what the survey community in general still lacks
Thus the ESS effort and activities meet real needs
It is a necessary amount of effort
![Page 28: Asking Survey Questions in Many Tongues Paper presented at the ESS Launch Conference Brussels, November 25-26, 2003 Janet A Harkness, ZUMA Germany.](https://reader035.fdocuments.net/reader035/viewer/2022062417/551525885503465e608b550d/html5/thumbnails/28.jpg)
In addition
• No design costs, only translation and assessment • Core questionnaire designed for replication
• Even ESS translations (in a deluxe version) are a very small part of total national survey costs.
![Page 29: Asking Survey Questions in Many Tongues Paper presented at the ESS Launch Conference Brussels, November 25-26, 2003 Janet A Harkness, ZUMA Germany.](https://reader035.fdocuments.net/reader035/viewer/2022062417/551525885503465e608b550d/html5/thumbnails/29.jpg)
Lessons learned from:
Contact with national teams
Documentation from national teams
Discussions within the Expert Panel and CCT
Cognitive interviews with external translators
Related work in other projects(eg SHARE, EUYOUPART, ISSP)
![Page 30: Asking Survey Questions in Many Tongues Paper presented at the ESS Launch Conference Brussels, November 25-26, 2003 Janet A Harkness, ZUMA Germany.](https://reader035.fdocuments.net/reader035/viewer/2022062417/551525885503465e608b550d/html5/thumbnails/30.jpg)
Some lessons learned
1. Procedures and guidelines crucial Team work Iterative translation and review Sharing and harmonisation
2. More hands-on input would help more On the job training Training opportunities and materials
![Page 31: Asking Survey Questions in Many Tongues Paper presented at the ESS Launch Conference Brussels, November 25-26, 2003 Janet A Harkness, ZUMA Germany.](https://reader035.fdocuments.net/reader035/viewer/2022062417/551525885503465e608b550d/html5/thumbnails/31.jpg)
Lessons learned continued
3. Pretesting –TRApD –more needed on source questionnaires
4. Version documentation is essential – but the Ugly Duckling of the work package. Need to help it have its proper place in a user-friendly form
5. Version harmonization raises scheduling issues. No harmonization discussion raises comparability issues
6. More intensive cross-cultural input neededQuestionnaire contentQuestionnaire formulations
[Raises planning, timing and procedural questions]
![Page 32: Asking Survey Questions in Many Tongues Paper presented at the ESS Launch Conference Brussels, November 25-26, 2003 Janet A Harkness, ZUMA Germany.](https://reader035.fdocuments.net/reader035/viewer/2022062417/551525885503465e608b550d/html5/thumbnails/32.jpg)
Thank you very much for your time and attention