Goodness of Measures Reliability A matter of whether a particular technique, applied repeatedly to...
-
Upload
patricia-foster -
Category
Documents
-
view
216 -
download
0
Transcript of Goodness of Measures Reliability A matter of whether a particular technique, applied repeatedly to...
Goodness of MeasuresReliability
A matter of whether a particular technique, applied repeatedly to the same object, yields the same result each time.
How stable and consistent is the measuring instrument?
ValidityThe extent to which an empirical measure
adequately reflects the real meaning of the concept under consideration.
Are we measuring the right thing?
ReliabilityStability
refers to the ability of a measure to maintain consistency over time, despite uncontrollable testing conditions or the state of the respondents themselves
Internal consistencyindicates how well the items ‘hang together as
a set’ and can independently measure the same concept, so respondents attach the same overall meaning to each of the items
Forms of validityFace validity
That quality of an indicator that makes it seem a reasonable measure of a variable.
Criterion related validityThe degree to which a measure relates to some
external criterion. For example, the validity of the VCE tests is shown in their ability to predict the college success of students.
Forms of validity (cont.)Construct validity
The degree to which a measure relates to other variables as expected within a system of theoretical relationships.
Content validityRefers to how much a measure covers the
range of meanings included within a concept.
The Nature of Questionnaire Surveys (Veal)
Data collected via a printed questionnaire or interview schedule
Usually based on a sample selected from a population (see Chapter 11)
Reliance on information from individuals – self-reported data
Often large samples require computer-aided analysis
Merits of the Questionnaire Survey MethodQuantified data for decision making
Ideal for providing quantified information for organisations which need quantified information for decision-making.
TransparencyProvide a 'transparent' set of research
procedures. Procedures clear for all to see. Data can be re-analysed for alternative
interpretation.
Merits of the Questionnaire Survey Method (continued)Succinct presentation
Quantification can provide complex information in a succinct form.
ComparabilityLongitudinal and annual surveys enable the
study of change over time.Capturing complexity
An effective means of gathering a wide range of complex information on individuals or organisations.
Why do people answer questionnaires?
Social exchange theory (Blau, 1964, Dillman 1978)
Likelihood of response is a function of:Rewards: positive regard, verbal appreciation,
helping, group values, interest, social validation
Costs: subordination, embarrassment, inconvenience, complexity, privacy
Trust: tokens of appreciation, sponsorship, importance, other relationships
When to use questionnairesGOOD FORDescriptive researchExplanatory research
BUT NOT FORExploratory research
Consider MULTIMETHOD approaches
Types of questionnaire(Veal, Figure 8.2)Self-administeredMail surveys -
questionnaires sent and returned by mail;
E-surveys - questionnaires sent and returned by email/Internet;
Captive group surveys - members of groups are surveyed
Delivery and collectionOrganisation surveys -
members of an organisation/ organisations are surveyed
Interviewer-administeredHousehold surveys –
respondents selected on the basis of where they live and interviewed in their home
Street/quota/intercept surveys - respondents selected by stopping in street, shopping malls, etc
Telephone surveys - interviews are conducted by telephone
Customer/visitor on-site intercept surveys – users/customers surveyed on site
Household surveysAdvantagesRapport and
motivation for resp.Clarification
possibleRead non-verbal
cuesVisual aids possibleRicher dataCAPI available
DisadvantagesTake personal timeCosts are higherConfidentiality
concernsInterviewer training
vitalInterviewer biases
possible
Telephone InterviewsLess costly and faster than mail or personal
Wide geographical areaHigher anonymity than
personalCan be completed
using CATI - Computer-aided Telephone InterviewingAutomatic sampling and
diallingAnswers keyed directly
into computer (quick, cost-effective)
Respondents can terminate at any time
Non-verbal cues not available
Need to be kept short (AMSRS - 20 min.)
Representativeness of telephone sampling lists?
Mail QuestionnairesHigh anonymityWide geographic
regionsCan include token
giftsAt respondent
convenience
Relatively low response rate
No opportunity for clarification
Follow-up procedures requiredTotal design method
(Dillman, 1978)
Mail survey response ratesResponse rate affected by:
Interest of the respondent in the survey topicLength of the questionnaireQuestionnaire design, presentation, perceived
complexityStyle, content and authorship of accompanying
letterProvision of a reply-paid envelopeRewards for respondingNumber/timing of reminders/follow-ups – see
Fig. 8.3
Electronic QuestionnairesEasy to administerGlobal reachInexpensiveFastAt respondent’s
convenience
Require computer literacy
Respondents must have computer access
Must be willing to complete
Questionnaire design (Veal)Topics:
Research problems and information requirements
Types of information Open-ended and pre-coded questions Wording of questions Measuring attitudes and opinions Ordering of questions Layout Filters Introductory remarks
Questionnaire design process
Type of QuestionsOpen-ended questions
Freedom to answer the question any way one wants Could pose problems for researcher in coding the
response
Closed questionsAlternatives that are mutually exclusive and
collectively exhaustive, given Respondent can make a quick decision Easy to intepret and code
Open-ended and pre-coded questions
Wording of questionsAvoid jargonSimplify wherever possible: Would the respondent
understand the words in the questionnaire?Avoid ambiguityAvoid leading questionsAsk only one question at a time (avoid multi-
purpose questions) - Are there double-barrelled questions?
Are there any recall-dependent questions?Is there any potential of social desirability?
Validity of questionnaire surveysValidity: the extent to which the data truly reflect what they are thought to reflect
Validity may be affected by:Exaggeration or under-statement (to give good
impression)Halo effectsInaccuracy of recall (recency bias)Desire to please the interviewerOrder effects
Some checks:Include ‘dummy’ categories in some questions–
eg. books that do not exist, events that have not happened
As the same question twice – in different wayRepeat interviews – some time later
Possible sources of questionsADOPT questions used in other
questionnairesGenerally already tested for validityCan compare consistency of responses (i.e.
reliability)ADAPT questions used in other
questionnairesDESIGN your own questions
Forms of closed questions(SLT p.292)List (tick all that apply)Category (which one applies to you?)Ranking (relative importance)Rating (agree-disagree, good-bad, …)Quantity (how many – record a number)Grid (multiple question responses)
Measuring attitudes and opinions
Formats
a.Open-ended or direct questionsb.Checklistc.Ranking d.Likert scalese.Attitude statementsf.Semantic differential
Classification Data – Personal Information or Demographic DataDecisions on:Name, number required? (anonymity)What info required? (income, marital
status, etc needed?)Age, income, etc, should a range be given?
If so, what are the appropriate ranges?Eg, <20, 20 – 40, >40
Should personal info be at beginning or end of questionnaire?
Decisions should be based on the research questions you need to answer!!
Ordering of questionsStart with easy questionsStart with 'relevant' questionsLeave sensitive questions until later
Sequencing of QuestionsThe Funnel ApproachFrom general to specific questionsFrom easy to difficult questions
Questionnaire layoutBe aware of reader/user – interviewer or
respondent?Special care with mail survey questionnairesCompactness (eg. single page) = ease of
handlingTwo-column layout often helps
Filtering
Introductory remarksEg. ‘Hallo, my name is _____ : we are
conducting a survey of _______: would you mind answering a few questions?
Different content/formats for different survey typesInterviewer-completion: include in interviewer
instructions – additional information available if required
Respondent-completion – printed on questionnaire
Mail survey: this is dealt with in covering letter
Introduction to respondents
Consider social exhange – increase benefits and trust, reduce costs
identity of researcher and purpose of surveyconfidentiality & anonymitydistribution and/or publication of resultscourteous note, thanksAim is to provide opportunity for INFORMED
CONSENTOptions: covering letter, introductory statement
InstructionsDefine a desired navigational path for each
pageCreate visual navigational guides to assist
respondents to adhere to navigational pathDevelop additional visual guides for
disruption where redirection is required
(Dillman, 2000: 96)
Pre-testing QuestionnairesFace validityContent validityPilot study
With sample of respondents from target population
Ideal research design Factor analysis, reliability analysis