Applied Opinion Research Training Workshop Day 3.

Applied Opinion Research Training Workshop

Day 3

Preparing to Conduct Research

Sharon Felzer

Conducting Research

Going into the Field Prepare Materials

Translate and back-translate questionnaires and guidelines

Provide sufficient copies, return mail envelopes, etc. Determine Schedule

Do not field right before or during holidays Allow enough time to recruit sample, conduct research,

analyze data, and prepare reports

Conducting Research

Going into the FieldGather Your Resources

Translators Local contacts to recruit samples Local interviewers/moderators Transcribers/Data entry Data analysts Contractors (ESOMAR Website)

Conducting Research

Instrument Quality Control Vet among core research team

Be sure research objectives will be met Pilot test

Be sure participants will understand instructions, questions, and skip patterns

Be sure length is appropriate Be sure guideline encourages discussion

Back-translate Be sure questions ask what they were intended

to ask

Conducting Research

Avoid Common Pitfalls Poorly designed instruments

Research is only as useful as the instruments used Poorly recruited sample

Findings are highly dependent on the sample used Unskilled interviewers/moderators Generalizing beyond research/sample No buy-in Poor timing of research Poorly designed Terms of Reference Cost overrun

Conducting Research

When to Bring in Contractors/Experts Language barrier Cultural barrier Large or multiple samples Local contacts needed to encourage participation Preventing bias Complicated research objectives

Analyzing and Reporting Results

Mary McIntosh

Analyzing & Reporting Results

Essential Components of Successful Research: Appropriate sampling Valid instruments Accurate translations Skilled interviewers/moderators Effective timing Accurate data entry/transcriptions Appropriate analyses and reporting Reasonable response rate


Analyzing DataUse statistics appropriate to responses

Rating scale: means and standard deviations or ranges

Dichotomous and multiple choice:frequencies of respondents’ responses

Ranking: counts of #1 ratings, etc. Open-ended/Qualitative: no statistics appropriate

unless transcripts are coded


Analyzing Quantitative DataPay attention to:

Sample base Number of respondents

– The smaller the n, the less stable the parameter estimates

Mean or frequency of response Standard deviation

– The larger the standard deviation, the less reliable is your estimate of the mean

Test of significance– If the test is not significant, you cannot say it is a

significant difference


Analyzing Quantitative DataTests of significanceTells you whether differences in numbers are meaningful, or significant

Most commonly conducted on frequency and mean data

A significance test of p < .05 tells you that there is less than a 5% chance that this difference in mean responses is due to chance

A test of significance must be conducted before you can say that a difference exists

However, if a test is significant, it is not necessarily meaningful


Group 1– Mean = 3.00– Std. Dev. = 1.00– 95% of respondents’

ratings between 1 and 5







Analyzing Quantitative DataTests of significance

Not Not differentdifferent

SignificanSignificantly tly

differentdifferent


Group 1– 45% agreed– N = 200




Analyzing Quantitative DataTests of significance

Not Not differentdifferent

SignificanSignificantly tly

differentdifferent


Analyzing Qualitative DataPay attention to:

Sample base Trends in responses Potential group differences in responses Any potential bias in responding


Reporting ResultsKeep in Mind:

Your data is from a sample of the population only– Be very careful about generalizing your results to the entire

population Only a subset of your intended sample actually took part

in your research– Low response rates suggest potential response bias

You may want to consider weighting your data to correct for bias



Your questionnaire or guideline asked a finite number of questions– It is quite possible that you did not ask about an important

factor Despite your best efforts, there may still have been

biases– Researcher biases (more likely in qualitative)– Sample biases– Cultural biases



Although the data from quantitative data seems quite scientific, it is still subject to interpretation

Reporting attitudes and opinions is not the same as reporting facts


Reporting ResultsReporting the Sample:

The population Sampling frame Sample design Rationale for sample design Sample size Response rate You may want to report the number of respondents who

answered each question if there are high numbers of “Don’t Know” or “Refused” responses


Reporting ResultsReporting the Sample:

What if your response rate is lower than expected or what is traditional for that type of research in that country? Report that this may be a potential limitation. For instance,

some respondents may have been discouraged to participate by superiors. Therefore, the sample of respondents that did take part in the research may not be representative of the population

If possible, check the characteristics of those who participated versus those who did not


Reporting ResultsReporting the SampleBecause your data is only from a sample of the population, you must be very careful about generalizing your results to the entire population

WRONG: NGOs believe the Bank is not effectiveRIGHT: NGOs who were interviewed believe

that the Bank was not effective


Reporting ResultsQuantitative Data:

Although the data from quantitative data seems quite scientific, it is still subject to interpretation

– A mean of 7 on a 10-point scale may be a somewhat positive response in one context (e.g., culture) or a very positive response in another context (e.g., culture)

– Don’t assume scales are uniform and consistent across topics and cultures



Part of your interpretation involves deciding how to report the data– Mean scores

» Respondents from NGOs gave Bank effectiveness a mean rating of 7.5

– Aggregate frequencies (e.g., High/Medium/Low)» 35% of respondents from NGOs gave a high rating for the Bank’s

effectiveness– Response frequencies

» 25% of respondents from NGOs gave a rating of 6 for the Bank’s effectiveness



Report the data in the way that your readers will be most likely to understand

Include appropriate charts, graphs, or tables to represent the data pictorially to assist your readers


Reporting ResultsQualitative Data:

Because this data is highly subjective and completely dependent on the particular sample you have drawn, you must be very careful in reporting qualitative findings

WRONG: The Bank program failed because it was poorly designed

RIGHT: Respondents in a beneficiaries focus group reported that one potential reason the Bank program was not that effective was that the design was not optimal given the situation

Always include the caveat that your findings are not necessarily representative of the population (or the +/-)


Reporting Results:Right & Wrong– Finding: When asked to choose what the Bank’s greatest value

was, 25% of respondents (the largest %) chose financial resources

WRONG: The Bank’s value is only in its financial resources

RIGHT: A quarter of respondents chose financial resources as the Bank’s greatest value. This was followed by…


Reporting ResultsRight & Wrong– Finding: When asked about the Bank’s overall effectiveness,

NGO’s mean was 6.5 and private sector’s mean was 7.8, a significant difference

WRONG: NGOs do not think that the Bank is an effective organization. In contrast, people in the private sector think that the Bank is highly effective

RIGHT: Respondents from private sector organizations rated the Bank’s overall effectiveness significantly higher (7.8) than respondents from NGOs (6.5).


Reporting ResultsRight & Wrong– Finding: When asked about the Bank’s overall effectiveness, the

3 media respondents gave a mean of 9.0

WRONG: The media think that the Bank is very effectiveRIGHT: Although respondents from the media rated the

Bank’s overall effectiveness quite high (9.0), there were only three media respondents, thus, these results are suggestive at best

» Another alternative is to not report results from samples sizes that you judge too small to be reliable


Reporting ResultsRight & Wrong– Finding: In a focus group of medical professionals,

several said that the Bank needs to initiate a vaccine program

WRONG: The Bank needs to initiate a vaccine programRIGHT: In a focus group of medical professionals, it was

recommended that the Bank initiate a vaccine program

Hands On Work:Analyzing and Reporting Results

Applied Opinion Research Training Workshop Day 3.

Documents

Transcript of Applied Opinion Research Training Workshop Day 3.